Spectral bandwidth adaptive adjustment method, device, equipment, medium and program product

Abstract

The invention provides a spectral bandwidth adaptive adjustment method and device, equipment, a medium and a program product. The method comprises the following steps: acquiring a topological relation among a plurality of components in a satellite broadband communication system and a data index set of each component; constructing an adjacent matrix based on the topological relation; constructing a running state matrix based on the plurality of data index sets; inputting an adjacent matrix and the operation state matrix into an optimal Actor network to obtain a spectral bandwidth adjustment action; and issuing the spectral bandwidth adjustment action to the satellite broadband communication system to enable the satellite broadband communication system to execute the spectral bandwidth adjustment action. According to the spectral bandwidth adaptive adjustment method provided by the invention, the optimal Actor network is trained and deployed through the reinforcement learning algorithm, and the optimal Actor network can output an accurate spectral bandwidth adjustment action according to the adjacent matrix and the operation state matrix, so that the satellite spectral bandwidth is dynamically adjusted according to the real-time communication condition, and the flexibility of spectral bandwidth adjustment is improved; and the spectrum utilization rate is maximized.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technology, and in particular to a spectrum bandwidth adaptive adjustment method, device, equipment, medium and program product. Background Art

[0002] As a key means to make up for the blind spots of traditional terrestrial network coverage, satellite broadband communication technology has been widely used in remote areas, aviation, marine communications and other fields around the world. The satellite broadband communication system architecture includes multiple components such as satellite transmission domain, satellite data domain, and satellite voice domain, each of which is responsible for data transmission, Internet access and voice services. Existing satellite broadband communication systems usually pre-configure fixed satellite spectrum bandwidth to ensure basic communication service quality. This configuration method is based on statistical predictions of past communication needs and can meet business needs most of the time.

[0003] Although existing technologies provide basic communication guarantees, their static spectrum resource allocation strategies are obviously insufficient. Due to the scarcity and preciousness of satellite spectrum resources, fixed allocation often leads to uneven resource allocation, with resources being tight during peak hours and idle during underestimation periods, resulting in resource waste. In addition, with the dynamic changes in business needs and the continuous emergence of new businesses, static configuration is difficult to flexibly respond to sudden surges in communication needs, affecting user experience and service quality. Summary of the invention

[0004] The present invention provides a spectrum bandwidth adaptive adjustment method, device, equipment, medium and program product, which are used to solve the problem of low flexibility in the static configuration of spectrum resources in the prior art.

[0005] In a first aspect, the present invention provides a spectrum bandwidth adaptive adjustment method, comprising: Obtain the topological relationship between multiple components in the satellite broadband communication system and the data indicator set of each component; Based on the topological relationship, construct an adjacency matrix of the plurality of components; Based on the multiple data indicator sets, construct an operation status matrix of the multiple components; Inputting the adjacency matrix and the operation state matrix into an optimal Actor network to obtain a spectrum bandwidth adjustment action output by the optimal Actor network; the optimal Actor network is determined by a deep deterministic policy gradient DDPG algorithm combined with a deep learning network architecture of a graph convolutional network GCN and a deep neural network DNN; The spectrum bandwidth adjustment action is sent to the satellite broadband communication system so that the satellite broadband communication system executes the spectrum bandwidth adjustment action.

[0006] In one embodiment, the adjacency matrix includes at least a physical relationship adjacency matrix; and constructing the adjacency matrix of the plurality of components based on the topological relationship includes: Based on the topological relationship between the multiple components, determine whether there is a physical connection relationship between any two components; If there is a physical connection relationship between the two components, then determining the physical connection relationship value between the two components is 1; If there is no physical connection relationship between the two components, determining the physical connection relationship value between the two components to be 0; The physical relationship adjacency matrix is ​​constructed based on the physical connection relationship values ​​between the arbitrary two components.

[0007] In one embodiment, constructing the operation status matrix of the multiple components based on the multiple data indicator sets includes: Determine all data indicators included in the plurality of data indicator sets as target data indicators respectively; According to the data indicator set of each component, a target data indicator involved in the component is determined as a state indicator, and a target data indicator not involved in the component is determined as a non-state indicator; Determine the running state value between the component and the state indicator by using the data indicator state value corresponding to the state indicator, and set the running state value between the component and the non-state indicator to 0, so as to obtain the running state value between the component and each target data indicator; Constructing an operating state vector of the component according to an operating state value between the component and each target data indicator; Concatenate the running state vectors of each component into a running state matrix.

[0008] In one embodiment, the optimal Actor network is obtained by performing the following steps: Initialize the parameters of the Actor network and the Critic network; both the Actor network and the Critic network are constructed based on the GCN network architecture; Building a target Actor network based on the parameters of the Actor network, and building a target Critic network based on the parameters of the Critic network; Acquire current state information of the satellite broadband communication system; the current state information includes an adjacency matrix sample and an operation state matrix sample; Inputting the current state information into the Actor network to obtain a target action output by the Actor network for adjusting the spectrum bandwidth; Acquire the next moment state information of the satellite broadband communication system; the next moment state information is the state information of the satellite broadband communication system after executing the target action; Acquire first performance indicator information before the satellite broadband communication system performs the target action and second performance indicator information after the satellite broadband communication system performs the target action; Inputting the first performance indicator information and the second performance indicator information into a reward generation network to obtain an instant reward value output by the reward generation network; Generate an experience sample based on the current state information, the target action, the immediate reward value and the next moment state information; The next moment state information is updated as the current state information, and the step of inputting the current state information into the Actor network is iteratively executed until a preset number of iterations is reached, a plurality of experience samples are obtained, and the plurality of experience samples are stored in an experience playback buffer; Based on the experience replay buffer, the Actor network and the Critic network are trained to obtain an optimal Actor network.

[0009] In one embodiment, the reward generation network includes a first DNN, a second DNN, and a difference calculation layer; the step of inputting the first performance indicator information and the second performance indicator information into the reward generation network to obtain an instant reward value output by the reward generation network includes: Inputting the first performance indicator information into the first DNN to obtain a first evaluation score output by the first DNN; Inputting the second performance indicator information into the second DNN to obtain a second evaluation score output by the second DNN; The first evaluation score and the second evaluation score are input into a difference calculation layer to obtain an immediate reward value output by the difference calculation layer.

[0010] In one embodiment, the step of training the Actor network and the Critic network based on the experience replay buffer to obtain an optimal Actor network includes: Randomly taking out experience samples from the experience playback buffer; Inputting the current state information and the target action in the experience sample into the Critic network to obtain a first current Q value output by the Critic network; Inputting the next moment state information in the experience sample into the target Actor network to obtain the next moment execution action output by the target Actor network; Input the next moment state information and the next moment execution action into the target Critic network to obtain the next moment Q value output by the target Critic network; Determine a label value of the first current Q value based on the next moment Q value; Based on the first current Q value and its corresponding label value, training the Critic network to obtain a trained Critic network; Based on the experience replay buffer and the trained Critic network, the Actor network is trained to obtain an optimal Actor network.

[0011] In one embodiment, the step of training the Actor network based on the experience replay buffer and the trained Critic network to obtain an optimal Actor network includes: Randomly taking out experience samples from the experience playback buffer; Inputting the current state information in the experience sample into the Actor network to obtain a new action output by the Actor network; Inputting the new action and the current state information into the trained Critic network to obtain a second current Q value output by the trained Critic network; Determining a loss function of the Actor network based on the current state information, the new action, and the second current Q value; Based on the loss function of the Actor network, an optimization algorithm is used to train the Actor network, and the parameters of the Actor network are updated to obtain an optimal Actor network.

[0012] In one embodiment, the satellite broadband communication system includes a satellite transmission domain subsystem, a satellite data domain subsystem, and a satellite voice domain subsystem; The plurality of components of the satellite broadband communication system include a plurality of first components of the satellite transmission domain subsystem, a plurality of second components of the satellite data domain subsystem, and a plurality of third components of the satellite voice domain subsystem.

[0013] In a second aspect, the present invention further provides a spectrum bandwidth adaptive adjustment device, comprising: An acquisition module, used to acquire the topological relationship between multiple components in the satellite broadband communication system and a data indicator set of each component; A first construction module, configured to construct an adjacency matrix of the plurality of components based on the topological relationship; A second building module is used to build an operation status matrix of the multiple components based on multiple data indicator sets; A spectrum bandwidth adjustment action prediction module, used for inputting the adjacency matrix and the operation state matrix into an optimal Actor network to obtain a spectrum bandwidth adjustment action output by the optimal Actor network; the optimal Actor network is determined by a deep deterministic policy gradient DDPG algorithm combined with a deep learning network architecture of a graph convolutional network GCN and a deep neural network DNN; The sending module is used to send the spectrum bandwidth adjustment action to the satellite broadband communication system so that the satellite broadband communication system performs the spectrum bandwidth adjustment action.

[0014] In a third aspect, the present invention provides a device, comprising an electronic device, wherein the electronic device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the above-mentioned spectrum bandwidth adaptive adjustment methods are implemented.

[0015] In a fourth aspect, the present invention further provides a medium, comprising a non-transitory computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the above-mentioned spectrum bandwidth adaptive adjustment methods are implemented.

[0016] In a fifth aspect, the present invention also provides a product, comprising a computer program product, wherein the computer program product comprises a computer program, wherein the computer program can be stored on a non-transitory computer-readable storage medium, and wherein when the computer program is executed by the processor, the steps of any of the above-mentioned spectrum bandwidth adaptive adjustment methods are implemented.

[0017] The spectrum bandwidth adaptive adjustment method, device, equipment, medium and program product provided by the present invention collect the topological relationship and data indicator set of multiple components from the satellite broadband communication system to construct the adjacency matrix and operation status matrix of multiple components. Furthermore, since the optimal Actor network is an efficient and accurate decision model provided for spectrum bandwidth adjustment by using the DDPG algorithm combined with the deep learning network architecture of GCN and DNN, the optimal Actor network can output accurate spectrum bandwidth adjustment actions according to the adjacency matrix and the operation status matrix, realize dynamic adjustment of satellite spectrum bandwidth according to real-time communication conditions, improve the flexibility of spectrum bandwidth adjustment, maximize spectrum utilization, and reduce resource waste. At the same time, the real-time monitoring and adaptive adjustment mechanism ensures that stable services can be provided even during peak hours, reduces communication congestion caused by uneven resource allocation, and improves user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flow chart of the spectrum bandwidth adaptive adjustment method provided by the present invention.

[0020] Figure 2 It is an architecture diagram of the spectrum bandwidth adaptive adjustment method provided by the present invention.

[0021] Figure 3 This is a diagram of the satellite broadband communication system architecture provided by the present invention.

[0022] Figure 4 It is a schematic diagram of the intelligent agent training process provided by the present invention.

[0023] Figure 5 This is a deep learning network architecture diagram provided by the present invention.

[0024] Figure 6 It is a structural schematic diagram of the spectrum bandwidth adaptive adjustment device provided by the present invention.

[0025] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein.

[0028] Combine the following Figure 1-Figure 7 The present invention describes the spectrum bandwidth adaptive adjustment method, device, equipment, medium and program product provided by the present invention.

[0029] It should be noted that the spectrum bandwidth adaptive adjustment method provided in the embodiment of the present invention is implemented in the network management system of the satellite broadband communication system, and the execution of corresponding steps by each functional module in the control gateway system can be considered to be implemented by the spectrum bandwidth adaptive adjustment device. The embodiment of the present invention takes the spectrum bandwidth adaptive adjustment device as an example of the execution subject to describe the spectrum bandwidth adaptive adjustment method.

[0030] Combination Figure 1 , Figure 1 is a flow chart of a spectrum bandwidth adaptive adjustment method provided by the present invention, Figure 2 It is an architecture diagram of the spectrum bandwidth adaptive adjustment method provided by the present invention.

[0031] like Figure 1 As shown, the method includes the following: Step 101: obtaining a topological relationship between multiple components in a satellite broadband communication system and a data indicator set of each component; Step 102: construct an adjacency matrix of the plurality of components based on the topological relationship; Step 103: constructing an operation status matrix of the plurality of components based on the plurality of data indicator sets; Step 104: input the adjacency matrix and the operation state matrix into the optimal Actor network to obtain the spectrum bandwidth adjustment action output by the optimal Actor network; Step 105: Send the spectrum bandwidth adjustment action to the satellite broadband communication system so that the satellite broadband communication system performs the spectrum bandwidth adjustment action.

[0032] It should be noted that if Figure 2 As shown, an embodiment of the present invention adds a data acquisition and processing unit, an agent training unit, an agent reasoning unit and a spectrum bandwidth adjustment unit in the network management system of the satellite broadband communication system, wherein the data acquisition and processing unit is used to collect data from the satellite broadband communication system and process the collected data; the agent training unit is used to train and deploy the agent through a reinforcement learning algorithm; the agent reasoning unit is used to store and update the optimal Actor network that has been trained, and output the spectrum bandwidth adjustment action through the optimal Actor network; the spectrum bandwidth adjustment unit is used to generate a spectrum bandwidth adjustment instruction and send it to the satellite broadband communication system.

[0033] Specifically, firstly, multiple components in the satellite broadband communication system are determined. Since the satellite broadband communication system is composed of a satellite transmission domain subsystem, a satellite data domain subsystem and a satellite voice domain subsystem, the multiple components in the satellite broadband communication system respectively include multiple first components of the satellite transmission domain subsystem, multiple second components of the satellite data domain subsystem and multiple third components of the satellite voice domain subsystem. Figure 3 , Figure 3 This is a diagram of the satellite broadband communication system architecture provided by the present invention.

[0034] Furthermore, the control data acquisition processing unit collects data indicator sets of each first component from the satellite transmission domain subsystem, collects data indicator sets of each second component from the satellite data domain subsystem, and collects data indicator sets of each third component from the satellite voice domain subsystem, wherein each data indicator set includes multiple data indicators, and the collection of these data indicators can be achieved through various sensors, log files, network monitoring tools, etc. The collected data indicators should be as comprehensive as possible, because the more comprehensive the data indicators are, the more accurate the decision made by the intelligent agent will be.

[0035] The following is a detailed description of the data indicators that need to be collected for each component: 1. Satellite transmission domain subsystem

[0036] The first component includes: satellite main station, satellite terminal station, and communication satellite.

[0037] (1) Data indicators of the satellite master station include but are not limited to: Performance specifications: antenna aperture, power amplifier power, number of subnets, number of carriers, carrier frequency, outbound baseband signal processing capability, inbound baseband signal processing capability, outbound air interface bandwidth, inbound air interface bandwidth, maximum throughput, maximum number of access terminals, maximum number of access users; Data throughput: the total amount of data, uplink data, and downlink data processed by the master station; Connection status: number of online terminals, number of online users; Routing information: the routing path and routing efficiency of data packets; System load: The central processing unit (CPU) and memory usage of the master station to evaluate the system processing capacity; Fault and maintenance logs: record information such as system faults, maintenance activities, and system upgrades.

[0038] (2) Satellite terminal data indicators include but are not limited to: Performance specifications: antenna aperture, power amplifier power, maximum throughput, maximum number of connected users; User activity: online status of end-site users, number of active users, etc. Quality of Service (QoS): includes indicators such as delay, jitter and packet loss rate, reflecting user experience; Local traffic statistics: total traffic, peak traffic and average traffic generated by the terminal; Equipment status: health status, battery power, hardware status, etc. of the terminal equipment; User feedback: User feedback on service quality, including satisfaction surveys and complaint records.

[0039] (3) Data indicators of communication satellites include but are not limited to: Signal strength: Satellite transmitted and received signal strength and rain attenuation, used to evaluate communication quality; Bandwidth utilization: the use of satellite spectrum resources, including occupied bandwidth and idle bandwidth; Error rate: includes bit error rate (BER) and frame error rate (FER), reflecting signal quality; Channel state information: such as signal-to-noise ratio, Doppler frequency shift, etc. Satellite attitude and orbit parameters: satellite position, speed, attitude, etc.

[0040] 2. Satellite data domain subsystem

[0041] The second component includes: Broadband Remote Access Server (BRAS), Authentication, Authorization, and Accounting (AAA) 3A authentication system, log retention system, Deep Packet Inspection (DPI) system, and traffic control system.

[0042] (1) BRAS data indicators include but are not limited to: User session information: including user login time, session duration, amount of data used, etc.; Traffic statistics: total data traffic entering and leaving the BRAS, peak traffic, average traffic, etc.; Connection status: the number of currently active connections, the maximum number of connections, the frequency of connection establishment and disconnection, etc. Error and failure logs: record connection-related error information, such as authentication failure, connection timeout, etc.

[0043] (2) Data indicators of the 3A certification system include but are not limited to: Authentication attempts: the number of user authentication attempts, and the ratio of success and failure; Authorization information: a list of resources and services that the user is authorized to access; Billing data: billing information of users’ use of services, which may include time, data volume, etc.; User status: number of online users, user status changes (login, logout).

[0044] (3) Data indicators of the log retention system include but are not limited to: Access log: records the user's access time, accessed resources, operation results, etc.; System log: records system operation status, system errors, maintenance activities, etc.; Security event log: records security-related events such as intrusion attempts, abnormal access patterns, etc.

[0045] (4) The data indicators of the DPI system include but are not limited to: Packet details: Check the header and payload information of the packets passing through; Application identification: Identify which application or service the passing data traffic belongs to; Traffic classification: classify traffic according to application type, user, etc.; Quality of Service (QoS) parameters: Allocate bandwidth and service priority to different types of traffic.

[0046] (5) Data indicators of the traffic control system include but are not limited to: Bandwidth usage: the amount of bandwidth used by each user or service; Traffic quota: bandwidth quota and used quota of a user or service; Priority setting: Set the traffic priority according to business needs.

[0047] 3. Satellite Voice Domain Subsystem

[0048] The third component includes: Internet Protocol Multimedia Subsystem (IMS) core network, satellite broadband access service (AS), SMS center, PUSH&SIP server, security control server, and voice management and control platform.

[0049] (1) Data indicators of the IMS core network include but are not limited to: Session establishment success rate: the ratio of successfully established voice sessions to attempted sessions; Session duration: the average duration of a user's voice communication; Call attempt and call completion rates: the number of calls attempted versus the number of calls successfully completed; Quality of Service (QoS) parameters: including delay, jitter, and packet loss rate, which affect the quality of voice communications; Network throughput: The amount of data processed by the IMS core network, including voice and signaling.

[0050] (2) Satellite broadband AS data indicators include but are not limited to: Call parameters: call establishment success rate, call duration, call completion rate, call blocking rate; Voice quality: bit error rate, signal-to-noise ratio, echo and echo cancellation effect, network delay, jitter, and packet loss rate; Bandwidth usage: the amount of bandwidth used by voice communications and the peak bandwidth usage; System load: The CPU and memory usage of the voice center server, reflecting the system processing capacity.

[0051] (3) The data indicators of the SMS center include but are not limited to: SMS sent and received: the number of SMS messages sent and received; SMS success rate: the ratio of successfully sent and received SMS to attempted sent and received SMS; SMS delay: the average delay time from sending to receiving SMS; System load: CPU and memory usage of the SMS center, reflecting the system processing capacity.

[0052] (4) PUSH&SIP server data indicators include but are not limited to: PUSH notification volume: the number of PUSH notifications sent to users; SIP signaling load: The signaling load processed by the SIP server, including registration, invitation, response, etc. SIP session establishment time: the time from when a user initiates a call to when the call is established.

[0053] (5) Data indicators of the security control server include but are not limited to: Security events: recorded security warnings, intrusion attempts, and other security-related events; Encryption and decryption operations: frequency and efficiency of encryption and decryption of voice communications; Access Control List (ACL): Access control policy enforcement and violation attempts.

[0054] (6) Data indicators of the voice control platform include but are not limited to: Voice traffic monitoring: voice traffic transmitted through the platform; Service quality monitoring: QoS indicators of voice communication, such as delay and packet loss rate; User satisfaction: survey results on user satisfaction with voice services; Fault and Maintenance Logs: Logs of fault and maintenance activities for voice services.

[0055] At the same time, the data acquisition processing unit is controlled to collect the topological relationship between multiple components from the satellite broadband communication system, that is, the topological relationship between all first components, all second components and all third components.

[0056] Furthermore, the control data acquisition processing unit performs data preprocessing on the acquired topological relationship between the multiple components and the data indicator set of each component.

[0057] Among them, data preprocessing methods include but are not limited to data cleaning, data normalization and word vectorization. Data cleaning mainly removes outliers, missing values ​​and duplicate values ​​to ensure data quality; data normalization mainly normalizes feature data of different dimensions to a unified range to enable better model training; word vectorization mainly converts each word in the text into a vector to enable the model to better understand and process text information.

[0058] After the data preprocessing is completed, the control data acquisition processing unit constructs an adjacency matrix of multiple components based on the topological relationship between the multiple components to reflect the connection relationship between the various components in the satellite broadband communication system.

[0059] At the same time, the control data acquisition processing unit constructs an operation status matrix of multiple components based on the data indicator sets corresponding to the multiple components, so as to reflect the operation status of each component.

[0060] Furthermore, the intelligent agent reasoning unit is controlled to input the adjacency matrix and the operation state matrix of the multiple components into the optimal Actor network to obtain the spectrum bandwidth adjustment action output by the optimal Actor network.

[0061] Among them, the deep learning network architecture of deep deterministic policy gradient DDPG algorithm combined with graph convolution network GCN and deep neural network DNN can provide an efficient and accurate decision model for spectrum bandwidth adjustment, namely the optimal Actor network. The network is trained by DDPG algorithm, uses GCN to process the structural information in graph data, and uses DNN to improve the expression ability and performance of the model.

[0062] The above integrates the data and data processing operations of multiple subsystems such as the satellite transmission domain, satellite data domain and satellite voice domain, forming a cross-domain collaborative spectrum management strategy, which improves the resource utilization efficiency and flexibility of the overall system.

[0063] After obtaining the spectrum bandwidth adjustment action output by the optimal Actor network, the spectrum bandwidth adjustment unit is controlled to generate a spectrum bandwidth adjustment instruction based on the spectrum bandwidth adjustment action, and the spectrum bandwidth adjustment instruction is sent to the satellite broadband communication system.

[0064] After receiving the spectrum bandwidth adjustment instruction, the satellite broadband communication system parses the spectrum bandwidth adjustment instruction and obtains the spectrum bandwidth adjustment action. The spectrum bandwidth adjustment action is actually a continuous value. For example, the spectrum bandwidth adjustment action can be set to a floating point number between -1 and 1. -0.3 means that the spectrum bandwidth is recovered by 30%, and 0.5 means that the spectrum bandwidth is expanded by 50%. Therefore, the satellite broadband communication system can adjust the spectrum bandwidth accordingly according to the spectrum bandwidth adjustment action.

[0065] The spectrum bandwidth adaptive adjustment method provided by the present invention constructs an adjacency matrix and an operation status matrix of multiple components by collecting the topological relationship and data indicator set of multiple components from a satellite broadband communication system. Furthermore, since the optimal Actor network is an efficient and accurate decision model provided for spectrum bandwidth adjustment by using a deep learning network architecture combining a DDPG algorithm with GCN and DNN, the optimal Actor network can output accurate spectrum bandwidth adjustment actions according to the adjacency matrix and the operation status matrix, realize dynamic adjustment of satellite spectrum bandwidth according to real-time communication conditions, improve the flexibility of spectrum bandwidth adjustment, maximize spectrum utilization, and reduce resource waste. At the same time, the real-time monitoring and adaptive adjustment mechanism ensures that stable services can be provided even during peak hours, reduces communication congestion caused by uneven resource allocation, and improves user experience.

[0066] In some embodiments, based on step 102, the adjacency matrix of the plurality of components is constructed based on the topological relationship, and the specific process includes: Based on the topological relationship between the multiple components, determine whether there is a physical connection relationship between any two components; If there is a physical connection relationship between the two components, then determining the physical connection relationship value between the two components is 1; If there is no physical connection relationship between the two components, determining the physical connection relationship value between the two components to be 0; The physical relationship adjacency matrix is ​​constructed based on the physical connection relationship values ​​between the arbitrary two components.

[0067] It should be noted that the adjacency matrix of multiple components is A two-dimensional array represents the connection relationship between the various components in the satellite broadband communication system, where N is the total number of components, and the values ​​in the array represent whether there is an association between the components. If there is an association, the value is 1, and if there is no association, the value is 0.

[0068] The adjacency matrix includes at least a physical adjacency matrix, and the physical adjacency matrix reflects the physical connection relationship between components.

[0069] Specifically, based on the topological relationship between the multiple components, it is determined whether there is a physical connection relationship between any two components.

[0070] If there is a physical connection relationship between the two components, the physical connection relationship value between the two components is determined to be 1; if there is no physical connection relationship between the two components, the physical connection relationship value between the two components is determined to be 0.

[0071] Through the above determination method, the physical connection relationship value between any two components among the multiple components can be determined, and the physical relationship adjacency matrix of the multiple components can be constructed through these physical connection relationship values.

[0072] For example, if the satellite transmission domain consists of one or more communication satellites, satellite master stations, and satellite terminal stations, then the physical relationship adjacency matrix will contain the physical connection relationships between these components. For example: the satellite master station and the communication satellite are physically connected, and the physical connection relationship value between the satellite master station and the communication satellite is 1; the satellite terminal station and the communication satellite are physically connected, and the physical connection relationship value between the satellite terminal station and the communication satellite is 1; the satellite master station and the satellite terminal station are not physically connected, and the physical connection relationship value between the satellite master station and the satellite terminal station is 0. Referring to Table 1, you can understand the physical relationship adjacency matrix more intuitively.

[0073] Table 1 Physical relationship adjacency matrix

[0074] In addition, the adjacency matrix also includes at least a logical adjacency matrix, and the logical adjacency matrix reflects the logical connection relationship between components.

[0075] Specifically, based on the topological relationship between the multiple components, it is determined whether there is a logical connection relationship between any two components.

[0076] If there is a logical connection relationship between the two components, the logical connection relationship value between the two components is determined to be 1; if there is no logical connection relationship between the two components, the logical connection relationship value between the two components is determined to be 0.

[0077] Through the above determination method, the logical connection relationship value between any two components among the multiple components can be determined, and the logical relationship adjacency matrix of the multiple components can be constructed through these logical connection relationship values.

[0078] For example, the satellite main station and the communication satellite are logically connected, and the logical connection relationship value between the satellite main station and the communication satellite is 1; the satellite terminal station and the communication satellite are logically connected, and the logical connection relationship value between the satellite terminal station and the communication satellite is 1; the satellite main station and the satellite broadband AS are not logically connected, and the logical connection relationship value between the satellite main station and the satellite broadband AS is 0. Referring to Table 2, the logical relationship adjacency matrix can be more intuitively understood.

[0079] Table 2 Logical relationship adjacency matrix

[0080] The embodiment of the present invention utilizes the fully collected topological relationship of each component of the satellite communication system to construct an adjacency matrix of multiple components, and simultaneously utilizes the fully collected data indicators of each component of the satellite communication system to construct an operation status matrix of multiple components, and then comprehensively utilizes the adjacency matrix and the operation status matrix to train an intelligent agent, so that the spectrum adjustment decision is more accurate and avoids subjective judgment based on experience.

[0081] In some embodiments, based on step 103, the operation status matrix of the multiple components is constructed based on the multiple data indicator sets, and the specific process includes: Determine all data indicators included in the plurality of data indicator sets as target data indicators respectively; According to the data indicator set of each component, a target data indicator involved in the component is determined as a state indicator, and a target data indicator not involved in the component is determined as a non-state indicator; Determine the running state value between the component and the state indicator by using the data indicator state value corresponding to the state indicator, and set the running state value between the component and the non-state indicator to 0, so as to obtain the running state value between the component and each target data indicator; Constructing an operating state vector of the component according to an operating state value between the component and each target data indicator; Concatenate the running state vectors of each component into a running state matrix.

[0082] It should be noted that the operating state matrix is The two-dimensional array of the operating status matrix contains the operating status information of each component, where N is the total number of components, M is the total number of quantifiable data indicators collected by all components, which is considered to be the total number of target data indicators, and the values ​​in the array represent the data indicator status values ​​of the components.

[0083] Specifically, for ease of understanding, all data indicators included in the multiple data indicator sets are respectively determined as target data indicators. It should be noted that if there are duplicate target data indicators, only one duplicate item is retained.

[0084] According to the data indicator set of each component, it is determined whether each target data indicator is involved in the component, the target data indicator involved in the component is determined as a state indicator, and the target data indicator not involved in the component is determined as a non-state indicator.

[0085] Furthermore, the data indicator status value corresponding to the status indicator is used to determine the operating status value between the component and the status indicator, and the operating status value between the component and the non-status indicator is set to 0. That is to say, when the operating status value between the component and each status indicator and the operating status value between the component and each non-status indicator are determined, the operating status value between the component and each target data indicator can be obtained.

[0086] Furthermore, an operating state vector of the component is constructed according to the operating state value between the component and each target data indicator.

[0087] Therefore, through the above method, the operating state vector of each component can be determined.

[0088] Furthermore, the running state vector of each component is concatenated into a running state matrix.

[0089] For example, if the satellite transmission domain consists of one or more communication satellites, satellite master stations, and satellite terminal stations, then the operation status matrix will contain the operation status information of these components. For example, the data indicators of the communication satellite include the transmission signal strength, reception signal strength, rain attenuation, occupied bandwidth, and idle bandwidth, a total of 5 items; the data indicators of the satellite master station include antenna aperture, power amplifier power, number of subnets, number of carriers, and carrier frequency, a total of 5 items; the data indicators of the satellite terminal station include antenna aperture, power amplifier power, maximum throughput, and maximum number of access users, a total of 4 items, then M is 12 (5+5+4-2, because the antenna aperture and power amplifier power are repeated twice, so 2 duplicate items are removed). It can be understood that the target data indicators of the components of the communication satellite, satellite master station, and satellite terminal station essentially include transmission signal strength, reception signal strength, rain attenuation, occupied bandwidth, idle bandwidth antenna aperture, power amplifier power, number of subnets, number of carriers, carrier frequency, maximum throughput, and maximum number of access users, a total of 12 items. The value of the operation status matrix represents the state value of the target data indicator corresponding to the component. If the component does not involve the target data indicator, the value is 0. Referring to Table 3, the operating state matrix can be understood more intuitively.

[0090] Table 3 Operation status matrix

[0091] The embodiment of the present invention utilizes the fully collected data indicators of each component of the satellite communication system to construct an operation status matrix of multiple components, and further utilizes the fully collected topological relationships of each component of the satellite communication system to construct an adjacency matrix of multiple components, and then comprehensively utilizes the adjacency matrix and the operation status matrix to train an intelligent agent, so that the spectrum adjustment decision is more accurate and avoids subjective judgment based on experience.

[0092] In some embodiments, the optimal Actor network is obtained by performing the following steps: Initialize the parameters of the Actor network and the Critic network; both the Actor network and the Critic network are constructed based on the GCN network architecture; Building a target Actor network based on the parameters of the Actor network, and building a target Critic network based on the parameters of the Critic network; Acquire current state information of the satellite broadband communication system; the current state information includes an adjacency matrix sample and an operation state matrix sample; Inputting the current state information into the Actor network to obtain a target action output by the Actor network for adjusting the spectrum bandwidth; Acquire the next moment state information of the satellite broadband communication system; the next moment state information is the state information of the satellite broadband communication system after executing the target action; Acquire first performance indicator information before the satellite broadband communication system performs the target action and second performance indicator information after the satellite broadband communication system performs the target action; Inputting the first performance indicator information and the second performance indicator information into a reward generation network to obtain an instant reward value output by the reward generation network; Generate an experience sample based on the current state information, the target action, the immediate reward value and the next moment state information; The next moment state information is updated as the current state information, and the step of inputting the current state information into the Actor network is iteratively executed until a preset number of iterations is reached, a plurality of experience samples are obtained, and the plurality of experience samples are stored in an experience playback buffer; Based on the experience replay buffer, the Actor network and the Critic network are trained to obtain an optimal Actor network.

[0093] The following content describes how the agent training unit uses the data indicator samples and topological relationship samples collected by the data acquisition and processing unit to train the agent. This process uses the DDPG algorithm combined with the deep learning network architecture of GCN and DNN to solve problems with complex structures and dynamic changes. It can capture the topological structure and dynamic changes of the data, thereby providing more accurate prediction and decision support. The following content can be combined with Figure 4 , Figure 4 It is a schematic diagram of the intelligent agent training process provided by the present invention.

[0094] It should be noted that the deep learning network architecture of the DDPG algorithm combined with GCN and DNN can be referenced Figure 5 , Figure 5 This is a deep learning network architecture diagram provided by the present invention. Figure 5 The three deep learning networks related to the agent are the reward generation network, the actor network, and the critic network. The reward generation network uses two parallel deep neural networks (DNNs), which input the system performance indicator information before and after the spectrum bandwidth adjustment, respectively, and output the evaluation scores before and after the spectrum bandwidth adjustment. The difference between the evaluation scores is output as the immediate reward value of the spectrum bandwidth adjustment action. The actor network uses GCN (graph convolutional network), inputs adjacency matrix samples and operation status matrix samples, and outputs spectrum bandwidth adjustment actions. The critic network uses GCN (graph convolutional network), inputs adjacency matrix samples, operation status matrix samples and spectrum bandwidth adjustment actions, and outputs the Q value of the spectrum bandwidth adjustment action.

[0095] Before starting to train the Actor network and the Critic network, you need to train and deploy the reward generation network. The following describes the process of training and deploying the reward generation network.

[0096] The reward generation network predicts the instantaneous reward value of the spectrum bandwidth adjustment action based on the performance indicator information of the satellite broadband communication system before and after the adjustment action. The instantaneous reward is an important feedback mechanism that reflects the impact of the spectrum bandwidth adjustment action on the system performance, such as operation stability and spectrum utilization. The intelligent agent adjusts parameters according to the reward during training to optimize performance.

[0097] 1. Training the Reward Generation Network

[0098] The DNN is trained using the historical performance indicator information of the satellite broadband communication system. The DNN can output the evaluation score of the system operation quality by learning the potential rules and correlations in the historical multi-dimensional data. The DNN training process is as follows: (1) Data collection

[0099] The historical performance indicator information of the satellite broadband communication system is collected as a training data set, including but not limited to: throughput, connection status, system load, quality of service (QoS), spectrum utilization, error rate, user satisfaction, etc.

[0100] (2) Data preprocessing

[0101] Data cleaning: remove outliers, missing values ​​and duplicate values ​​to ensure data quality; Data normalization: normalize feature data to a uniform range to facilitate model training; Data labeling: Let experts with domain knowledge evaluate and score the training data as labels for the training data; Data splitting: Divide the total dataset into training, validation, and test sets to facilitate model training, validation, and testing.

[0102] (3) Network training

[0103] Initialize the weights and biases of the DNN; Use the training set to perform forward propagation and back propagation calculations on the network model, and update the network model weights and bias items; After each iteration or a certain number of training batches, use the validation set to evaluate the performance of the network model to prevent overfitting and adjust hyperparameters; After training, use the test set to evaluate the generalization ability of the network model on unseen data; When the network model performs best on the validation set, training is stopped and the network model weights are saved for practical application.

[0104] 2. Deploy the Reward Generation Network

[0105] Deploy the trained reward generation network model to the agent training unit.

[0106] After the reward generation network is deployed, the training of the Actor network and the Critic network can be assisted. The following describes the training process of the Actor network and the Critic network.

[0107] Before starting to train the agent, initialize the weights and bias parameters of the Actor network and Critic network by random initialization or using the parameters of the pre-trained model, and then copy the parameters of the Actor network and the Critic network to the target Actor network and the target Critic network respectively to construct the target Actor network and the target Critic network.

[0108] Furthermore, the Actor network is used to interact with the satellite broadband communication system. The satellite broadband communication system inputs the current state information s (state information, i.e., adjacency matrix samples and operation state matrix samples) to the Actor network, and the Actor network generates an initial action a'. In order to ensure a certain level of exploration, the initial action a' is superimposed with noise to generate a target action a (the target action is the spectrum bandwidth adjustment action). Further, based on the target action a, a spectrum bandwidth adjustment instruction is generated and sent to the satellite broadband communication system.

[0109] Furthermore, the satellite broadband communication system executes the target action a and returns the next moment state information s' to the Actor network.

[0110] At the same time, the satellite broadband communication system inputs the first performance indicator information and the second performance indicator information before and after adjusting the spectrum bandwidth into the reward generation network. The reward generation network predicts the immediate reward value r of the target action a and outputs it to the Actor network.

[0111] During each interaction, the current state information s, the target action a, the immediate reward value r and the next moment state information s' are combined into an experience sample, which is expressed as a four-tuple (s, a, r, s'), which means that when the current state information is s, action a is taken, and the immediate reward value r and the next moment state information s' are obtained.

[0112] Repeat the above process of generating experience samples until the preset number of iterations is reached to obtain multiple experience samples. The experience samples generated by multiple interactions are stored in the experience playback buffer as the data set for training the Actor network and the Critic network.

[0113] The embodiment of the present invention can further train the Actor network in the above manner to solve the optimal Actor network. The optimal Actor network can quickly respond to new business needs and sudden communication needs without manual intervention, thereby enhancing the adaptability of the system to market changes and technological advances, effectively enhancing flexibility and adaptability, and at the same time, reducing the frequency of manual intervention through automated and intelligent adjustment management, reducing operation and maintenance costs, and improving operation and maintenance efficiency.

[0114] In the above steps, the first performance indicator information and the second performance indicator information are input into the reward generation network to obtain the instant reward value output by the reward generation network. The specific process includes: Inputting the first performance indicator information into the first DNN to obtain a first evaluation score output by the first DNN; Inputting the second performance indicator information into the second DNN to obtain a second evaluation score output by the second DNN; The first evaluation score and the second evaluation score are input into a difference calculation layer to obtain an immediate reward value output by the difference calculation layer.

[0115] It should be noted that the reward generation network uses two parallel deep neural networks. For the convenience of description, these two parallel deep neural networks are divided into a first DNN and a second DNN.

[0116] Specifically, the first performance indicator information of the satellite broadband communication system before adjustment is input into the first DNN to obtain a first evaluation score output by the first DNN.

[0117] At the same time, the second performance indicator information adjusted by the satellite broadband communication system is input into the second DNN to obtain a second evaluation score output by the second DNN.

[0118] Furthermore, the first evaluation score and the second evaluation score are input to the difference calculation layer, and the difference calculation layer performs a difference calculation on the first evaluation score and the second evaluation score, and outputs the difference calculation result as the immediate reward value of the target action a.

[0119] For example, the evaluation score can be set to a value between 0 and 10. If the first evaluation score before the spectrum bandwidth adjustment is 6, and the second evaluation score after the spectrum bandwidth adjustment is 8, then the immediate reward value r of the target action a is 2 (8-6=2, which is a positive reward), and the agent is rewarded for making the correct adjustment; if the first evaluation score before the spectrum bandwidth adjustment is 7, and the second evaluation score after the spectrum bandwidth adjustment is 4, then the immediate reward value r of the target action a is -3 (4-7=-3, which is a negative reward), and the agent is punished for making the wrong adjustment.

[0120] The embodiment of the present invention predicts the instant reward value of the spectrum bandwidth adjustment action through a reward generation network, provides key feedback for the training and decision-making of the intelligent agent, thereby realizing autonomous learning and optimization of the intelligent agent and improving the intelligence level of spectrum management.

[0121] In the above steps, the Actor network and the Critic network are trained based on the experience replay buffer to obtain the optimal Actor network. The specific process includes: Randomly taking out experience samples from the experience playback buffer; Inputting the current state information and the target action in the experience sample into the Critic network to obtain a first current Q value output by the Critic network; Inputting the next moment state information in the experience sample into the target Actor network to obtain the next moment execution action output by the target Actor network; Input the next moment state information and the next moment execution action into the target Critic network to obtain the next moment Q value output by the target Critic network; Determine a label value of the first current Q value based on the next moment Q value; Based on the first current Q value and its corresponding label value, training the Critic network to obtain a trained Critic network; Based on the experience replay buffer and the trained Critic network, the Actor network is trained to obtain an optimal Actor network.

[0122] It should be noted that the purpose of the Critic network is to output a Q value based on the state and action. The more accurate the Q value is, the better the network is trained.

[0123] Specifically, an experience sample is randomly taken out from the experience playback buffer. The following first describes taking out an experience sample (s, a, r, s') as an example.

[0124] The current state information s and target action a in (s,a,r,s') are input into the Critic network to obtain the first current Q value Q(s,a) output by the Critic network, which can be considered as the predicted value.

[0125] Furthermore, the next moment state information s' in (s, a, r, s') is first input into the target Actor network to obtain the next moment execution action a' output by the target Actor network, and then the next moment state information s' and the next moment execution action a' are input into the target Critic network to obtain the next moment Q value Q(s', a') output by the target Critic network.

[0126] Based on the next moment Q value Q(s',a'), determine the label value of the first current Q value Q(s,a), that is, , which can be considered as the actual value Yi.

[0127] By using the above method, an experience sample is continuously randomly taken out from the experience playback buffer, and the above steps are executed to obtain a training data set of the first current Q value and its label value.

[0128] Similar to supervised learning, the loss function is defined as the first current Q value Q(s,a) and the label value of the first current Q value The mean squared error (MSE) of the Critic network is trained using an optimization algorithm (such as gradient descent) to update all parameters θ of the Critic network so that the first current Q value Q(s,a) output by the Critic network is as close as possible to the label value of the first current Q value .

[0129] It should be noted that during the training of the agent, the parameters of the critic network will be updated to the target critic network at regular intervals. A soft update method is used, that is, a learning rate τ (a very small number, such as 0.1 or 0.01) is introduced to perform a weighted average of the old and new parameters of the target network and then assign them to the target network.

[0130] Target Critic network parameter update process: .

[0131] The embodiment of the present invention can further train the Actor network in the above manner to solve the optimal Actor network. The optimal Actor network can quickly respond to new business needs and sudden communication needs without manual intervention, thereby enhancing the adaptability of the system to market changes and technological advances, effectively enhancing flexibility and adaptability, and at the same time, reducing the frequency of manual intervention through automated and intelligent adjustment management, reducing operation and maintenance costs, and improving operation and maintenance efficiency.

[0132] In the above steps, the Actor network is trained based on the experience replay buffer and the trained Critic network to obtain the optimal Actor network. The specific process includes: Randomly taking out experience samples from the experience playback buffer; Inputting the current state information in the experience sample into the Actor network to obtain a new action output by the Actor network; Inputting the new action and the current state information into the trained Critic network to obtain a second current Q value output by the trained Critic network; Determining a loss function of the Actor network based on the current state information, the new action, and the second current Q value; Based on the loss function of the Actor network, an optimization algorithm is used to train the Actor network, and the parameters of the Actor network are updated to obtain an optimal Actor network.

[0133] It should be noted that the purpose of the Actor network is to output the action that maximizes the Q value based on the state. The larger the Q value of this action, the better the network training is.

[0134] Specifically, an experience sample is randomly taken out from the experience playback buffer. The following first describes taking out an experience sample (s, a, r, s') as an example.

[0135] Input the current state information s in (s,a,r,s') into the Actor network and obtain the new action a-new output by the Actor network.

[0136] Furthermore, the current state information s and the new action a-new are input into the trained Critic network to obtain the second current Q value Q(s,a-new) output by the trained Critic network.

[0137] By using the above method, an experience sample is randomly taken from the experience replay buffer, and the above steps are executed to obtain the current state information and the training data set of the new action.

[0138] Define the loss function as -Q(s,a-new), use an optimization algorithm (such as gradient descent) to train the Actor network, update all parameters w of the Actor network, minimize -Q and maximize the Q value output.

[0139] It should be noted that during the training of the agent, the parameters of the Actor network will be updated to the target Actor network at regular intervals. A soft update method is used, that is, a learning rate τ (a very small number, such as 0.1 or 0.01) is introduced to perform a weighted average of the old and new parameters of the target network and then assign them to the target network.

[0140] Target Actor network parameter update process: .

[0141] Finally, after the agent training is completed, the optimal Actor network is solved, and the optimal Actor network model trained by the agent training unit is deployed to the agent reasoning unit. In actual use, a state of the satellite broadband communication system is input, and the Actor network will output an optimal spectrum bandwidth adjustment action.

[0142] In addition, during the adaptive adjustment of the spectrum bandwidth, new four-tuple sample data (s, a, r, s') will be generated, which will be stored in the experience playback buffer and the training data set will be updated. The agent regularly uses the latest data set to iteratively train the Actor network and the Critic network to help the agent adapt to changes in communication needs and continuously optimize its decision-making strategy.

[0143] The embodiment of the present invention can quickly respond to new business needs and sudden communication needs through the trained optimal Actor network without human intervention, thereby enhancing the adaptability of the system to market changes and technological advances, effectively enhancing flexibility and adaptability, and at the same time, reducing the frequency of human intervention through automated and intelligent adjustment management, reducing operation and maintenance costs, and improving operation and maintenance efficiency.

[0144] The spectrum bandwidth adaptive adjustment device provided by the present invention is described below. The spectrum bandwidth adaptive adjustment device described below and the spectrum bandwidth adaptive adjustment method described above can be referred to each other.

[0145] Reference Figure 6 , Figure 6 It is a structural schematic diagram of the spectrum bandwidth adaptive adjustment device provided by the present invention.

[0146] The spectrum bandwidth adaptive adjustment device comprises: The acquisition module 610 is used to acquire the topological relationship between multiple components in the satellite broadband communication system and the data indicator set of each component.

[0147] The first construction module 620 is used to construct an adjacency matrix of the multiple components based on the topological relationship.

[0148] The second construction module 630 is used to construct the operation status matrix of the multiple components based on the multiple data indicator sets.

[0149] The spectrum bandwidth adjustment action prediction module 640 is used to input the adjacency matrix and the operation state matrix into the optimal Actor network to obtain the spectrum bandwidth adjustment action output by the optimal Actor network; the optimal Actor network is determined by the deep deterministic policy gradient DDPG algorithm combined with the deep learning network architecture of the graph convolution network GCN and the deep neural network DNN.

[0150] The sending module 650 is used to send the spectrum bandwidth adjustment action to the satellite broadband communication system, so that the satellite broadband communication system performs the spectrum bandwidth adjustment action.

[0151] The spectrum bandwidth adaptive adjustment device provided by the present invention constructs an adjacency matrix and an operation status matrix of multiple components by collecting the topological relationship and data indicator set of multiple components from the satellite broadband communication system. Furthermore, since the optimal Actor network adopts the DDPG algorithm combined with the deep learning network architecture of GCN and DNN to provide an efficient and accurate decision model for spectrum bandwidth adjustment, the optimal Actor network can output accurate spectrum bandwidth adjustment actions according to the adjacency matrix and the operation status matrix, realize dynamic adjustment of satellite spectrum bandwidth according to real-time communication conditions, improve the flexibility of spectrum bandwidth adjustment, maximize spectrum utilization, and reduce resource waste. At the same time, the real-time monitoring and adaptive adjustment mechanism ensures that stable services can be provided even during peak hours, reduces communication congestion caused by uneven resource allocation, and improves user experience.

[0152] Furthermore, the first building module 620 is also used for: Based on the topological relationship between the multiple components, determine whether there is a physical connection relationship between any two components; If there is a physical connection relationship between the two components, then determining the physical connection relationship value between the two components is 1; If there is no physical connection relationship between the two components, determining the physical connection relationship value between the two components to be 0; The physical relationship adjacency matrix is ​​constructed based on the physical connection relationship values ​​between the arbitrary two components.

[0153] Furthermore, the second building module 630 is also used for: Determine all data indicators included in the plurality of data indicator sets as target data indicators respectively; According to the data indicator set of each component, a target data indicator involved in the component is determined as a state indicator, and a target data indicator not involved in the component is determined as a non-state indicator; Determine the running state value between the component and the state indicator by using the data indicator state value corresponding to the state indicator, and set the running state value between the component and the non-state indicator to 0, so as to obtain the running state value between the component and each target data indicator; Constructing an operating state vector of the component according to an operating state value between the component and each target data indicator; Concatenate the running state vectors of each component into a running state matrix.

[0154] Furthermore, the spectrum bandwidth adaptive adjustment device is also used for: Initialize the parameters of the Actor network and the Critic network; both the Actor network and the Critic network are constructed based on the GCN network architecture; Building a target Actor network based on the parameters of the Actor network, and building a target Critic network based on the parameters of the Critic network; Acquire current state information of the satellite broadband communication system; the current state information includes an adjacency matrix sample and an operation state matrix sample; Inputting the current state information into the Actor network to obtain a target action output by the Actor network for adjusting the spectrum bandwidth; Acquire the next moment state information of the satellite broadband communication system; the next moment state information is the state information of the satellite broadband communication system after executing the target action; Acquire first performance indicator information before the satellite broadband communication system performs the target action and second performance indicator information after the satellite broadband communication system performs the target action; Inputting the first performance indicator information and the second performance indicator information into a reward generation network to obtain an instant reward value output by the reward generation network; Generate an experience sample based on the current state information, the target action, the immediate reward value and the next moment state information; The next moment state information is updated as the current state information, and the step of inputting the current state information into the Actor network is iteratively executed until a preset number of iterations is reached, a plurality of experience samples are obtained, and the plurality of experience samples are stored in an experience playback buffer; Based on the experience replay buffer, the Actor network and the Critic network are trained to obtain an optimal Actor network.

[0155] Furthermore, the spectrum bandwidth adaptive adjustment device is also used for: Inputting the first performance indicator information into the first DNN to obtain a first evaluation score output by the first DNN; Inputting the second performance indicator information into the second DNN to obtain a second evaluation score output by the second DNN; The first evaluation score and the second evaluation score are input into a difference calculation layer to obtain an immediate reward value output by the difference calculation layer.

[0156] Furthermore, the spectrum bandwidth adaptive adjustment device is also used for: Randomly taking out experience samples from the experience playback buffer; Inputting the current state information and the target action in the experience sample into the Critic network to obtain a first current Q value output by the Critic network; Inputting the next moment state information in the experience sample into the target Actor network to obtain the next moment execution action output by the target Actor network; Input the next moment state information and the next moment execution action into the target Critic network to obtain the next moment Q value output by the target Critic network; Determine a label value of the first current Q value based on the next moment Q value; Based on the first current Q value and its corresponding label value, training the Critic network to obtain a trained Critic network; Based on the experience replay buffer and the trained Critic network, the Actor network is trained to obtain an optimal Actor network.

[0157] Furthermore, the spectrum bandwidth adaptive adjustment device is also used for: Randomly taking out experience samples from the experience playback buffer; Inputting the current state information in the experience sample into the Actor network to obtain a new action output by the Actor network; Inputting the new action and the current state information into the trained Critic network to obtain a second current Q value output by the trained Critic network; Determining a loss function of the Actor network based on the current state information, the new action, and the second current Q value; Based on the loss function of the Actor network, an optimization algorithm is used to train the Actor network, and the parameters of the Actor network are updated to obtain an optimal Actor network.

[0158] It should be noted that the spectrum bandwidth adaptive adjustment device provided by the present invention can execute the spectrum bandwidth adaptive adjustment method described in any of the above embodiments during specific operation, which will not be described in detail in this embodiment.

[0159] Figure 7 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the spectrum bandwidth adaptive adjustment method, which includes: obtaining the topological relationship between multiple components in the satellite broadband communication system and the data indicator set of each component; constructing the adjacency matrix of the multiple components based on the topological relationship; constructing the operation state matrix of the multiple components based on multiple data indicator sets; inputting the adjacency matrix and the operation state matrix into the optimal Actor network to obtain the spectrum bandwidth adjustment action output by the optimal Actor network; the optimal Actor network is determined by the deep deterministic policy gradient DDPG algorithm combined with the deep learning network architecture of the graph convolution network GCN and the deep neural network DNN; sending the spectrum bandwidth adjustment action to the satellite broadband communication system so that the satellite broadband communication system performs the spectrum bandwidth adjustment action.

[0160] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0161] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the spectrum bandwidth adaptive adjustment method provided by the above-mentioned embodiments, and the method includes: obtaining the topological relationship between multiple components in a satellite broadband communication system and a data indicator set of each component; based on the topological relationship, constructing an adjacency matrix of the multiple components; based on multiple data indicator sets, constructing an operating status matrix of the multiple components; inputting the adjacency matrix and the operating status matrix into an optimal Actor network to obtain a spectrum bandwidth adjustment action output by the optimal Actor network; the optimal Actor network is determined by a deep deterministic policy gradient DDPG algorithm combined with a deep learning network architecture of a graph convolutional network GCN and a deep neural network DNN; and sending the spectrum bandwidth adjustment action to the satellite broadband communication system so that the satellite broadband communication system performs the spectrum bandwidth adjustment action.

[0162] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the spectrum bandwidth adaptive adjustment method provided in the above-mentioned embodiments, the method comprising: obtaining the topological relationship between multiple components in a satellite broadband communication system and a data indicator set of each component; based on the topological relationship, constructing an adjacency matrix of the multiple components; based on multiple data indicator sets, constructing an operating status matrix of the multiple components; inputting the adjacency matrix and the operating status matrix into an optimal Actor network to obtain a spectrum bandwidth adjustment action output by the optimal Actor network; the optimal Actor network is determined by a deep deterministic policy gradient DDPG algorithm combined with a deep learning network architecture of a graph convolutional network GCN and a deep neural network DNN; and sending the spectrum bandwidth adjustment action to the satellite broadband communication system so that the satellite broadband communication system performs the spectrum bandwidth adjustment action.

[0163] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those of ordinary skill in the art may understand and implement the present invention without creative effort.

[0164] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A spectrum bandwidth adaptive adjustment method, characterized in that: include: Obtain the topological relationship between multiple components in the satellite broadband communication system and the data indicator set of each component; Based on the topological relationship, construct an adjacency matrix of the plurality of components; Based on the multiple data indicator sets, construct an operation status matrix of the multiple components; Inputting the adjacency matrix and the operation state matrix into an optimal Actor network to obtain a spectrum bandwidth adjustment action output by the optimal Actor network; the optimal Actor network is determined by a deep deterministic policy gradient DDPG algorithm combined with a deep learning network architecture of a graph convolutional network GCN and a deep neural network DNN; The spectrum bandwidth adjustment action is sent to the satellite broadband communication system so that the satellite broadband communication system executes the spectrum bandwidth adjustment action.

2. The spectrum bandwidth adaptive adjustment method according to claim 1, characterized in that: The adjacency matrix at least includes a physical relationship adjacency matrix; and constructing the adjacency matrix of the plurality of components based on the topological relationship includes: Based on the topological relationship between the multiple components, determine whether there is a physical connection relationship between any two components; If there is a physical connection relationship between the two components, then determining the physical connection relationship value between the two components is 1; If there is no physical connection relationship between the two components, determining the physical connection relationship value between the two components to be 0; The physical relationship adjacency matrix is ​​constructed based on the physical connection relationship values ​​between the arbitrary two components.

3. The spectrum bandwidth adaptive adjustment method according to claim 1, characterized in that: The step of constructing the operation status matrix of the plurality of components based on the plurality of data indicator sets includes: Determine all data indicators included in the plurality of data indicator sets as target data indicators respectively; According to the data indicator set of each component, a target data indicator involved in the component is determined as a state indicator, and a target data indicator not involved in the component is determined as a non-state indicator; Determine the running state value between the component and the state indicator by using the data indicator state value corresponding to the state indicator, and set the running state value between the component and the non-state indicator to 0, so as to obtain the running state value between the component and each target data indicator; Constructing an operating state vector of the component according to an operating state value between the component and each target data indicator; Concatenate the running state vectors of each component into a running state matrix.

4. The spectrum bandwidth adaptive adjustment method according to claim 1, characterized in that: The optimal Actor network is obtained by performing the following steps: Initialize the parameters of the Actor network and the Critic network; both the Actor network and the Critic network are constructed based on the GCN network architecture; Building a target Actor network based on the parameters of the Actor network, and building a target Critic network based on the parameters of the Critic network; Acquire current state information of the satellite broadband communication system; the current state information includes an adjacency matrix sample and an operation state matrix sample; Inputting the current state information into the Actor network to obtain a target action output by the Actor network for adjusting the spectrum bandwidth; Acquire the next moment state information of the satellite broadband communication system; the next moment state information is the state information of the satellite broadband communication system after executing the target action; Acquire first performance indicator information before the satellite broadband communication system performs the target action and second performance indicator information after the satellite broadband communication system performs the target action; Inputting the first performance indicator information and the second performance indicator information into a reward generation network to obtain an instant reward value output by the reward generation network; Generate an experience sample based on the current state information, the target action, the immediate reward value and the next moment state information; The next moment state information is updated as the current state information, and the step of inputting the current state information into the Actor network is iteratively executed until a preset number of iterations is reached, a plurality of experience samples are obtained, and the plurality of experience samples are stored in an experience playback buffer; Based on the experience replay buffer, the Actor network and the Critic network are trained to obtain an optimal Actor network.

5. The spectrum bandwidth adaptive adjustment method according to claim 4, characterized in that: The reward generation network includes a first DNN, a second DNN, and a difference calculation layer; the first performance indicator information and the second performance indicator information are input into the reward generation network to obtain an instant reward value output by the reward generation network, including: Inputting the first performance indicator information into the first DNN to obtain a first evaluation score output by the first DNN; Inputting the second performance indicator information into the second DNN to obtain a second evaluation score output by the second DNN; The first evaluation score and the second evaluation score are input into a difference calculation layer to obtain an immediate reward value output by the difference calculation layer.

6. The spectrum bandwidth adaptive adjustment method according to claim 4, characterized in that: The step of training the Actor network and the Critic network based on the experience replay buffer to obtain an optimal Actor network includes: Randomly taking out experience samples from the experience playback buffer; Inputting the current state information and the target action in the experience sample into the Critic network to obtain a first current Q value output by the Critic network; Inputting the next moment state information in the experience sample into the target Actor network to obtain the next moment execution action output by the target Actor network; Input the next moment state information and the next moment execution action into the target Critic network to obtain the next moment Q value output by the target Critic network; Determine a label value of the first current Q value based on the next moment Q value; Based on the first current Q value and its corresponding label value, training the Critic network to obtain a trained Critic network; Based on the experience replay buffer and the trained Critic network, the Actor network is trained to obtain an optimal Actor network.

7. The spectrum bandwidth adaptive adjustment method according to claim 6, characterized in that: The step of training the Actor network based on the experience replay buffer and the trained Critic network to obtain an optimal Actor network includes: Randomly taking out experience samples from the experience playback buffer; Inputting the current state information in the experience sample into the Actor network to obtain a new action output by the Actor network; Inputting the new action and the current state information into the trained Critic network to obtain a second current Q value output by the trained Critic network; Determining a loss function of the Actor network based on the current state information, the new action, and the second current Q value; Based on the loss function of the Actor network, an optimization algorithm is used to train the Actor network, and the parameters of the Actor network are updated to obtain an optimal Actor network.

8. The spectrum bandwidth adaptive adjustment method according to any one of claims 1 to 7, characterized in that: The satellite broadband communication system includes a satellite transmission domain subsystem, a satellite data domain subsystem and a satellite voice domain subsystem; The plurality of components of the satellite broadband communication system include a plurality of first components of the satellite transmission domain subsystem, a plurality of second components of the satellite data domain subsystem, and a plurality of third components of the satellite voice domain subsystem.

9. A spectrum bandwidth adaptive adjustment device, characterized in that: include: An acquisition module, used to acquire the topological relationship between multiple components in the satellite broadband communication system and a data indicator set of each component; A first construction module, configured to construct an adjacency matrix of the plurality of components based on the topological relationship; A second building module is used to build an operation status matrix of the multiple components based on multiple data indicator sets; A spectrum bandwidth adjustment action prediction module, used for inputting the adjacency matrix and the operation state matrix into an optimal Actor network to obtain a spectrum bandwidth adjustment action output by the optimal Actor network; the optimal Actor network is determined by a deep deterministic policy gradient DDPG algorithm combined with a deep learning network architecture of a graph convolutional network GCN and a deep neural network DNN; The sending module is used to send the spectrum bandwidth adjustment action to the satellite broadband communication system so that the satellite broadband communication system performs the spectrum bandwidth adjustment action.

10. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the spectrum bandwidth adaptive adjustment method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium, the computer-readable storage medium comprising a non-transitory computer-readable storage medium, on which a computer program is stored, characterized in that: When the computer program is executed by a processor, the steps of the spectrum bandwidth adaptive adjustment method according to any one of claims 1 to 8 are implemented.

12. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, the steps of the spectrum bandwidth adaptive adjustment method according to any one of claims 1 to 8 are implemented.

Images