Method for allocating bandwidth resources of satellite communication network and related devices

Abstract

The present application provides a method for allocating bandwidth resources in a satellite communication network and related equipment. The method takes into account the importance of samples in different storage areas and distinguishes the importance of samples in different storage areas to improve the training efficiency and accuracy of a deep reinforcement learning model. The method also uses combined target training samples as input for deep reinforcement learning model training. Since training experience samples corresponding to the current network environment state are added to the combined target training samples, the strategy learned by the deep reinforcement learning model can be avoided from being unsuitable for the current network environment, thereby improving the accuracy of the deep reinforcement learning model. The method allocates satellite communication network bandwidth resources to the satellite-to-ground network nodes corresponding to the maximum state-action value according to the target network bandwidth occupancy value, thereby avoiding the problem that fixed bandwidth allocation resources cannot adapt to the dynamic changes of the satellite network and the flexible needs of different communication services, thereby ensuring the full utilization and performance of bandwidth resources.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a method for allocating bandwidth resources of a satellite communication network and related equipment. Background Art

[0002] In the context of power system operation, satellite communications can provide wide-area coverage, enabling remote power plants and distributed energy facilities to seamlessly access the power system and transmit key information such as energy production in real time. Because satellite converged networks have limited bandwidth resources and need to meet the diverse communication needs of various services, the allocation of bandwidth resources requires more refined and rigorous requirements. For satellite converged networks, better bandwidth resource allocation can significantly improve the efficiency and utilization of communication systems, optimize network performance, and enhance the service experience.

[0003] However, the current traditional network bandwidth resource allocation strategy is fixed bandwidth allocation resources, which cannot adapt to the dynamic changes of satellite networks and the flexible needs of different communication services, and easily leads to bandwidth resource waste and poor performance. Summary of the Invention

[0004] In view of this, the purpose of this application is to propose a method for allocating bandwidth resources of a satellite communication network and related equipment to solve or partially solve the above technical problems.

[0005] Based on the above objectives, the first aspect of the present application provides a method for allocating bandwidth resources of a satellite communication network, comprising:

[0006] Determining a first priority of each training experience sample stored in a preset first storage area, and a second priority of each training experience sample stored in a preset second storage area, wherein the training experience sample represents an execution parameter of a communication service;

[0007] Selecting a training experience sample from each training experience sample in the first storage area according to the first priority, and selecting a training experience sample from each training experience sample in the second storage area according to the second priority;

[0008] Obtaining a training experience sample corresponding to the current network environment state, combining the training experience sample in the first storage area, the training experience sample selected in the second storage area, and the training experience sample corresponding to the current network environment state to obtain a combined target training sample;

[0009] Using the combined target training samples to train a pre-built deep reinforcement learning model to obtain a trained deep reinforcement learning model;

[0010] Acquire real-time network environment status information, and based on the real-time network environment status information, use the trained deep reinforcement learning model to determine the state action value obtained by selecting each satellite-to-ground network node to perform the communication service;

[0011] Determine the maximum state action value among the state action values corresponding to the respective satellite-to-ground network nodes, obtain a target network bandwidth occupancy value when the satellite-to-ground network node corresponding to the maximum state action value performs a communication service, and allocate satellite communication network bandwidth resources to the satellite-to-ground network node corresponding to the maximum state action value according to the target network bandwidth occupancy value.

[0012] Optionally, before determining the first priority of each training experience sample stored in the preset first storage area and the second priority of each training experience sample stored in the preset second storage area, the method further includes:

[0013] Obtaining a current transmission demand state of any user terminal among a plurality of user terminals;

[0014] selecting a target satellite-ground network node from a plurality of satellite-ground network nodes to execute a communication service based on the current transmission demand state, and obtaining a network bandwidth limit value of a transmission channel between any user terminal and the target satellite-ground network node, a network bandwidth occupancy value when executing the communication service, and a next transmission demand state of the current transmission demand state;

[0015] Determining a reward value using the network bandwidth limit and the network bandwidth occupancy value;

[0016] storing, according to the reward value, the training experience sample formed by combining the current transmission demand state, the target satellite-to-ground network node, the reward value, and the next transmission demand state in the first storage area or a preset third storage area;

[0017] Using the training experience samples stored in the first storage area to train the pre-built long short-term memory network model to obtain a trained long short-term memory network model;

[0018] The training experience samples stored in the third storage area are input into the trained long short-term memory network model for prediction to obtain predicted experience samples, and the predicted experience samples are stored in the second storage area.

[0019] Optionally, determining the reward value by using the network bandwidth limit and the network bandwidth occupancy value includes:

[0020] Obtaining the number of nodes in the satellite-to-ground network;

[0021] Performing a product processing on the number of satellite-to-ground network nodes and the network bandwidth limit to obtain a first product processing result;

[0022] Perform ratio processing on the network bandwidth occupancy value and the first product processing result to obtain the reward value.

[0023] Optionally, storing the training experience sample formed by combining the current transmission demand state, the target satellite-to-ground network node, the reward value, and the next transmission demand state in the first storage area or a preset third storage area according to the reward value includes:

[0024] Determine whether the reward value is greater than or equal to a preset reward value threshold, and obtain a determination result;

[0025] In response to the judgment result being yes, the training experience sample formed by combining the current transmission demand state, the target satellite-ground network node, the reward value, and the next transmission demand state is stored in the first storage area; or

[0026] In response to the judgment result being no, the training experience sample formed by combining the current transmission demand state, the target satellite-ground network node, the reward value, and the next transmission demand state is stored in the third storage area.

[0027] Optionally, determining the first priority of each training experience sample stored in the preset first storage area includes:

[0028] Determining a first cosine similarity between the selected training experience samples in the first storage area and the unselected training experience samples in the first storage area;

[0029] Obtaining the number of the selected training experience samples and a time difference error, where the time difference error is a difference between a reward value corresponding to a current transmission requirement state of any user terminal among the multiple user terminals and a reward value corresponding to a transmission requirement state next to the current transmission requirement state;

[0030] performing difference processing on the number of the selected training experience samples and the first cosine similarity to obtain a first difference processing result;

[0031] Performing ratio processing on the first difference processing result and the number of the selected training experience samples to obtain a first ratio processing result;

[0032] The first priority is obtained by multiplying the first ratio processing result by the absolute value of the time difference error.

[0033] Optionally, determining the second priority of each training experience sample stored in the preset second storage area includes:

[0034] Determining a second cosine similarity between the selected training experience samples in the second storage area and the unselected training experience samples in the second storage area;

[0035] Obtaining the number of the selected training experience samples and a time difference error, where the time difference error is a difference between a reward value corresponding to a current transmission requirement state of any user terminal among the multiple user terminals and a reward value corresponding to a transmission requirement state next to the current transmission requirement state;

[0036] performing difference processing on the number of the selected training experience samples and the second cosine similarity to obtain a second difference processing result;

[0037] performing ratio processing on the second difference processing result and the number of the selected training experience samples to obtain a second ratio processing result;

[0038] The second priority is obtained by multiplying the second ratio processing result by the absolute value of the time difference error.

[0039] Optionally, the using the combined target training samples to train a pre-built deep reinforcement learning model to obtain a trained deep reinforcement learning model includes:

[0040] Obtaining the true state action value corresponding to the combined target training sample;

[0041] Inputting the combined target training sample into the pre-built deep reinforcement learning model, and outputting the predicted state-action value through the pre-built deep reinforcement learning model;

[0042] A loss function is constructed based on the true state action value and the predicted state action value, the loss function is minimized using the ε-greedy strategy, and the pre-constructed deep reinforcement learning model is trained and adjusted according to the result of the minimization process to obtain a trained deep reinforcement learning model.

[0043] Optionally, the real-time network environment status information includes: current satellite-to-ground network status parameters;

[0044] The method of determining, based on the real-time network environment status information, the state action value obtained by selecting each satellite-to-ground network node to execute the communication service using the trained deep reinforcement learning model includes:

[0045] Obtaining an instant reward value and a discount factor obtained by executing any satellite-to-ground network node under the current satellite-to-ground network state parameters, and a current state action value obtained by selecting a satellite-to-ground network node corresponding to the current maximum state action value under the current satellite-to-ground network state parameters;

[0046] Performing a product process using the discount factor and the current state action value to obtain a second product processing result;

[0047] The second multiplication result and the instant reward value are summed to obtain a state action value obtained by selecting each satellite-ground network node to perform the communication service.

[0048] Based on the same inventive concept, a second aspect of the present application provides a device for allocating bandwidth resources of a satellite communication network, comprising:

[0049] a priority determination module configured to determine a first priority of each training experience sample stored in a preset first storage area, and a second priority of each training experience sample stored in a preset second storage area, wherein the training experience sample represents an execution parameter of a communication service;

[0050] a sample selection module configured to select a training experience sample from each training experience sample in the first storage area according to the first priority, and to select a training experience sample from each training experience sample in the second storage area according to the second priority;

[0051] a sample combination module configured to obtain a training experience sample corresponding to the current network environment state, and combine the training experience sample selected in the first storage area, the training experience sample selected in the second storage area, and the training experience sample corresponding to the current network environment state to obtain a combined target training sample;

[0052] a training module configured to train a pre-built deep reinforcement learning model using the combined target training samples to obtain a trained deep reinforcement learning model;

[0053] a value determination module configured to obtain real-time network environment status information and, based on the real-time network environment status information, use the trained deep reinforcement learning model to determine the state action value obtained by selecting each satellite-ground network node to perform the communication service;

[0054] The resource allocation module is configured to determine a maximum state action value among the state action values corresponding to the satellite-to-ground network nodes, obtain a target network bandwidth occupancy value when the satellite-to-ground network node corresponding to the maximum state action value performs a communication service, and allocate satellite communication network bandwidth resources to the satellite-to-ground network node corresponding to the maximum state action value according to the target network bandwidth occupancy value.

[0055] A third aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0056] As can be seen from the above, the satellite communication network bandwidth resource allocation method and related equipment provided by the present application fully consider the importance of samples in different storage areas by selecting training experience samples from the first storage area according to the first priority and selecting training experience samples from the second storage area according to the second priority, and distinguishing the importance of samples in different storage areas to improve the training efficiency and accuracy of the deep reinforcement learning model. The combined target training samples are formed as the input for deep reinforcement learning model training by combination. Since the combined target training samples are added with training experience samples corresponding to the current network environment state, the strategy learned by the deep reinforcement learning model can be avoided from being unsuitable for the current network environment, thereby improving the accuracy of the deep reinforcement learning model. The large state-action value determined by the deep reinforcement learning model is more accurate, and the satellite communication network bandwidth resources are allocated to the satellite-to-ground network nodes corresponding to the maximum state-action value according to the target network bandwidth occupancy value. This can avoid the problem that fixed bandwidth allocation resources cannot adapt to the dynamic changes of the satellite network and the flexible needs of different communication services, and can also ensure the full utilization and performance of bandwidth resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in this application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are merely embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0058] Figure 1 Flowchart of a method for allocating bandwidth resources of a satellite communication network according to an embodiment of the present application;

[0059] Figure 2 A schematic diagram of the allocation process of satellite communication network bandwidth resources according to an embodiment of the present application;

[0060] Figure 3This is a structural block diagram of a device for allocating bandwidth resources of a satellite communication network according to an embodiment of the present application;

[0061] Figure 4 A schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the objectives, technical solutions and advantages of this application more clear, this application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0063] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should have the usual meanings understood by people with ordinary skills in the field to which this application belongs. The "first", "second" and similar words used in the embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0064] It is understandable that before using the technical solutions of each embodiment of this application, the type, scope of use, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner, and the user's authorization will be obtained.

[0065] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. Thus, the user can independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operation of the technical solution of this application based on the prompt message.

[0066] As an optional but non-limiting implementation, in response to a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0067] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this application. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this application.

[0068] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0069] Bandwidth allocation in satellite-based converged networks refers to the process of effectively allocating and managing bandwidth to meet communication service needs. In the context of power system operations, satellite communications can provide wide-area coverage, enabling remote power plants and distributed energy facilities to seamlessly access the power system and transmit critical information such as energy production in real time. For satellite-based converged networks, optimal bandwidth resource allocation can significantly improve communication system efficiency and utilization, optimize network performance, and enhance the service experience.

[0070] Since satellite converged network bandwidth resources are limited and need to meet the communication needs of diverse services, the allocation requirements for bandwidth resources are more refined and strict.

[0071] For satellite converged networks, better bandwidth resource allocation methods can significantly improve the efficiency and utilization of communication systems, optimize network performance, and enhance the service experience. However, traditional network bandwidth resource allocation strategies use fixed bandwidth allocations, which cannot adapt to the dynamic changes of satellite networks and the flexible needs of different communication services, easily resulting in bandwidth waste and poor performance.

[0072] An embodiment of the present application provides a method for allocating bandwidth resources in a satellite communication network. By selecting training experience samples from a first storage area according to a first priority and selecting training experience samples from a second storage area according to a second priority, the importance of samples in different storage areas is fully considered, and the importance of samples in different storage areas is differentiated to improve the training efficiency and accuracy of a deep reinforcement learning model. Combined target training samples are formed by combination as input for deep reinforcement learning model training. Since the combined target training samples are added with training experience samples corresponding to the current network environment state, the strategy learned by the deep reinforcement learning model can be avoided from being unsuitable for the current network environment, thereby improving the accuracy of the deep reinforcement learning model. The large state-action value determined by the deep reinforcement learning model is more accurate, and the satellite communication network bandwidth resources are allocated to the satellite-to-ground network nodes corresponding to the maximum state-action value according to the target network bandwidth occupancy value. This can avoid the problem that fixed bandwidth allocation resources cannot adapt to the dynamic changes of the satellite network and the flexible needs of different communication services, and can also ensure the full utilization and performance of bandwidth resources.

[0073] like Figure 1 As shown, the method of this embodiment includes:

[0074] Step 101: Determine a first priority of each training experience sample stored in a preset first storage area and a second priority of each training experience sample stored in a preset second storage area, wherein the training experience samples represent execution parameters of a communication service.

[0075] In this step, the traditional method uses uniform random sampling to select experience samples for training the deep reinforcement learning model, so that the deep reinforcement learning model can converge faster, but it does not take into account the different useful information provided by different experience samples.

[0076] This application fully considers the importance of samples in different storage areas, distinguishes the importance of samples in different storage areas, and selects training experience samples from different storage areas by determining the first priority of each training experience sample stored in a preset first storage area and the second priority of each training experience sample stored in a preset second storage area, so as to improve the training efficiency and accuracy of the deep reinforcement learning model.

[0077] Step 102 : Selecting training experience samples from each training experience sample in the first storage area according to the first priority, and selecting training experience samples from each training experience sample in the second storage area according to the second priority.

[0078] In this step, training experience samples are selected from the first storage area based on the first priority, and training experience samples are selected from the second storage area based on the second priority, so that the selected experience samples contain as much useful information as possible, thereby reducing the number of states that the deep reinforcement learning model must explore or utilize, and helping the deep reinforcement learning model to converge quickly.

[0079] Step 103: Obtain training experience samples corresponding to the current network environment state, combine the training experience samples selected in the first storage area, the training experience samples selected in the second storage area, and the training experience samples corresponding to the current network environment state to obtain a combined target training sample.

[0080] In this step, since the training experience samples corresponding to the current network environment state are added to the combined target training samples, using the combined target training samples as the input for deep reinforcement learning model training can avoid the situation where the strategy learned by the deep reinforcement learning model is not suitable for the current network environment, thereby improving the accuracy of the deep reinforcement learning model.

[0081] Step 104: Use the combined target training samples to train the pre-built deep reinforcement learning model to obtain a trained deep reinforcement learning model.

[0082] In this step, the trained deep reinforcement learning model can be a deep Q-network (Deep Q-Leaning Network, DQN) model, an improved deep Q-network (Double DQN, DDQN) model and a deterministic policy gradient model using deep learning (Deep Deterministic Policy Gradient, DDPG) model, and the DDQN model is preferably used here.

[0083] The deep reinforcement learning model of this application combines a neural network with a reinforcement learning algorithm to solve decision-making problems with delayed rewards, so that the deep reinforcement learning model can autonomously learn from the environment and make decisions based on the learned experience.

[0084] Deep reinforcement learning is performed during training, allowing the system to learn and improve through interaction with the environment. By observing the state of the environment, it selects and executes appropriate actions, and then receives reward signals from the environment as feedback. The goal is to learn a strategy that maximizes long-term cumulative rewards. In adaptive network resource allocation, by modeling the network bandwidth resource allocation problem as a reinforcement learning problem, the optimal network bandwidth resource allocation strategy can be learned through interaction with the environment. Because deep reinforcement learning can model complex environments and tasks, and can automatically optimize strategies through learning, it is well suited to address the adaptive bandwidth resource allocation problem in satellite-ground converged networks.

[0085] In addition, the combined target training samples fully consider the importance of samples in different storage areas to improve the training efficiency and accuracy of the deep reinforcement learning model, and form combined target training samples through combination as the input of the deep reinforcement learning model training. Since the combined target training samples add training experience samples corresponding to the current network environment state, it can avoid the strategy learned by the deep reinforcement learning model from being unsuitable for the current network environment, thereby improving the accuracy of the deep reinforcement learning model.

[0086] Step 105: Acquire real-time network environment status information, and based on the real-time network environment status information, use the trained deep reinforcement learning model to determine the state action value obtained by selecting each satellite-to-ground network node to perform communication services.

[0087] In this step, since the trained deep reinforcement learning model can be applied to real-time network environments at different times, the trained deep reinforcement learning model can be used to make the state-action value obtained by selecting each satellite-to-ground network node to perform communication services more accurate.

[0088] Step 106: Determine the maximum state action value among the state action values corresponding to the satellite-to-ground network nodes, obtain the target network bandwidth occupancy value when the satellite-to-ground network node corresponding to the maximum state action value performs a communication service, and allocate satellite communication network bandwidth resources to the satellite-to-ground network node corresponding to the maximum state action value according to the target network bandwidth occupancy value.

[0089] In this step, since the large state-action value determined by the deep reinforcement learning model can be more accurate, the satellite communication network bandwidth resources are allocated to the satellite-to-ground network nodes corresponding to the maximum state-action value according to the target network bandwidth occupancy value to adapt to the dynamic changes of the satellite network and the flexible needs of different communication services. In addition, it can also ensure the full utilization and performance of bandwidth resources. Therefore, this application can solve the problem that traditional fixed bandwidth allocation resources cannot adapt to the dynamic changes of the satellite network and the flexible needs of different communication services.

[0090] Through the above scheme, by selecting training experience samples from the first storage area according to the first priority and selecting training experience samples from the second storage area according to the second priority, the importance of samples in different storage areas is fully considered, and the importance of samples in different storage areas is distinguished to improve the training efficiency and accuracy of the deep reinforcement learning model, and a combined target training sample is formed by combination as the input for deep reinforcement learning model training. Since the combined target training sample is added with the training experience sample corresponding to the current network environment state, it can avoid that the strategy learned by the deep reinforcement learning model is not suitable for the current network environment, thereby improving the accuracy of the deep reinforcement learning model. The large state-action value determined by the deep reinforcement learning model is more accurate, and then the satellite communication network bandwidth resources are allocated to the satellite-to-ground network node corresponding to the maximum state-action value according to the target network bandwidth occupancy value. This can avoid the problem that fixed bandwidth allocation resources cannot adapt to the dynamic changes of the satellite network and the flexible needs of different communication services, and can also ensure the full utilization and performance of bandwidth resources.

[0091] In some embodiments, before step 101, the method further includes:

[0092] Step A1: obtaining a current transmission requirement state of any user terminal among a plurality of user terminals.

[0093] Step A2: Select a target satellite-ground network node from a plurality of satellite-ground network nodes to execute a communication service based on the current transmission demand state, and obtain a network bandwidth limit of a transmission channel between any user terminal and the target satellite-ground network node, a network bandwidth occupancy value when executing the communication service, and a next transmission demand state after the current transmission demand state.

[0094] Step A3: determining a reward value using the network bandwidth limit and the network bandwidth occupancy value.

[0095] Step A4: storing the training experience sample formed by combining the current transmission demand state, the target satellite-ground network node, the reward value and the next transmission demand state into the first storage area or a preset third storage area according to the reward value.

[0096] Step A5: Use the training experience samples stored in the first storage area to train the pre-built long short-term memory network model to obtain a trained long short-term memory network model.

[0097] Step A6: input the training experience samples stored in the third storage area into the trained long short-term memory network model for prediction to obtain predicted experience samples, and store the predicted experience samples in the second storage area.

[0098] In the above scheme, the satellite fusion network is modeled to obtain the set N of user terminals, the set M of satellite-ground network nodes, and the action space A. n . At each time step, the current transmission demand state S is obtained from the network environment, and an action a is selected based on this state, which represents the selection of a satellite-ground network node for communication services. This action is executed in the satellite fusion network, and then the next transmission demand state s′ and the reward value for the decision are returned. At this time, the success of the action selection is judged based on the reward value. If successful, the four-tuple (s, a, r, s′) is placed in the successful playback buffer (i.e., the first storage area). If it does not work, it is placed in the failed playback buffer (i.e., the third storage area).

[0099] Specifically, the satellite fusion network under consideration is modeled as follows: the set of user terminals is N = {1, 2, ..., M}, and the set of satellite-ground network nodes is M = {1, 2, ..., M}. The action space of any user is A n = {0, 1, 2, ..., M}, that is, the user can select any node in the set M of satellite-ground network nodes to transmit task data. The state space s is represented by the current transmission demand state of each user, where 0 means that the current user has no transmission demand and 1 means that the current user has transmission demand. Every time user n transmits task data to satellite-ground network node m, it will occupy a certain bandwidth, which is represented by b nm , and the maximum bandwidth limit of each transmission channel (i.e. network bandwidth limit) is b max Each user terminal corresponds to at most one transmission task at a time.

[0100] After the user terminal selects the satellite-ground network node at time t to perform the communication service, it will obtain a universal reward (i.e. reward value) for the satellite-ground network r t (s t , a t ).

[0101] Based on the universal reward of the user's bandwidth allocation decision in each round, the success of the current round of actions is determined. The successful experience is stored in the corresponding successful experience buffer, and the failed experience is temporarily stored in the failed experience buffer.

[0102] When a user accumulates a certain amount of successful experience in bandwidth resource allocation decision-making in a satellite-ground converged network, the successful experience (i.e., the training experience samples stored in the first storage area) is used to train the LSTM network (i.e., the long short-term memory network model). The LSTM network is then used to process the user's failed experience (i.e., the training experience samples stored in the third storage area), and the processed experience (i.e., the predicted experience samples) is stored in the predicted experience buffer (i.e., the second storage area) to extract potential patterns and maximize the use of user experience data.

[0103] Specifically, the successful experience is used to train the LSTM network, so that the LSTM network can learn and predict the bandwidth allocation decision data in the satellite-ground fusion network environment.

[0104] The failure experience is passed to the network for prediction. The prediction results are stored in the prediction replay buffer for subsequent sampling.

[0105] Once a certain amount of valid experience has accumulated, samples are uniformly and randomly drawn from different experience pools according to preset weights. The sampled experience is fed into the DDQN network (a deep reinforcement learning model) to calculate Q values (state-action values), and the loss is calculated to update the network until the main network converges. Based on the resulting, continuously optimized DDQN target network, network bandwidth resource allocation decisions are optimized to achieve efficient and adaptive bandwidth allocation in the satellite-ground fusion network, improving the overall performance of the satellite-ground network.

[0106] Specifically, by mixing experience samples from different sources as input to the neural network (i.e., the deep reinforcement learning model), a mixed experience set E (i.e., the combined target training sample) is finally obtained. The sample sources include samples E0 from the successful experience pool (i.e., the training experience samples selected in the first storage area), samples E1 from the predicted experience pool (i.e., the training experience samples selected in the second storage area), and the latest samples E2 from the current satellite-to-ground network environment (i.e., the training experience samples corresponding to the current network environment state). The purpose of adding samples generated by the latest interaction with the environment is to add samples that are closest to the current state of the satellite-to-ground network for learning, to prevent the learned strategy from being unsuitable for the current network environment.

[0107] The dataset obtained by sampling the empirical mixture (i.e., the combined target training samples) is used to train the DDQN network to obtain bandwidth resource allocation decisions for the satellite-ground fusion network.

[0108] In some embodiments, step A3 includes:

[0109] Step A31, obtaining the number of nodes in the satellite-ground network.

[0110] Step A33: Perform product processing on the number of satellite-to-ground network nodes and the network bandwidth limit to obtain a first product processing result.

[0111] Step A33: performing ratio processing on the network bandwidth occupancy value and the first product processing result to obtain the reward value.

[0112] In the above scheme, the number of satellite-ground network nodes M and the maximum bandwidth of each transmission channel (i.e., network bandwidth limit b) are used. max Perform product processing to obtain the first product processing result Mb max .

[0113] Transmitting task data to satellite-ground network node a for user terminal n n Bandwidth occupied and the first product processing result Mb max Perform ratio processing to obtain the reward value r t (s t , a t ).

[0114] It can be expressed as follows:

[0115]

[0116] Taking into account the number of satellite-ground network nodes and the network bandwidth limit can make the determined reward value more accurate.

[0117] In some embodiments, step A4 includes:

[0118] Step A41: determine whether the reward value is greater than or equal to a preset reward value threshold, and obtain a determination result.

[0119] Step A42: In response to the judgment result being yes, the training experience sample formed by combining the current transmission demand state, the target satellite-ground network node, the reward value, and the next transmission demand state is stored in the first storage area. Or,

[0120] Step A43: In response to the judgment result being negative, the training experience sample formed by combining the current transmission demand state, the target satellite-ground network node, the reward value, and the next transmission demand state is stored in the third storage area.

[0121] In the above scheme, when the reward value is greater than or equal to the preset reward value threshold, it indicates that the action selection is successful, and the training experience sample formed by the combination of the current transmission demand state, the target satellite-ground network node, the reward value and the next transmission demand state is stored in the first storage area.

[0122] When the reward value is less than the preset reward threshold, it means that the action selection is ineffective, and the training experience sample formed by the current transmission demand state, the target satellite-ground network node, the reward value and the next transmission demand state is stored in the third storage area.

[0123] For example, the reward value threshold is 0.5. When the reward value is greater than or equal to 0.5, the four-tuple (s, a, r, s′) is stored in the successful playback buffer (i.e., the first storage area), and the four-tuple with a reward value less than 0.5 is stored in the failed playback buffer (i.e., the third storage area).

[0124] The four-tuple is a training experience sample formed by the combination of the current transmission demand state, the target satellite-ground network node, the reward value and the next transmission demand state.

[0125] The importance of samples can be differentiated based on the reward value, thereby improving sample utilization and training efficiency.

[0126] In some embodiments, in step 101, determining the first priority of each training experience sample stored in the preset first storage area includes:

[0127] Step B1: Determine a first cosine similarity between selected training experience samples in the first storage area and unselected training experience samples in the first storage area.

[0128] Step B3: Obtain the number of the selected training experience samples and the time difference error, where the time difference error is the difference between the reward value corresponding to the current transmission demand state of any user terminal among multiple user terminals and the reward value corresponding to the next transmission demand state of the current transmission demand state.

[0129] Step B3: performing difference processing on the number of the selected training experience samples and the first cosine similarity to obtain a first difference processing result.

[0130] Step B4: performing ratio processing on the first difference processing result and the number of the selected training experience samples to obtain a first ratio processing result.

[0131] Step B5: multiplying the first ratio processing result by the absolute value of the time difference error to obtain the first priority.

[0132] In the above scheme, the traditional DDQN uses uniform random sampling to select experience samples to train the network, which enables the network to converge faster, but does not take into account the different useful information provided by different experience samples in the experience replay pool.

[0133] This application proposes a priority sampling mechanism based on cosine similarity and time difference, so that the extracted experience samples contain as much useful information as possible, thereby reducing the number of states that the network must explore or utilize, and helping the network converge quickly. Define the first cosine similarity between the selected training experience samples in the first storage area and the unselected training experience samples in the first storage area as:

[0134]

[0135] Furthermore, the first priority of each experience sample is defined as:

[0136]

[0137] Where S i is the i-th experience tuple in the extracted experience sample (i.e., the selected training experience sample in the first storage area), n is the total number of extracted experience samples (i.e., the number of selected training experience samples), E t is the t-th experience sample in the experience samples to be extracted (i.e., the unselected training experience sample in the first storage area), δ t The empirical TD-err (time difference error) is the difference between the reward of the current state and the expected reward of the next state. Sampling is done sequentially and independently, with the highest-priority sample being drawn each time. The priority of the next sample is updated based on the current sampling result, and this step is repeated until a sufficient number of samples are drawn to update the DDQN network.

[0138] In some embodiments, in step 101, determining the second priority of each training experience sample stored in the preset second storage area includes:

[0139] Step C1: Determine a second cosine similarity between the selected training experience samples in the second storage area and the unselected training experience samples in the second storage area.

[0140] Step C2: Obtain the number of the selected training experience samples and the time difference error, where the time difference error is the difference between the reward value corresponding to the current transmission demand state of any user terminal among the multiple user terminals and the reward value corresponding to the next transmission demand state of the current transmission demand state.

[0141] Step C3: performing difference processing on the number of the selected training experience samples and the second cosine similarity to obtain a second difference processing result.

[0142] Step C4: performing ratio processing on the second difference processing result and the number of the selected training experience samples to obtain a second ratio processing result.

[0143] Step C5: multiplying the second ratio processing result by the absolute value of the time difference error to obtain the second priority.

[0144] In the above scheme, the traditional DDQN uses uniform random sampling to select experience samples to train the network, which enables the network to converge faster, but does not take into account the different useful information provided by different experience samples in the experience replay pool.

[0145] This application proposes a priority sampling mechanism based on cosine similarity and time difference, so that the extracted experience samples contain as much useful information as possible, thereby reducing the number of states that the network must explore or utilize, and helping the network converge quickly. Define the second cosine similarity between the selected training experience samples in the second storage area and the unselected training experience samples in the second storage area as:

[0146]

[0147] Furthermore, the second priority of each experience sample is defined as:

[0148]

[0149] Where S i is the i-th experience tuple in the extracted experience sample (i.e., the selected training experience sample in the second storage area), n is the total number of extracted experience samples (i.e., the number of selected training experience samples), E tis the t-th experience sample in the experience samples to be extracted (i.e., the unselected training experience sample in the second storage area), δ t The empirical TD-err (time difference error) is the difference between the reward of the current state and the expected reward of the next state. Sampling is done sequentially and independently, with the highest-priority sample being drawn each time. The priority of the next sample is updated based on the current sampling result, and this step is repeated until a sufficient number of samples are drawn to update the DDQN network.

[0150] In some embodiments, step 104 includes:

[0151] Step 1041: Obtain the real state action value corresponding to the combined target training sample.

[0152] Step 1042: Input the combined target training sample into the pre-built deep reinforcement learning model, and output the predicted state-action value through the pre-built deep reinforcement learning model.

[0153] Step 1043: construct a loss function based on the true state action value and the predicted state action value, minimize the loss function using the ε-greedy strategy, and train and adjust the pre-constructed deep reinforcement learning model based on the result of the minimization process to obtain a trained deep reinforcement learning model.

[0154] In the above scheme, the deep reinforcement learning model updates the weights through the gradient descent method to reduce the loss between the target Q value (i.e., the actual state action value) and the predicted Q value (i.e., the predicted state action value) to maximize the optimization of bandwidth allocation decisions.

[0155] Since the DDQN model has different degrees of learning about the environment in the early and late stages of the algorithm, the corresponding probabilities of exploration and utilization should also be different. When the algorithm obtains less environmental information in the early stage, a larger exploration probability should be adopted to obtain environmental information. In the later stage of the algorithm, a smaller exploration probability should be adopted to achieve the development of the best strategy. In order to achieve a balance between detection and utilization of the algorithm, this application adopts an ε adaptive adjustment mechanism, which uses the reward value obtained from the environment to decide whether to attenuate the exploration rate. Only when a certain reward threshold is crossed, it indicates that enough information has been learned from the environment to support better decision-making, so the detection probability can be reduced and the utilization probability can be increased, the value of ε will decrease, and the reward threshold will be increased at the same time. In addition, after each state transition, the exploration probability ε(s) of each state is calculated based on the difference in the Boltzmann distribution of the value.

[0156]

[0157]

[0158]

[0159] Where σ is a positive constant that determines the effect of the selected action on the detection probability of the relevant state, and δ is the inverse of the number of actions in state S, that is, threadhold is the reward threshold, λ increment is the reward threshold growth factor, which is a constant greater than 1.

[0160] Furthermore, the ε-greedy strategy is expressed as follows:

[0161]

[0162] Where p e is a random number between 0 and 1, and ε(s) (0 < ε(s) < 1) is the exploration probability calculated according to the above algorithm. The strategy randomly captures an action in the action space |A(s)| with probability ε(s) to avoid falling into local optimality.

[0163] By obtaining an adaptive bandwidth allocation decision-making scheme in a satellite-ground fusion network, bandwidth allocation decisions can be made in real time based on the current network status and predicted bandwidth demand. This enables efficient network bandwidth resource allocation in the satellite-ground fusion network, meeting the personalized needs of various services and improving the performance of the satellite-ground fusion network.

[0164] In some embodiments, the real-time network environment status information includes: current satellite-to-ground network status parameters.

[0165] In step 105, based on the real-time network environment status information, the state action value obtained by selecting each satellite-to-ground network node to perform the communication service is determined using the trained deep reinforcement learning model, including:

[0166] Step 1051: Obtain an instant reward value and a discount factor obtained by executing any satellite-ground network node under the current satellite-ground network state parameters, and obtain a current state action value obtained by selecting a satellite-ground network node corresponding to the current maximum state action value under the current satellite-ground network state parameters.

[0167] Step 1052: perform product processing on the discount factor and the current state action value to obtain a second product processing result.

[0168] Step 1053: Add the second multiplication result and the instant reward value to obtain a state action value obtained by selecting each satellite-ground network node to perform the communication service.

[0169] In the above scheme, the deep reinforcement learning model reduces overestimation by decomposing the maximum operation in the goal into action selection and action evaluation. At the same time, the calculation method of the target Q value in the deep Q-network is further improved to ensure that the Q value (i.e., the state action value) is more realistic.

[0170] The target Q value (i.e., state-action value) is finally expressed as:

[0171]

[0172] Where Y t is the target Q value, r t+1 is the instant reward value designed for the satellite-ground fusion network, which represents the current satellite-ground network state parameter (s t ) to perform satellite-to-ground network node selection action (a t ) is the immediate reward obtained after θ, and γ is the discount factor used to measure the importance of future reward values. t represents the current Q network parameter, that is, the network weight at the current time (t), Represents the target Q network parameters, used to calculate the target Q value. Indicates the given satellite fusion network state parameter (s t+1 ), select the action with the highest Q value (the current maximum state action value) (a t ), It indicates the current state action value obtained by selecting the satellite-ground network node corresponding to the current maximum state action value under the current satellite-ground network state parameters.

[0173] In some embodiments, as Figure 2 As shown, S1 selects and executes an action (i.e., selects a target satellite-ground network node to perform communication services) based on the current state (i.e., the current transmission demand state), obtains the next moment state (i.e., the next transmission demand state of the current transmission demand state) and the reward (i.e., the reward value), and stores the experience according to the reward.

[0174] For example, S1 selects and executes the satellite-ground network node selection action according to the current state, obtains the state at the next moment, and calculates the general reward (i.e., reward value) according to formula (1). The four-tuple (s, a, r, s′) with a reward value greater than 0.5 (i.e., the preset reward value threshold) is stored in the successful playback buffer (i.e., the first storage area), and the four-tuple with a reward value less than 0.5 is stored in the failed playback buffer (i.e., the third storage area).

[0175] Get the success experience matrix

[0176] Wherein, formula (1) is expressed as follows:

[0177]

[0178] S1. Use successful experience (i.e., training experience samples stored in the first storage area) to train LSTM (i.e., long short-term memory network model), and then use LSTM to process failed experience (i.e., training experience samples stored in the third storage area), and save the processed experience (i.e., predicted experience samples).

[0179] For example, in S2, when the number of experiences in the successful experience pool reaches the required number, the successful experiences are used to train the LSTM. Then, the LSTM is used to process the failed network bandwidth decision experiences to obtain the predicted experience matrix (i.e., predicted experience samples).

[0180] S3. Collect mixed experience sets (i.e., combined target training samples) from different experience pools and train the DDQN network (i.e., deep reinforcement learning model).

[0181] For example, S3 uses formula (2) and formula (3) to calculate the priority of samples in the successful experience pool (i.e., the first storage area) and the predicted experience pool (i.e., the second storage area), and first extracts high-priority experience samples until 500 playback experience samples are extracted to obtain the mixed experience matrix

[0182]

[0183] Wherein, formula (2) is expressed as follows:

[0184]

[0185] Formula (3) is expressed as follows:

[0186]

[0187] The DDQN network is trained using the mixed experience set and the latest experience set (the training experience sample corresponding to the current network environment state), and the ε-greedy strategy in formulas (5)-(8) is used to reduce the loss between the target Q value (i.e., the actual state action value) and the predicted Q value (i.e., the predicted state action value), where σ is 1 and λ is increment Take 1.1, the initial value of threadhold is 0.75, and the initial value of ε in each state is set to 0.5. Finally, the target Q value is obtained through formula (4): This target Q value is used to evaluate and select the action with the highest Q value (i.e., the satellite-ground network node that performs the communication service corresponding to the state action value with the highest value). This continuous action selection based on the target Q value ultimately results in a reasonable allocation of network bandwidth resources, thereby optimizing the performance of the satellite-ground converged network.

[0188] Wherein, formula (4) is expressed as follows:

[0189]

[0190] Formula (5) is expressed as follows:

[0191]

[0192] Formula (6) is expressed as follows:

[0193]

[0194] Formula (7) is expressed as follows:

[0195]

[0196] Formula (8) is expressed as follows:

[0197]

[0198] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied in a distributed scenario and performed by multiple devices working together. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the method.

[0199] It should be noted that the above description is limited to some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0200] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides a device for allocating bandwidth resources of a satellite communication network.

[0201] refer to Figure 3 , the satellite communication network bandwidth resource allocation device includes:

[0202] The priority determination module 301 is configured to determine a first priority of each training experience sample stored in a preset first storage area, and a second priority of each training experience sample stored in a preset second storage area, wherein the training experience sample represents an execution parameter of a communication service;

[0203] a sample selection module 302 configured to select a training experience sample from each training experience sample in the first storage area according to the first priority, and to select a training experience sample from each training experience sample in the second storage area according to the second priority;

[0204] The sample combination module 303 is configured to obtain a training experience sample corresponding to the current network environment state, and combine the training experience sample selected in the first storage area, the training experience sample selected in the second storage area, and the training experience sample corresponding to the current network environment state to obtain a combined target training sample;

[0205] A training module 304 is configured to train a pre-built deep reinforcement learning model using the combined target training samples to obtain a trained deep reinforcement learning model;

[0206] The value determination module 305 is configured to obtain real-time network environment status information and, based on the real-time network environment status information, use the trained deep reinforcement learning model to determine the state action value obtained by selecting each satellite-ground network node to perform the communication service;

[0207] The resource allocation module 306 is configured to determine the maximum state action value among the state action values corresponding to the satellite-to-ground network nodes, obtain a target network bandwidth occupancy value for the satellite-to-ground network node corresponding to the maximum state action value when performing a communication service, and allocate satellite communication network bandwidth resources to the satellite-to-ground network node corresponding to the maximum state action value according to the target network bandwidth occupancy value.

[0208] In some embodiments, the satellite communication network bandwidth resource allocation device further includes a storage module, and the storage module is specifically configured to:

[0209] A first acquiring unit is configured to acquire a current transmission demand state of any user terminal among a plurality of user terminals;

[0210] a second acquiring unit configured to select a target satellite-ground network node from a plurality of satellite-ground network nodes to execute a communication service based on the current transmission demand state, and acquire a network bandwidth limit value of a transmission channel between any user terminal and the target satellite-ground network node, a network bandwidth occupancy value when executing the communication service, and a next transmission demand state of the current transmission demand state;

[0211] a reward value determining unit, configured to determine a reward value by using the network bandwidth limit and the network bandwidth occupancy value;

[0212] A first storage unit is configured to store the training experience sample formed by combining the current transmission demand state, the target satellite-ground network node, the reward value, and the next transmission demand state into the first storage area or a preset third storage area according to the reward value;

[0213] a training unit configured to train a pre-built long short-term memory network model using the training experience samples stored in the first storage area to obtain a trained long short-term memory network model;

[0214] The second storage unit is configured to input the training experience samples stored in the third storage area into the trained long short-term memory network model for prediction, obtain predicted experience samples, and store the predicted experience samples in the second storage area.

[0215] In some embodiments, the reward value determination unit is specifically configured to:

[0216] Obtaining the number of nodes in the satellite-to-ground network;

[0217] Performing a product processing on the number of satellite-to-ground network nodes and the network bandwidth limit to obtain a first product processing result;

[0218] Perform ratio processing on the network bandwidth occupancy value and the first product processing result to obtain the reward value.

[0219] In some embodiments, the first storage unit is specifically configured to:

[0220] Determine whether the reward value is greater than or equal to a preset reward value threshold, and obtain a determination result;

[0221] In response to the judgment result being yes, the training experience sample formed by combining the current transmission demand state, the target satellite-ground network node, the reward value, and the next transmission demand state is stored in the first storage area; or

[0222] In response to the judgment result being no, the training experience sample formed by combining the current transmission demand state, the target satellite-ground network node, the reward value, and the next transmission demand state is stored in the third storage area.

[0223] In some embodiments, the priority determination module 301 is specifically configured to:

[0224] The first priority of each stored training experience sample includes:

[0225] Determining a first cosine similarity between the selected training experience samples in the first storage area and the unselected training experience samples in the first storage area;

[0226] Obtaining the number of the selected training experience samples and a time difference error, where the time difference error is a difference between a reward value corresponding to a current transmission requirement state of any user terminal among the multiple user terminals and a reward value corresponding to a transmission requirement state next to the current transmission requirement state;

[0227] performing difference processing on the number of the selected training experience samples and the first cosine similarity to obtain a first difference processing result;

[0228] Performing ratio processing on the first difference processing result and the number of the selected training experience samples to obtain a first ratio processing result;

[0229] The first priority is obtained by multiplying the first ratio processing result by the absolute value of the time difference error.

[0230] In some embodiments, the priority determination module 301 is specifically configured to:

[0231] Determining a second cosine similarity between the selected training experience samples in the second storage area and the unselected training experience samples in the second storage area;

[0232] Obtaining the number of the selected training experience samples and a time difference error, where the time difference error is a difference between a reward value corresponding to a current transmission requirement state of any user terminal among the multiple user terminals and a reward value corresponding to a transmission requirement state next to the current transmission requirement state;

[0233] performing difference processing on the number of the selected training experience samples and the second cosine similarity to obtain a second difference processing result;

[0234] performing ratio processing on the second difference processing result and the number of the selected training experience samples to obtain a second ratio processing result;

[0235] The second priority is obtained by multiplying the second ratio processing result by the absolute value of the time difference error.

[0236] In some embodiments, the training module 304 is specifically configured to:

[0237] Obtaining the true state action value corresponding to the combined target training sample;

[0238] Inputting the combined target training sample into the pre-built deep reinforcement learning model, and outputting the predicted state-action value through the pre-built deep reinforcement learning model;

[0239] A loss function is constructed based on the true state action value and the predicted state action value, the loss function is minimized using the ε-greedy strategy, and the pre-constructed deep reinforcement learning model is trained and adjusted according to the result of the minimization process to obtain a trained deep reinforcement learning model.

[0240] In some embodiments, the real-time network environment status information includes: current satellite-to-ground network status parameters;

[0241] The value determination module 305 is specifically configured to:

[0242] Obtaining an instant reward value and a discount factor obtained by executing any satellite-to-ground network node under the current satellite-to-ground network state parameters, and a current state action value obtained by selecting a satellite-to-ground network node corresponding to the current maximum state action value under the current satellite-to-ground network state parameters;

[0243] Performing a product process using the discount factor and the current state action value to obtain a second product processing result;

[0244] The second multiplication result and the instant reward value are summed to obtain a state action value obtained by selecting each satellite-ground network node to perform the communication service.

[0245] For the convenience of description, the above devices are described as being divided into various modules according to their functions. Of course, when implementing this application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0246] The apparatus of the above embodiment is used to implement the corresponding satellite communication network bandwidth resource allocation method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0247] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, the method for allocating satellite communication network bandwidth resources described in any of the above embodiments is implemented.

[0248] Figure 4A more specific hardware structure diagram of an electronic device provided in this embodiment is shown. The device may include: a processor 401, a memory 402, an input / output interface 403, a communication interface 404, and a bus 405. The processor 401, the memory 402, the input / output interface 403, and the communication interface 404 are communicatively connected to each other within the device via the bus 405.

[0249] The processor 401 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0250] The memory 402 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 402 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 402 and is called and executed by the processor 401.

[0251] The input / output interface 403 is used to connect to input / output modules to implement information input and output. The input / output modules can be configured as components within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0252] The communication interface 404 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WIFI, Bluetooth, etc.).

[0253] The bus 405 comprises a pathway for transmitting information between various components of the device (eg, the processor 401 , the memory 402 , the input / output interface 403 , and the communication interface 404 ).

[0254] It should be noted that although the above device only shows the processor 401, the memory 402, the input / output interface 403, the communication interface 404, and the bus 405, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0255] The electronic device of the above embodiment is used to implement the corresponding satellite communication network bandwidth resource allocation method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0256] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the satellite communication network bandwidth resource allocation method as described in any of the above embodiments.

[0257] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0258] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the satellite communication network bandwidth resource allocation method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0259] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. Within the scope of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0260] In addition, for simplicity of description and discussion, and in order not to make the embodiment of the application difficult to understand, the known power supply / ground connection with integrated circuit (IC) chip and other components may or may not be shown in the accompanying drawings provided. In addition, the device can be shown in the form of a block diagram to avoid making the embodiment of the application difficult to understand, and this also takes into account the following fact, that is, the details of the embodiment of these block diagram devices are highly dependent on the platform to be implemented in the embodiment of the application (that is, these details should be fully within the scope of understanding of those skilled in the art). When specific details (for example, circuit) are set forth to describe exemplary embodiments of the application, it will be apparent to those skilled in the art that the embodiment of the application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0261] Although the present invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may utilize the embodiments discussed.

[0262] The embodiments of the present application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of this application.

Claims

1. A method for allocating bandwidth resources of a satellite communication network, characterized in that: include: Determining a first priority of each training experience sample stored in a preset first storage area, and a second priority of each training experience sample stored in a preset second storage area, wherein the training experience sample represents an execution parameter of a communication service; Selecting a training experience sample from each training experience sample in the first storage area according to the first priority, and selecting a training experience sample from each training experience sample in the second storage area according to the second priority; Obtaining a training experience sample corresponding to the current network environment state, combining the training experience sample selected in the first storage area, the training experience sample selected in the second storage area, and the training experience sample corresponding to the current network environment state to obtain a combined target training sample; Using the combined target training samples to train a pre-constructed deep reinforcement learning model to obtain a trained deep reinforcement learning model, including: obtaining a true state-action value corresponding to the combined target training samples; inputting the combined target training samples into the pre-constructed deep reinforcement learning model, and outputting a predicted state-action value through the pre-constructed deep reinforcement learning model; constructing a loss function based on the true state-action value and the predicted state-action value, minimizing the loss function using an ε-greedy strategy, and training and adjusting the pre-constructed deep reinforcement learning model based on the result of the minimization process to obtain a trained deep reinforcement learning model; Acquire real-time network environment status information, and based on the real-time network environment status information, use the trained deep reinforcement learning model to determine the state action value obtained by selecting each satellite-to-ground network node to perform the communication service; Determine the maximum state action value among the state action values corresponding to the respective satellite-to-ground network nodes, obtain a target network bandwidth occupancy value when the satellite-to-ground network node corresponding to the maximum state action value performs a communication service, and allocate satellite communication network bandwidth resources to the satellite-to-ground network node corresponding to the maximum state action value according to the target network bandwidth occupancy value.

2. The method according to claim 1, characterized in that Before determining the first priority of each training experience sample stored in the preset first storage area and the second priority of each training experience sample stored in the preset second storage area, the method further includes: Obtaining a current transmission demand state of any user terminal among a plurality of user terminals; selecting a target satellite-ground network node from a plurality of satellite-ground network nodes to execute a communication service based on the current transmission demand state, and obtaining a network bandwidth limit value of a transmission channel between any user terminal and the target satellite-ground network node, a network bandwidth occupancy value when executing the communication service, and a next transmission demand state of the current transmission demand state; Determining a reward value using the network bandwidth limit and the network bandwidth occupancy value; storing, according to the reward value, the training experience sample formed by combining the current transmission demand state, the target satellite-to-ground network node, the reward value, and the next transmission demand state in the first storage area or a preset third storage area; Using the training experience samples stored in the first storage area to train the pre-built long short-term memory network model to obtain a trained long short-term memory network model; The training experience samples stored in the third storage area are input into the trained long short-term memory network model for prediction to obtain predicted experience samples, and the predicted experience samples are stored in the second storage area.

3. The method according to claim 2, characterized in that The determining of the reward value by using the network bandwidth limit and the network bandwidth occupancy value includes: Obtaining the number of nodes in the satellite-to-ground network; Performing a product processing on the number of satellite-to-ground network nodes and the network bandwidth limit to obtain a first product processing result; Perform ratio processing on the network bandwidth occupancy value and the first product processing result to obtain the reward value.

4. The method according to claim 2, characterized in that The step of storing the training experience sample formed by combining the current transmission demand state, the target satellite-to-ground network node, the reward value, and the next transmission demand state according to the reward value in the first storage area or a preset third storage area includes: Determine whether the reward value is greater than or equal to a preset reward value threshold, and obtain a determination result; In response to the judgment result being yes, the training experience sample formed by combining the current transmission demand state, the target satellite-ground network node, the reward value, and the next transmission demand state is stored in the first storage area; or In response to the judgment result being no, the training experience sample formed by combining the current transmission demand state, the target satellite-ground network node, the reward value, and the next transmission demand state is stored in the third storage area.

5. The method according to claim 1, wherein The determining of the first priority of each training experience sample stored in the preset first storage area includes: Determining a first cosine similarity between the selected training experience samples in the first storage area and the unselected training experience samples in the first storage area; Obtaining the number of the selected training experience samples and a time difference error, where the time difference error is a difference between a reward value corresponding to a current transmission requirement state of any user terminal among the multiple user terminals and a reward value corresponding to a transmission requirement state next to the current transmission requirement state; performing difference processing on the number of the selected training experience samples and the first cosine similarity to obtain a first difference processing result; Performing ratio processing on the first difference processing result and the number of the selected training experience samples to obtain a first ratio processing result; The first priority is obtained by multiplying the first ratio processing result by the absolute value of the time difference error.

6. The method according to claim 1, characterized in that The determining of the second priority of each training experience sample stored in the preset second storage area includes: Determining a second cosine similarity between the selected training experience samples in the second storage area and the unselected training experience samples in the second storage area; Obtaining the number of the selected training experience samples and a time difference error, where the time difference error is a difference between a reward value corresponding to a current transmission requirement state of any user terminal among the multiple user terminals and a reward value corresponding to a transmission requirement state next to the current transmission requirement state; performing difference processing on the number of the selected training experience samples and the second cosine similarity to obtain a second difference processing result; performing ratio processing on the second difference processing result and the number of the selected training experience samples to obtain a second ratio processing result; The second priority is obtained by multiplying the second ratio processing result by the absolute value of the time difference error.

7. The method according to claim 1, characterized in that The real-time network environment status information includes: current satellite-to-ground network status parameters; The method of determining, based on the real-time network environment status information, the state action value obtained by selecting each satellite-to-ground network node to execute the communication service using the trained deep reinforcement learning model includes: Obtaining an instant reward value and a discount factor obtained by executing any satellite-to-ground network node under the current satellite-to-ground network state parameters, and a current state action value obtained by selecting a satellite-to-ground network node corresponding to the current maximum state action value under the current satellite-to-ground network state parameters; Performing a product process using the discount factor and the current state action value to obtain a second product processing result; The second multiplication result and the instant reward value are summed to obtain a state action value obtained by selecting each satellite-ground network node to perform the communication service.

8. A device for allocating bandwidth resources of a satellite communication network, characterized in that: include: a priority determination module configured to determine a first priority of each training experience sample stored in a preset first storage area, and a second priority of each training experience sample stored in a preset second storage area, wherein the training experience sample represents an execution parameter of a communication service; a sample selection module configured to select a training experience sample from each training experience sample in the first storage area according to the first priority, and to select a training experience sample from each training experience sample in the second storage area according to the second priority; a sample combination module configured to obtain a training experience sample corresponding to the current network environment state, and combine the training experience sample selected in the first storage area, the training experience sample selected in the second storage area, and the training experience sample corresponding to the current network environment state to obtain a combined target training sample; a training module configured to train a pre-constructed deep reinforcement learning model using the combined target training samples to obtain a trained deep reinforcement learning model, including: obtaining a true state-action value corresponding to the combined target training samples; inputting the combined target training samples into the pre-constructed deep reinforcement learning model, and outputting a predicted state-action value through the pre-constructed deep reinforcement learning model; constructing a loss function based on the true state-action value and the predicted state-action value, minimizing the loss function using an ε-greedy strategy, and training and adjusting the pre-constructed deep reinforcement learning model based on the result of the minimization process to obtain a trained deep reinforcement learning model; a value determination module configured to obtain real-time network environment status information and, based on the real-time network environment status information, use the trained deep reinforcement learning model to determine the state action value obtained by selecting each satellite-ground network node to perform the communication service; The resource allocation module is configured to determine a maximum state action value among the state action values corresponding to the satellite-to-ground network nodes, obtain a target network bandwidth occupancy value when the satellite-to-ground network node corresponding to the maximum state action value performs a communication service, and allocate satellite communication network bandwidth resources to the satellite-to-ground network node corresponding to the maximum state action value according to the target network bandwidth occupancy value.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images