Resource allocation method based on depth deterministic policy gradient algorithm and related equipment

Abstract

The invention provides a resource allocation method based on a depth deterministic policy gradient algorithm and related equipment. The method comprises the following steps: acquiring state parameters; the state parameter is used for representing a channel environment state of the satellite station; inputting the state parameters into a deep reinforcement learning model to generate at least one action decision; the deep reinforcement learning model is obtained by training based on a deep deterministic strategy gradient algorithm and a sodat swarm optimization algorithm; the action decision is used for indicating a resource allocation and power control strategy; calculating an expected value of the at least one action decision, and determining the action decision with the maximum expected value as a target action decision; and executing the resource allocation and power control strategy based on the target action decision. The problem of joint optimization of resource allocation and power control is solved by using the deep reinforcement learning model, so that the resource allocation efficiency is improved, and the network performance is optimized.

Description

Technical Field

[0001] The present application relates to the field of satellite communications, and in particular to a resource allocation method and related equipment based on a deep deterministic policy gradient algorithm. Background Art

[0002] As the foundation of the national economy, the safe and stable operation of the power industry is vital to the livelihood of the people and national security. With the increase of distributed energy and remote power plants, the power system needs a communication technology that can provide wide-area coverage and transmit key information in real time to ensure the efficiency and reliability of the power system.

[0003] Although satellite communications can provide wide-area coverage, their spectrum resources are limited and it is difficult to meet the needs of power business for seamless connection and high-throughput services. In addition, as the integration of satellite and terrestrial networks becomes an important trend in future communication networks, it also faces challenges such as overlapping frequency resources, insufficient security and difficulty in ensuring reliability during the integration process. Summary of the invention

[0004] In view of this, the purpose of this application is to propose a resource allocation method and related equipment based on a deep deterministic policy gradient algorithm.

[0005] As one aspect of the present application, a resource allocation method based on a deep deterministic policy gradient algorithm is provided, which is applied to a satellite station, including:

[0006] Acquiring state parameters; the state parameters are used to characterize the channel environment state of the satellite station;

[0007] Inputting the state parameters into a deep reinforcement learning model to generate at least one action decision; the deep reinforcement learning model is trained based on a deep deterministic policy gradient algorithm and a sand cat swarm optimization algorithm; the action decision is used to indicate resource allocation and power control strategies;

[0008] Calculating an expected value of at least one action decision, and determining the action decision with the largest expected value as a target action decision;

[0009] Based on the target action decision, the resource allocation and power control strategy is executed.

[0010] Optionally, the deep reinforcement learning model includes: a policy network, a target policy network, a Q network and a target Q network, including:

[0011] Obtain multiple strategy network individuals in the initial model;

[0012] Iteratively optimize the multiple strategy network individuals based on the sand cat swarm optimization algorithm to obtain multiple target strategy network individuals;

[0013] Calculating the fitness of the multiple target strategy network individuals, and saving the target strategy network individual with the highest fitness into the initial model;

[0014] In response to the target strategy network individual reaching a first preset condition, determining that the initial model is the deep reinforcement learning model; the first preset condition includes at least one of the following: the number of iterative optimizations reaches a first preset threshold and the fitness is greater than or equal to a second preset threshold.

[0015] Optionally, calculating the expected value of the at least one action decision and determining the action decision with the largest expected value as the target action decision includes:

[0016] Inputting at least one state-action pair into the Q network to obtain an expected value of the at least one state-action pair; the state-action pair includes the state parameter and the corresponding action decision;

[0017] In response to the expected value of the state-action pair being the largest, the action decision in the state-action pair is determined to be the target decision.

[0018] Optionally, after the decision based on the target action is made and the resource allocation and power control strategy is executed, the method further includes:

[0019] Get user data;

[0020] Based on the resource extension multiple access transmission mechanism, the user data is divided into public stream data and private stream data, and sent to the ground station.

[0021] Optionally, determine a public flow rate and a private flow rate; determine a reward value based on the public flow rate and the private flow rate; the reward value is the sum of the public flow rate and the private flow rate; and store the state parameters, the target action decision, the reward value, and the next state parameters as experience samples in an experience sample database.

[0022] Optionally, in response to the number of experience samples reaching a third preset threshold, based on a priority mechanism, the priority of the experience samples is determined; based on the priority of the experience samples, an experience sample group is determined; based on the experience sample group, the deep reinforcement learning model is trained; in response to the deep reinforcement learning model reaching a second preset condition, the training is terminated; the second preset condition includes at least one of the following: the number of training times reaches a fourth preset threshold and the expected value of the action decision output by the deep reinforcement learning model is greater than or equal to a fifth preset threshold.

[0023] Optionally, the priority mechanism is a priority mechanism based on timing differential error and Q value;

[0024] The priority mechanism based on the timing difference error and the Q value satisfies the following expression:

[0025]

[0026] Wherein, P(i) is the priority of the i-th experience sample based on the priority mechanism, μ TD is the average value of the time series difference error of all empirical samples, σ TD is the standard deviation of the time series difference error of all empirical samples, P TD (i) is the priority of the i-th empirical sample calculated based on the time series difference error, P Q (i) is the priority of the i-th experience sample calculated based on the Q value, and ε is a small positive value to ensure that P(i)>0;

[0027] The priority of the i-th experience sample calculated based on the timing difference error satisfies the following expression:

[0028]

[0029] The priority of the i-th experience sample calculated based on the Q value satisfies the following expression:

[0030]

[0031] Among them, the softsign function and ISRU function are used to normalize the priority, δ i is the time series difference error of the i-th empirical sample, s i is the state parameter in the i-th experience sample, a i The target action decision is made for the state parameter.

[0032] As a second aspect of the present application, a resource allocation device based on a deep deterministic policy gradient algorithm is provided, which is applied to a satellite station and includes: an acquisition module, a generation module, a determination module, and an execution module;

[0033] The acquisition module is used to acquire state parameters; the state parameters are used to characterize the channel environment state of the satellite station;

[0034] The generation module is used to input the state parameters into a deep reinforcement learning model to generate at least one action decision; the deep reinforcement learning model is trained based on a deep deterministic policy gradient algorithm and a sand cat swarm optimization algorithm; the action decision is used to indicate resource allocation and power control strategies;

[0035] The determination module is used to calculate the expected value of the at least one action decision, and determine the action decision with the largest expected value as the target action decision;

[0036] The execution module is used to execute the resource allocation and power control strategy based on the target action decision.

[0037] Optionally, the deep reinforcement learning model includes: a policy network, a target policy network, a Q network and a target Q network, and the resource allocation device based on the deep deterministic policy gradient algorithm also includes an optimization module and a processing module;

[0038] The acquisition module is also used to acquire multiple strategy network individuals in the initial model;

[0039] The optimization module is used to iteratively optimize the multiple strategy network individuals based on the sand cat swarm optimization algorithm to obtain multiple target strategy network individuals;

[0040] The processing module is used to calculate the fitness of the multiple target strategy network individuals and save the target strategy network individual with the highest fitness into the initial model;

[0041] The determination module is also used to determine that the initial model is the deep reinforcement learning model in response to the target strategy network individual reaching a first preset condition; the first preset condition includes at least one of the following: the number of iterative optimizations reaches a first preset threshold and the fitness is greater than or equal to a second preset threshold.

[0042] Optionally, the determination module is specifically used to input at least one state-action pair into the Q network to obtain an expected value of the at least one state-action pair; the state-action pair includes the state parameters and the corresponding action decision; in response to the expected value of the state-action pair being the largest, determine the action decision in the state-action pair as the target decision.

[0043] Optionally, the resource allocation device based on the deep deterministic policy gradient algorithm further includes a sending module;

[0044] The acquisition module is also used to acquire user data;

[0045] The sending module is used to divide the user data into public stream data and private stream data based on a resource extended multiple access transmission mechanism, and send the data to a ground station.

[0046] Optionally, the determination module is further used to determine a public flow rate and a private flow rate;

[0047] The determination module is further configured to determine a reward value based on the public flow rate and the private flow rate; the reward value is the sum of the public flow rate and the private flow rate;

[0048] The processing module is further configured to store the state parameter, the target action decision, the reward value, and the next state parameter as experience samples in an experience sample database.

[0049] Optionally, the resource allocation device based on the deep deterministic policy gradient algorithm further includes a training module;

[0050] The determination module is further configured to, in response to the number of the experience samples reaching a third preset threshold, determine the priority of the experience samples based on a priority mechanism;

[0051] The determination module is further configured to determine an experience sample group based on the priority of the experience sample;

[0052] The training module is used to train the deep reinforcement learning model based on the experience sample group;

[0053] The training module is also used to end training in response to the deep reinforcement learning model reaching a second preset condition; the second preset condition includes at least one of the following: the number of training times reaches a fourth preset threshold and the expected value of the action decision output by the deep reinforcement learning model is greater than or equal to a fifth preset threshold.

[0054] Optionally, the priority mechanism is a priority mechanism based on timing differential error and Q value;

[0055] The priority mechanism based on the timing difference error and the Q value satisfies the following expression:

[0056]

[0057] Wherein, P(i) is the priority of the i-th experience sample based on the priority mechanism, μ TD is the average value of the time series difference error of all empirical samples, σ TD is the standard deviation of the time series difference error of all empirical samples, P TD (i) is the priority of the i-th empirical sample calculated based on the time series difference error, P Q (i) is the priority of the i-th experience sample calculated based on the Q value, and ε is a small positive value to ensure that P(i)>0;

[0058] The priority of the i-th experience sample calculated based on the timing difference error satisfies the following expression:

[0059]

[0060] The priority of the i-th experience sample calculated based on the Q value satisfies the following expression:

[0061]

[0062] Among them, the softsign function and ISRU function are used to normalize the priority, δ i is the time series difference error of the i-th empirical sample, s i is the state parameter in the i-th experience sample, a i The target action decision is made for the state parameter.

[0063] As the third aspect of the present application, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the resource allocation method based on the deep deterministic policy gradient algorithm as described above is implemented.

[0064] As the fourth aspect of the present application, a non-transitory computer-readable storage medium is provided, which stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned resource allocation method based on the deep deterministic policy gradient algorithm provided in the present application.

[0065] From the above, it can be seen that the resource allocation method and related equipment based on the deep deterministic policy gradient algorithm provided by the present application obtain a deep reinforcement learning model by using the sand cat swarm optimization algorithm combined with the deep deterministic policy gradient algorithm (deepdeterministic policy gradient, DDPG) training, which solves the problem of slow training speed caused by the scarcity of experience rewards in the early stage of training of the deep deterministic policy gradient algorithm, and improves the decision-making effect. Furthermore, by inputting the satellite station state parameters into the deep reinforcement learning model, a resource allocation plan can be generated in real time. Furthermore, by predicting the execution effects of different action decisions, the action decision with the largest expected value is selected as the optimal resource allocation plan, thereby optimizing the resource allocation and power control strategies, improving the resource utilization and communication capacity of the satellite-to-ground network, and further improving the overall performance of the power communication system. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the present application or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0067] Figure 1 A schematic diagram of a communication system for a resource allocation method based on a deep deterministic policy gradient algorithm provided in an embodiment of the present application;

[0068] Figure 2 A schematic diagram of a process flow of a resource allocation method based on a deep deterministic policy gradient algorithm provided in an embodiment of the present application;

[0069] Figure 3 A schematic diagram of a process flow of a resource allocation method based on a deep deterministic policy gradient algorithm provided in an embodiment of the present application;

[0070] Figure 4 A schematic diagram of a process flow of a resource allocation method based on a deep deterministic policy gradient algorithm provided in an embodiment of the present application;

[0071] Figure 5 A schematic diagram of a process flow of a resource allocation method based on a deep deterministic policy gradient algorithm provided in an embodiment of the present application;

[0072] Figure 6 A schematic diagram of a process flow of a resource allocation method based on a deep deterministic policy gradient algorithm provided in an embodiment of the present application;

[0073] Figure 7 A schematic diagram of a process flow of a resource allocation method based on a deep deterministic policy gradient algorithm provided in an embodiment of the present application;

[0074] Figure 8 A schematic diagram of the composition of a resource allocation device based on a deep deterministic policy gradient algorithm provided in an embodiment of the present application;

[0075] Fig. 9 A schematic diagram of the composition of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0077] It should be noted that, in the embodiments of the present application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a concrete way. Unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should have the usual meaning understood by people with ordinary skills in the field to which the present application belongs. SUMMARY OF THE INVENTION

[0079] In related technologies, since resource allocation between satellite networks and ground networks and resource spread multiple access (RSMA) power control are non-deterministic polynomially difficult (NP-hard) problems, many studies use heuristic algorithms and convex optimization methods to obtain suboptimal solutions. However, these methods are model-dependent, and their effectiveness greatly depends on a comprehensive understanding of instantaneous channel state information (CSI). However, the instantaneous CSI of the transmitter is difficult to obtain in actual scenarios, especially for satellite communication systems. Due to the fast channel time variation and extended propagation time, CSI is usually outdated. Considering the dynamics and uncertainty of the system, traditional methods cannot effectively optimize resource allocation and power control at the same time when channel distribution information is limited. The solution speed and solution quality of these two methods are difficult to guarantee.

[0080] The inventors of this application found that, on the one hand, the power allocation method based on RSMA optimizes the power allocation of satellite base stations through reinforcement learning, but because it adopts a proximal strategy optimization algorithm, the training process is slow and only uses the previous round of strategy data, resulting in poor decision-making before the neural network converges, and does not fully consider the frequency resource allocation problem between satellites and ground networks, and the actual application effect is limited. On the other hand, the method of using continuous convex approximation to solve non-convex optimization problems can handle complex optimization problems, but it will introduce approximation errors, resulting in a gap between the solution and the optimal solution, and the high computational complexity and long solution time affect the real-time and response speed of the system. At the same time, the convergence and stability of the algorithm vary in different environments and conditions, which increases the uncertainty of practical applications.

[0081] In order to solve the above problems, the present application provides a resource allocation method based on a deep deterministic policy gradient algorithm. A deep reinforcement learning model is obtained by using a sand cat swarm optimization algorithm combined with a deep deterministic policy gradient algorithm for training. The problem of slow training speed caused by the scarcity of experience rewards in the early stage of training of the deep deterministic policy gradient algorithm is solved, and the decision-making effect is improved. Furthermore, by inputting the satellite station state parameters into the deep reinforcement learning model, a resource allocation plan can be generated in real time. Furthermore, by predicting the execution effects of different action decisions, the action decision with the largest expected value is selected as the optimal resource allocation plan, thereby optimizing the resource allocation and power control strategies, improving the resource utilization and communication capacity of the satellite-to-ground network, and further improving the overall performance of the power communication system.

[0082] After introducing the basic principles of the present application, various non-limiting implementation methods of the present application are described in detail below.

[0083] Application Scenario Overview

[0084] Figure 1 A communication system diagram of a resource allocation method based on a deep deterministic policy gradient algorithm provided in an embodiment of the present application. Figure 1 As shown, the communication system includes a satellite station 101, a ground station 102 and a terminal 103. The satellite station 101 is connected to the ground station 102 via a wireless communication network, and the satellite station 101 is connected to the terminal 103 via a wireless communication network.

[0085] The satellite station 101 and the ground station 102 mentioned here can be independent physical servers, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Optionally, the server can also be implemented on a cloud platform. For example, the cloud platform can include private clouds, public clouds, hybrid clouds, community clouds, distributed clouds, inter-clouds, and multi-clouds, etc., or any combination thereof. The embodiments of the present application are not limited to this.

[0086] In some embodiments, the terminal 103 includes but is not limited to a desktop computer, a mobile phone, a mobile computer, a tablet computer, a media player, a smart wearable device, a personal digital assistant (PDA) or other electronic devices capable of realizing the above functions.

[0087] In some embodiments, the terminal 103 is responsible for collecting and sending user data, such as voice, video, text information or sensor data, and converting the data into a format suitable for wireless transmission. The terminal 103 uploads the user data to the satellite station 101 via a satellite communication link to achieve remote communication.

[0088] In some embodiments, the satellite station 101 is responsible for resource allocation and data forwarding. The satellite station 101 receives user data from the terminal 103, and forwards the user data to the ground station 102 according to the resource allocation and power control strategy, and receives control signals and possible data updates from the ground station 102.

[0089] In some embodiments, the ground station 102 receives the user data forwarded by the satellite station 101, processes the received user data, including demodulation, decoding, verification, etc., and then transmits the data to the final destination, such as the Internet or other communication networks. The ground station 102 can also send control instructions to the satellite station 101, including orbit adjustment, attitude control, fault diagnosis, etc., to ensure that the satellite station can operate stably and meet communication requirements.

[0090] It should be understood that Figure 1 is an exemplary structural diagram, Figure 1 The communication system shown includes an unlimited number of devices, for example, an unlimited number of ground stations 102 and an unlimited number of terminals 103. Figure 1 In addition to the equipment shown, Figure 1 The communication system shown may also include other devices, which is not limited thereto.

[0091] Combine the following Figure 1 The resource allocation method based on the deep deterministic policy gradient algorithm according to the exemplary embodiment of the present application is described by using a communication system. It should be noted that the above communication system is only shown to facilitate understanding of the spirit and principle of the present application, and the implementation of the present application is not limited in this regard. On the contrary, the implementation of the present application can be applied to any applicable scenario.

[0092] Figure 2 The following is a flow chart of a resource allocation method based on a deep deterministic policy gradient algorithm provided in an embodiment of the present application. Figure 2 As shown, the resource allocation method based on the deep deterministic policy gradient algorithm provided in the present application can be implemented by the above-mentioned satellite station 101, and specifically includes the following steps:

[0093] S201. Obtain status parameters.

[0094] Among them, the state parameter is used to characterize the channel environment state of the satellite station

[0095] In some embodiments, the channel environment state includes the propagation characteristics of signals, such as path loss, Doppler shift, shadowing effect, etc. These factors will affect the signal quality and the performance of the communication system. The acquisition of the state parameters can be achieved through real-time measurement and estimation of signals. For example, the impulse response or path loss of the channel can be obtained through channel estimation technology.

[0096] It should be noted that in a satellite communication system, the state parameters refer to a series of measures or data that can characterize the current channel environment state of the satellite station. These parameters may include but are not limited to signal strength, signal-to-noise ratio (SNR), bit error rate (BER), channel capacity, interference level, multipath effect, Doppler shift, etc. These parameters are crucial for evaluating channel quality, predicting communication performance, and optimizing resource allocation and power control strategies.

[0097] In some embodiments, in a satellite fusion network, there are m terrestrial cells (T-Cells), and each cell orthogonally occupies a resource block (RB) to provide services for multiple cellular users within the cell. At the same time, a low earth orbit (LEO) satellite provides downlink services to N ground stations (GSs) by sharing the RBs of the m cells. Among them, K (K < n) of the n GSs are combined into a GS group and provide services through the same RB simultaneously to improve spectral efficiency. Each GS can belong to multiple groups and can receive services from the LEO satellite on multiple RBs. Since T-cell users are usually located in areas severely blocked by buildings, their antenna gain is usually much lower than that of GSs. Therefore, the interference from the LEO satellite can be ignored, and the interference from the T-Cell base station (BS) is considered. The satellite base station needs to sense the current environment state so that the agent can make appropriate decisions. Further, the historical signal-to-interference-plus-noise ratio (SINR) of the common stream and the private stream from the ground station is defined as the channel observation value of the mth RB. Specifically, the state parameters of the mth resource block at time t satisfy the following expression:

[0098]

[0099] where represents the SINR of the common stream observed by ground station n for the mth RB at time t - 1, represents the SINR of the private stream observed by ground station n for the mth RB at time t - 1.

[0100] S202: Input the state parameters into a deep reinforcement learning model to generate at least one action decision.

[0101] Among them, the deep reinforcement learning model is trained based on the deep deterministic policy gradient algorithm and the sand cat swarm optimization algorithm; the action decision is used to indicate the resource allocation and power control strategy.

[0102] It should be noted that the DDPG algorithm is an algorithm that combines the actor-critic framework and the policy gradient method. It is suitable for problems in continuous action space. By learning a deterministic strategy, the DDPG algorithm can find the optimal action under a given state. The sand cat swarm optimization algorithm (SCSO) is a swarm intelligence optimization algorithm inspired by the predation behavior of sand cats. It solves optimization problems by simulating the search and attack behavior of sand cats. It has good optimization ability and fast convergence speed.

[0103] In some embodiments, by passing the state parameters as input to the trained deep reinforcement learning model, the model will generate one or more action decisions based on the current state using the policy network. These action decisions include resource allocation strategies (such as frequency band allocation, time slot allocation), power control strategies (such as adjusting the transmit power), coding and modulation scheme selection, etc., and then, the deep reinforcement learning model will predict the value of each action decision through a neural network based on the current channel environment state and historical data, and select the action decision that can bring the maximum expected value.

[0104] It should be understood that in a communication system, resource allocation involves allocating resources to different tasks, users or devices according to certain rules or algorithms under limited resource conditions in order to optimize system performance, efficiency or meet specific needs, such as improving throughput, reducing latency or ensuring quality of service. Power control is the process of adjusting the power output of a device or system to meet performance requirements while reducing energy consumption, reducing interference and improving energy efficiency.

[0105] In some embodiments, the deep reinforcement learning model includes: a policy network, a target policy network, a Q network, and a target Q network, such as Figure 3 As shown, the resource allocation method based on the deep deterministic policy gradient algorithm provided in the embodiment of the present application also includes the following S301-S304:

[0106] S301, obtaining multiple strategy network individuals in the initial model.

[0107] In some embodiments, random parameters are used to initialize the policy network, target policy network, Q network, and target Q network of DDPG to obtain an initial model. Among them, the policy network (actor network) is used to generate actions in a given state in deep reinforcement learning. In practical applications, multiple policy network individuals are created by randomly initializing the weights and biases of the network. These individuals serve as the starting point of the subsequent optimization process, and each individual has the ability to explore and learn environmental strategies.

[0108] S302, iteratively optimizing multiple strategy network individuals based on the sand cat swarm optimization algorithm to obtain multiple target strategy network individuals.

[0109] In some embodiments, the diversity of the SCSO algorithm can help the algorithm avoid local optimality and potentially find reward signals in environments with scarce rewards. First, the policy network individuals in the SCSO algorithm are initialized. Then, the exploration phase (searching for prey) and the development phase (attacking prey) of the algorithm correspond to the exploration and utilization behaviors of the agent, respectively. By iteratively optimizing multiple policy network individuals, the SCSO algorithm continuously updates the network weights to improve the performance of the policy network and obtain multiple target policy network individuals.

[0110] S303, calculating the fitness of multiple target strategy network individuals, and saving the target strategy network individual with the highest fitness into the initial model.

[0111] In some embodiments, each target policy network individual is placed in an environment for testing, and its cumulative reward or other performance indicators obtained within a certain period of time or a certain number of steps are recorded. Then, the fitness of all target policy network individuals is compared, and the individual with the highest fitness is selected as the current optimal solution, and it is saved in the initial model to replace or supplement the original policy network.

[0112] For example, the population π of the SCSO algorithm pop There are multiple target strategy network individuals in the group π pop The fitness of each target strategy network individual in is calculated by the target strategy network individual in a complete cycle (episode) according to the state parameter s t And output the action decision a t , calculate the reward value r obtained by the satellite station after executing the action decision t+1 , and use it as fitness. From π popAfter obtaining the fitness of each target policy network individual, select the target policy network individual with the highest fitness. For each target policy network individual in the population, use the roulette wheel selection algorithm to obtain the random exploration angle θ, and perform the exploration and development phase. Use the best target policy network individual position, the current target policy network individual position, and the sensitivity range to update the position of the target policy network individual, that is, update the network parameters of the policy network. Two parameter trackers are set during the algorithm iteration process to migrate the network parameters of the policy network. When the fitness satisfies the following expression, the network parameters of the policy network in the DDPG algorithm are migrated to π pop The strategy network with the smallest fitness in the network:

[0113]

[0114] Among them, fit l is the fitness of the individual in the strategy network in the DDPG algorithm, and N is π pop The total number of individuals in the target strategy network;

[0115] When the fitness satisfies the following expression, the π with the largest fitness is pop The network parameters of the policy network in the algorithm are migrated to the policy network in the DDPG algorithm:

[0116]

[0117] S304: In response to the target strategy network individual reaching a first preset condition, determining that the initial model is a deep reinforcement learning model.

[0118] The first preset condition includes at least one of the following: the number of iterative optimization times reaches a first preset threshold and the fitness is greater than or equal to a second preset threshold.

[0119] In some embodiments, the target policy network is stored as an experience sample in the replay buffer of DDPG, realizing the information flow from the evolutionary population to the neural network in the initial training stage. When the number of iterative optimizations reaches a first preset threshold or the fitness is greater than or equal to a second preset threshold, the evolutionary algorithm (EA) process is stopped, and the network parameters of the policy network in the DDPG algorithm at this time are copied to the target policy network, and then the initial model containing the optimal policy network individuals is determined as a deep reinforcement learning model, that is, the initialization of the deep reinforcement learning model is completed.

[0120] S203: Calculate the expected value of at least one action decision, and determine the action decision with the largest expected value as the target action decision.

[0121] In some embodiments, the action that can bring the greatest return is selected by evaluating the potential value of different action decisions in a given state, that is, the expected value. Among them, the expected value calculation method may include: estimating the expected return that the agent can obtain by following a specific strategy starting from a certain state through the state value function; estimating the expected return of each action by simulating random trajectories from the current state multiple times through Monte Carlo tree search; using the system model to predict the future state through model predictive control, solving an optimization problem at each step to select the action that maximizes the future return; using the Q network in the deep neural network to estimate the value of the action in the state by learning the value function of the state-action pair, etc.

[0122] In some embodiments, Figure 4 As shown, S203 can be specifically implemented as follows S2031-S2032:

[0123] S2031. Input at least one state-action pair into the Q network to obtain an expected value of at least one state-action pair.

[0124] Among them, the state-action pair includes state parameters and corresponding action decisions.

[0125] It should be noted that the Q network (action value network) is a model that uses a neural network to approximate the Q function, which is used to estimate the expected return (expected value) that can be obtained by taking an action in a given state.

[0126] In some embodiments, each state-action pair is passed as input to the Q network, which processes these inputs through its internal neural network structure and outputs the expected value (Q value) of each state-action pair, where the Q value represents the expected reward that can be obtained by taking the action decision in this state.

[0127] For example, the satellite station inputs the current state parameters into the policy network and outputs the action decision. The action decision performed by the agent on the mth RB is a m ()=[ m (),p m ()], where α m ()=[ 1, (),…,α N,m ()] is an N-dimensional vector, indicating the allocation of the mth RB at time step t, where α n,m () = 1 means that the nth GS occupies the mth RB, otherwise α n,m ()=0. Similarly, p m ()=[ c,m (),p 1, (),…,p N,m()] is an N+1-dimensional vector, which represents the power allocation of GS on the mth RB. For the n randomly generated action decisions in the action space Input them into the Q network respectively to obtain the Q value of the state action pair { 1 ,2,…, n}, where the action decision executed on the mth RB satisfies the following expression:

[0128]

[0129] The specific generation of random actions satisfies the following expression:

[0130] a ′ t =(a t );

[0131]

[0132] It should be understood that since DDPG uses random perturbations to enhance the algorithm's exploration ability, the output action may exceed the action boundary. The method in DDPG is to set boundary values ​​for actions that exceed the space limit, which may cause the agent to repeat the boundary action and cannot effectively increase the exploration. Therefore, this solution calculates the expected value of the action decision through the Q network, avoiding the repeated execution of the boundary action, while increasing the rationality of the decision action and enhancing the exploration efficiency. In addition, for the power control problem, the action space is multidimensional. Therefore, the above f(*) will be performed in each dimension, improving the performance of the DDPG algorithm in the multidimensional action space.

[0133] S2032: In response to the expected value of the state-action pair being the largest, determine the action decision in the state-action pair as the target decision.

[0134] In some embodiments, after obtaining the expected values ​​of all state-action pairs, the expected values ​​of different action decisions are compared, and the action decision in the state-action pair with the largest expected value is selected as the target action decision to guide subsequent operations such as resource allocation and power control.

[0135] S204: Execute resource allocation and power control strategies based on the target action decision.

[0136] In some embodiments, according to the target action decision determined in S203, the corresponding resource allocation and power control strategy is executed, for example, in the communication network, the bandwidth allocation to a certain user is increased or decreased to balance the network load or improve the user experience; the power mode of the system component is switched to a low power mode or a sleep mode to achieve the purpose of energy saving or improving efficiency. In addition, after the target action decision is executed, the changes and effects of the environment are continuously monitored to evaluate the effectiveness of the action decision.

[0137] The resource allocation method and related equipment based on the deep deterministic policy gradient algorithm provided in the present application obtain a deep reinforcement learning model by using the sand cat swarm optimization algorithm combined with the deep deterministic policy gradient algorithm for training, so as to solve the problem that the deep deterministic policy gradient algorithm has a slow training speed due to the scarcity of experience rewards in the early stage of training, and improve the decision-making effect. Furthermore, by inputting the satellite station state parameters into the deep reinforcement learning model, a resource allocation plan can be generated in real time. Furthermore, by predicting the execution effects of different action decisions, the action decision with the largest expected value is selected as the optimal resource allocation plan, thereby optimizing the resource allocation and power control strategies, improving the resource utilization and communication capacity of the satellite-to-ground network, and further improving the overall performance of the power communication system.

[0138] In some embodiments, Figure 5 As shown, after S204, the resource allocation method based on the deep deterministic policy gradient algorithm provided in the embodiment of the present application further includes the following S401-S402:

[0139] S401: Obtain user data.

[0140] In some embodiments, the satellite station collects user data from the terminal device through a wireless interface, such as voice, video, text information, file transfer or any other form of digital communication content. The collected user data is temporarily stored in the local storage or cloud storage of the satellite station, awaiting further processing or transmission.

[0141] S402: Based on the resource extension multiple access transmission mechanism, the user data is divided into public stream data and private stream data, and sent to the ground station.

[0142] In some embodiments, the basic idea of ​​the resource extended multiple access transmission mechanism (RSMA) is to divide the user data of each user into two parts: public messages and private messages. The public messages of all users are encoded into a public stream data and decoded by multiple users, while the private messages are independently encoded into private stream data and decoded by the corresponding users. The transmitter adopts a superposition transmission method to superimpose all code streams for signal transmission. The total received signal is decoded at the receiving end of the ground station. The public stream data is first decoded by the successive interference cancellation (SIC) technology, and then other private stream data are regarded as noise to decode its corresponding private stream data.

[0143] It should be understood that RSMA is an advanced multiple access scheme proposed for future 6G networks, which has been proven to have obvious performance advantages over non-orthogonal multiple access transmission mechanism (NOMA) and space division multiple access transmission mechanism (SDMA).

[0144] In some embodiments, Figure 6 As shown, the resource allocation method based on the deep deterministic policy gradient algorithm provided in the embodiment of the present application also includes the following S501-S503:

[0145] S501: Determine a public flow rate and a private flow rate.

[0146] In some embodiments, after the satellite station sends the public stream data and the private stream data to the ground station, the ground station first decodes the public stream data using the SIC technology, then subtracts it from the total received signal, and regards the remaining signal as containing the private stream data and noise. At this time, the ground station can decode its own private stream data and regard other private stream data as noise. Further, the public stream rate and the private stream rate are calculated based on the decoded public stream data, private stream data and the resource allocation scheme.

[0147] In some embodiments, the SINR of the public stream data and the SINR of the private stream data are first calculated, and the SINR of the public stream data and the private stream data of the nth (n=1, ..., N) GS on the mth RB satisfies the following expression:

[0148] Public stream data SINR:

[0149]

[0150] Private flow data SINR:

[0151]

[0152] Among them, P T is the transmit power of T-Cell BS, σ 2 represents the noise power spectral density. g n,mis the link gain between the nth GS and the LEO satellite on the mth RB, is the terrestrial interference link gain between the T-Cell BS using the mth RB and the nth (n=1, 2, ..., N) GS.

[0153] g n,m The rate satisfies the following expression:

[0154]

[0155] Among them, δ n,m is the small-scale fading with Rayleigh distribution, G k ,G s are the gains of the GS antenna and the satellite antenna, respectively, and f c is the carrier frequency, c is the speed of light, d s is the distance between GS and the low-Earth orbit satellite.

[0156] Satisfies the following expression:

[0157]

[0158] Among them, β n,m is the large-scale fading component, is the small-scale fading of the Rice distribution. The large-scale fading β n,m The model is: β n,m ()=-120.9-37.6log 10 ( n,m )+10log 10 (), Among them, d n,m is the distance between T-cell BS and GS (km), z is A lognormal random variable of

[0159] The private flow rate of the nth GS on the mth RB satisfies the following expression:

[0160]

[0161] The common flow rate on the mth RB satisfies the following expression:

[0162]

[0163] S502: Determine a reward value based on the public flow rate and the private flow rate.

[0164] The reward value is the sum of the public flow rate and the private flow rate.

[0165] In some embodiments, the reward value is the sum of the public flow rate and all private flow rates, and the reward value satisfies the following expression:

[0166]

[0167] in, is the common flow rate of the mth resource block, It is the sum of the private flow rates of all GSs on the mth resource block.

[0168] S503: storing the state parameters, target action decisions, reward values, and next state parameters as experience samples in an experience sample database.

[0169] In some embodiments, the state parameter represents the state of the current communication system or environment, which may include but is not limited to the current network conditions, resource usage, data stream queue length, etc. The target action decision is the decision made by the communication system based on the current state, such as adjusting the priority of data transmission, allocating more bandwidth to a certain type of data stream, etc. The reward value is the result evaluation of the current action decision; the next state parameter is the new state reached by the communication system after the current action is executed. These experience samples are stored in the experience sample database, that is, the playback buffer or experience pool, for subsequent model training.

[0170] It should be understood that in reinforcement learning, this storage and playback mechanism helps to break the correlation between data and improve the stability and efficiency of model training.

[0171] In some embodiments, Figure 7 As shown, the resource allocation method based on the deep deterministic policy gradient algorithm provided in the embodiment of the present application also includes the following S601-S604:

[0172] S601: In response to the number of experience samples reaching a third preset threshold, determine the priority of the experience samples based on a priority mechanism.

[0173] In some embodiments, when the number of experience samples in the experience sample database reaches a third preset threshold, in order to improve the model learning efficiency, a priority mechanism may be introduced, and each experience sample may be scored or ranked based on the priority mechanism. The priority may be based on a variety of factors, such as the novelty of the sample, the reward value of the sample, the contribution of the sample to the model improvement, etc.

[0174] In some embodiments, the priority mechanism is a priority mechanism based on the timing differential error and the Q value. The priority mechanism based on the timing differential error and the Q value satisfies the following expression:

[0175]

[0176] Where P(i) is the priority of the i-th experience sample based on the priority mechanism, μ TD is the average value of the time series difference error of all empirical samples, σ TD is the standard deviation of the time series difference error of all empirical samples, P TD (i) is the priority of the i-th experience sample calculated based on the time series difference error, P Q (i) is the priority of the i-th experience sample calculated based on the Q value, and ε is a small positive value to ensure that P(i)>0;

[0177] The priority of the i-th experience sample calculated based on the time series difference error satisfies the following expression:

[0178]

[0179] The priority of the i-th experience sample calculated based on the Q value satisfies the following expression:

[0180]

[0181] Among them, the softsign function and ISRU function are used to normalize the priority, δ i is the time series difference error of the i-th experience sample, s i is the state parameter in the i-th experience sample, a i It is the target action decision corresponding to the state parameter.

[0182] It should be noted that prioritized experience replay (PER) is a technique for improving sample efficiency in reinforcement learning. It optimizes the learning process by giving different priorities to experience samples. In PER, when the TD Error of an experience sample is large, it means that the sample contains a large amount of information, so these experience samples are given higher priority and are more likely to be selected for training. However, this method has an overfitting problem because samples with high TD Error are oversampled. In DDPG, the Q value represents the expected total reward from taking an action from the current state to the final state, so the size of the Q value can be used as another criterion for evaluating sample priority. By combining TDError and Q value to calculate the priority, the weight of TD Error in the priority calculation can be dynamically adjusted. When the standard deviation of TD Error is large, the weight of TD Error can be reduced, and more attention can be paid to Q value to reduce the impact of noise and improve the stability of learning. In addition, in PER, TD error is used as a priority throughout the training stage, but at the beginning of training, the TD error of experience is large, and the advantage of this priority replay mechanism is not obvious. Therefore, in the early stage of training, paying more attention to the Q value of the experience sample, and paying attention to the TD error of the experience sample in the later stage will be more conducive to the role of the sample.

[0183] S602: Determine an experience sample group based on the priority of the experience samples.

[0184] In some embodiments, the experience sample group is a set of samples used for training. After the priorities of all experience samples are updated, experience extraction is performed based on the priorities, and a certain number of experience samples with higher priorities are selected to form an experience sample group for subsequent model training.

[0185] S603: Training a deep reinforcement learning model based on the experience sample group.

[0186] In some embodiments, the experience sample group selected in S602 is used to train or update the deep reinforcement learning model by inputting these experience samples into the model and adjusting the parameters of the model according to the output of the model and the actual reward.

[0187] In some embodiments, the deep reinforcement learning model is trained using gradient ascent and gradient descent methods, and the target policy network and the target Q network are updated using a soft update method, introducing a learning rate, taking a weighted average of the old network parameters and the new corresponding network parameters, and then assigning them to the target policy network and the target Q network.

[0188] S604: In response to the deep reinforcement learning model reaching a second preset condition, end training.

[0189] Among them, the second preset condition includes at least one of the following: the number of training times reaches a fourth preset threshold and the expected value of the action decision output by the deep reinforcement learning model is greater than or equal to a fifth preset threshold.

[0190] In some embodiments, the deep reinforcement learning model is iteratively trained until the network converges and the iteration ends.

[0191] In some embodiments, when the number of deep reinforcement learning model training reaches a fourth preset threshold, or the expected value of the action decision output by the deep reinforcement learning model is greater than or equal to a fifth preset threshold, training is stopped to avoid unnecessary calculations and resource consumption.

[0192] The resource allocation method and related equipment based on the deep deterministic policy gradient algorithm provided in this application are aimed at the problem that the priority evaluation mechanism based on TD error has repeated sampling of high TD error experience, which easily leads to overfitting. A solution combining Q value calculation priority is proposed. In the early stage of training, more attention is paid to the Q value of the experience sample, and in the later stage, attention is paid to the TD error of the experience sample, which makes full use of the effectiveness of the sample, improves the network learning speed and algorithm performance, and makes the DDPG algorithm play a more effective role in different training stages.

[0193] Exemplarily, the resource allocation method based on the deep deterministic policy gradient algorithm provided in the embodiment of the present application includes the following steps:

[0194] Step 1: Initialize the DDPG neural network. The policy network and the target policy network have the same structure, including an input layer, 4 hidden layers and an output layer. The hidden layer contains 48 neural network nodes, using a fully connected method and using ReLU as the activation function. The Q network has the same structure as the target Q network, including an input layer, 4 hidden layers and an output layer. The hidden layer contains 48 neural network nodes, using a fully connected method and using ReLU as the activation function. The number of individuals in the population of the sand cat swarm optimization algorithm is set to 20, and the maximum number of iterations is set to 50. After initializing the individuals, the fitness of each individual is calculated to obtain [6.4…3.2]. The sand cat swarm optimization algorithm is used for iteration. Each iteration process updates the position of the individual, that is, the parameters of the policy network, and records the optimal policy network parameters under the current generation. After the iteration is completed, the final optimal policy parameters [5.3…4.7] (expand the input layer, hidden layer, and output layer parameters) are obtained and copied to the DDPG policy network and target policy network.

[0195] Step 2: In the communication system, consider a satellite fusion network with 2 LEO satellites and 3 T-cells, that is, M = 3, with a total of 5 GSs. Each RB can support 3 GSs to access at the same time, that is, N = 5, K = 3. The satellite station obtains the current environmental status s m (t) = [8.4…7.2], input to the actor network to get action a m (t) = [0…2.3], where the first N dimensions represent the resource block allocation scheme, and the last N+1 dimensions represent the power allocation scheme. The satellite station executes the RB allocation and power control strategy according to the action, pre-codes the user information and transmits it in the same time-frequency resource through RSMA. The ground station receiver decodes the total received signal, calculates the public and private stream SINRs respectively, and returns the results to the satellite station. The satellite station calculates the reward value r for this time m (t+1)=24.7. Further, the experience samples ([8.4…7.2], [0…2.3], 24.7, [8.3…7.5]) are stored in the playback buffer pool.

[0196] Step 3: Prioritize the samples in the experience pool. The TD error of all samples is [12.5…17.2], the Q value is [32.1…47.8], and the average TD error μ within 5 times is TD is [14.0…18.3], standard deviation σ TD =[2.8…2.4], calculate the priority of each experience sample and extract experience according to the priority to obtain 16 experience groups The extracted experience group is used to train the network, and the Q network and policy network are updated using gradient descent and gradient ascent methods respectively. The target Q and target policy network parameters are updated using a soft update method with a learning rate of τ = 0.1.

[0197] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the described method.

[0198] It should be noted that the above describes some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0199] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a resource allocation device based on a deep deterministic policy gradient algorithm.

[0200] refer to Figure 8 , the resource allocation device based on the deep deterministic policy gradient algorithm is applied to a satellite station, comprising: an acquisition module 801, a generation module 802, a determination module 803 and an execution module 804;

[0201] The acquisition module 801 is used to acquire a state parameter; the state parameter is used to characterize the channel environment state of the satellite station;

[0202] The generating module 802 is used to input the state parameters into a deep reinforcement learning model to generate at least one action decision; the deep reinforcement learning model is trained based on a deep deterministic policy gradient algorithm and a sand cat swarm optimization algorithm; the action decision is used to indicate resource allocation and power control strategies;

[0203] The determination module 803 is used to calculate the expected value of the at least one action decision, and determine the action decision with the largest expected value as the target action decision;

[0204] The execution module 804 is used to execute the resource allocation and power control strategy based on the target action decision.

[0205] In some embodiments, the deep reinforcement learning model includes: a policy network, a target policy network, a Q network, and a target Q network, and the resource allocation device based on the deep deterministic policy gradient algorithm also includes an optimization module 805 and a processing module 806;

[0206] The acquisition module 801 is also used to acquire multiple strategy network individuals in the initial model;

[0207] The optimization module 805 is used to iteratively optimize the multiple strategy network individuals based on the sand cat swarm optimization algorithm to obtain multiple target strategy network individuals;

[0208] The processing module 806 is used to calculate the fitness of the multiple target strategy network individuals and save the target strategy network individual with the highest fitness into the initial model;

[0209] The determination module 803 is also used to determine that the initial model is the deep reinforcement learning model in response to the target strategy network individual reaching a first preset condition; the first preset condition includes at least one of the following: the number of iterative optimizations reaches a first preset threshold and the fitness is greater than or equal to a second preset threshold.

[0210] In some embodiments, the determination module 803 is specifically used to input at least one state-action pair into the Q network to obtain an expected value of the at least one state-action pair; the state-action pair includes the state parameters and the corresponding action decision; in response to the expected value of the state-action pair being the largest, determine the action decision in the state-action pair as the target decision.

[0211] In some embodiments, the resource allocation device based on the deep deterministic policy gradient algorithm further includes a sending module 807;

[0212] The acquisition module 801 is also used to acquire user data;

[0213] The sending module 807 is used to divide the user data into public stream data and private stream data based on the resource extended multiple access transmission mechanism, and send the data to the ground station.

[0214] In some embodiments, the determination module 803 is further used to determine a public flow rate and a private flow rate;

[0215] The determination module 803 is further configured to determine a reward value based on the public flow rate and the private flow rate; the reward value is the sum of the public flow rate and the private flow rate;

[0216] The processing module 806 is further configured to store the state parameter, the target action decision, the reward value, and the next state parameter as experience samples in an experience sample database.

[0217] In some embodiments, the resource allocation device based on the deep deterministic policy gradient algorithm further includes a training module 808;

[0218] The determining module 803 is further configured to, in response to the number of the experience samples reaching a third preset threshold, determine the priority of the experience samples based on a priority mechanism;

[0219] The determination module 803 is further configured to determine an experience sample group based on the priority of the experience sample;

[0220] The training module 808 is used to train the deep reinforcement learning model based on the experience sample group;

[0221] The training module 808 is also used to end training in response to the deep reinforcement learning model reaching a second preset condition; the second preset condition includes at least one of the following: the number of training times reaches a fourth preset threshold and the expected value of the action decision output by the deep reinforcement learning model is greater than or equal to a fifth preset threshold.

[0222] In some embodiments, the priority mechanism is a priority mechanism based on timing differential error and Q value;

[0223] The priority mechanism based on the timing difference error and the Q value satisfies the following expression:

[0224]

[0225] Wherein, P(i) is the priority of the i-th experience sample based on the priority mechanism, μ TD is the average value of the time series difference error of all empirical samples, σ TD is the standard deviation of the time series difference error of all empirical samples, P TD (i) is the priority of the i-th empirical sample calculated based on the time series difference error, P Q (i) is the priority of the i-th experience sample calculated based on the Q value, and ε is a small positive value to ensure that P(i)>0;

[0226] The priority of the i-th experience sample calculated based on the timing difference error satisfies the following expression:

[0227]

[0228] The priority of the i-th experience sample calculated based on the Q value satisfies the following expression:

[0229]

[0230] Among them, the softsign function and ISRU function are used to normalize the priority, δ i is the time series difference error of the i-th empirical sample, s i is the state parameter in the i-th experience sample, a i The target action decision is made for the state parameter.

[0231] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0232] The device of the above embodiment is used to implement the corresponding resource allocation method based on the deep deterministic policy gradient algorithm in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0233] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the resource allocation method based on the deep deterministic policy gradient algorithm described in any of the above embodiments is implemented.

[0234] Fig. 9 A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0235] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0236] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0237] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0238] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0239] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0240] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0241] The electronic device of the above embodiment is used to implement the corresponding resource allocation method based on the deep deterministic policy gradient algorithm in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0242] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the resource allocation method based on the deep deterministic policy gradient algorithm as described in any of the above embodiments.

[0243] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0244] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the resource allocation method based on the deep deterministic policy gradient algorithm as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0245] Based on the same inventive concept, corresponding to the resource allocation method based on the deep deterministic policy gradient algorithm described in any of the above embodiments, the present disclosure also provides a computer program product, which includes a computer program. In some embodiments, the computer program is executable by one or more processors so that the processor executes the resource allocation method based on the deep deterministic policy gradient algorithm. Corresponding to the execution subject corresponding to each step in each embodiment of the resource allocation method based on the deep deterministic policy gradient algorithm, the processor that executes the corresponding step may belong to the corresponding execution subject.

[0246] The computer program product of the above embodiment is used to enable the processor to execute the resource allocation method based on the deep deterministic policy gradient algorithm as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0247] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0248] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0249] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0250] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A resource allocation method based on a deep deterministic policy gradient algorithm, characterized in that: Applied to a satellite station, the method comprises: Acquiring state parameters; the state parameters are used to characterize the channel environment state of the satellite station; Inputting the state parameters into a deep reinforcement learning model to generate at least one action decision; the deep reinforcement learning model is trained based on a deep deterministic policy gradient algorithm and a sand cat swarm optimization algorithm; the action decision is used to indicate resource allocation and power control strategies; Calculating an expected value of at least one action decision, and determining the action decision with the largest expected value as a target action decision; Based on the target action decision, the resource allocation and power control strategy is executed.

2. The method according to claim 1, characterized in that The deep reinforcement learning model includes: a policy network, a target policy network, a Q network and a target Q network, and the method further includes: Obtain multiple strategy network individuals in the initial model; Iteratively optimize the multiple strategy network individuals based on the sand cat swarm optimization algorithm to obtain multiple target strategy network individuals; Calculating the fitness of the multiple target strategy network individuals, and saving the target strategy network individual with the highest fitness into the initial model; In response to the target strategy network individual reaching a first preset condition, determining that the initial model is the deep reinforcement learning model; the first preset condition includes at least one of the following: the number of iterative optimizations reaches a first preset threshold and the fitness is greater than or equal to a second preset threshold.

3. The method according to claim 2, characterized in that The calculating the expected value of the at least one action decision and determining the action decision with the largest expected value as the target action decision includes: Inputting at least one state-action pair into the Q network to obtain an expected value of the at least one state-action pair; the state-action pair includes the state parameter and the corresponding action decision; In response to the expected value of the state-action pair being the largest, the action decision in the state-action pair is determined to be the target decision.

4. The method according to claim 1, characterized in that After executing the resource allocation and power control strategy based on the target action decision, the method further includes: Get user data; Based on the resource extension multiple access transmission mechanism, the user data is divided into public stream data and private stream data, and sent to the ground station.

5. The method according to claim 4, characterized in that The method further comprises: Determining public flow rates and private flow rates; Determine a reward value based on the public flow rate and the private flow rate; the reward value is the sum of the public flow rate and the private flow rate; The state parameter, the target action decision, the reward value, and the next state parameter are stored as experience samples in an experience sample database.

6. The method according to claim 5, characterized in that The method further comprises: In response to the number of the experience samples reaching a third preset threshold, determining the priority of the experience samples based on a priority mechanism; Determining an experience sample group based on the priorities of the experience samples; Based on the experience sample group, training the deep reinforcement learning model; In response to the deep reinforcement learning model reaching a second preset condition, training is terminated; the second preset condition includes at least one of the following: the number of training times reaches a fourth preset threshold and the expected value of the action decision output by the deep reinforcement learning model is greater than or equal to a fifth preset threshold.

7. The method according to claim 6, characterized in that The priority mechanism is a priority mechanism based on timing differential error and Q value; The priority mechanism based on the timing difference error and the Q value satisfies the following expression: Wherein, P(i) is the priority of the i-th experience sample based on the priority mechanism, μ TD is the average value of the time series difference error of all empirical samples, σ TD is the standard deviation of the time series difference error of all empirical samples, P TD (i) is the priority of the i-th empirical sample calculated based on the time series difference error, P Q (i) is the priority of the i-th experience sample calculated based on the Q value, and ε is a small positive value to ensure that P(i)>0; The priority of the i-th experience sample calculated based on the timing difference error satisfies the following expression: The priority of the i-th experience sample calculated based on the Q value satisfies the following expression: Among them, the softsign function and ISRU function are used to normalize the priority, δ i is the time series difference error of the i-th empirical sample, s i is the state parameter in the i-th experience sample, a i The target action decision is made for the state parameter.

8. A resource allocation device based on a deep deterministic policy gradient algorithm, characterized in that: Applied to a satellite station, the device comprises: an acquisition module, a generation module, a determination module and an execution module; The acquisition module is used to acquire state parameters; the state parameters are used to characterize the channel environment state of the satellite station; The generation module is used to input the state parameters into a deep reinforcement learning model to generate at least one action decision; the deep reinforcement learning model is trained based on a deep deterministic policy gradient algorithm and a sand cat swarm optimization algorithm; the action decision is used to indicate resource allocation and power control strategies; The determination module is used to calculate the expected value of the at least one action decision, and determine the action decision with the largest expected value as the target action decision; The execution module is used to execute the resource allocation and power control strategy based on the target action decision.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program. 10 . A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to claim 1 .

Images