Resource allocation method based on deep deterministic policy gradient algorithm and related equipment

Abstract

The application provides a resource allocation method based on a deep deterministic policy gradient algorithm and related equipment, the method comprising: obtaining a state parameter; the state parameter is used to represent a channel environment state of the satellite station; inputting the state parameter into a deep reinforcement learning model to generate at least one action decision; the deep reinforcement learning model is trained based on a deep deterministic policy gradient algorithm and a sand cat group optimization algorithm; the action decision is used to indicate a resource allocation and power control strategy; calculating an expected value of the at least one action decision, and determining that the action decision with the maximum expected value is a target action decision; based on the target action decision, the resource allocation and power control strategy are executed. The application solves the joint optimization problem of resource allocation and power control by using a deep reinforcement learning model, thereby improving the resource allocation efficiency and optimizing the network performance.

Description

Technical Field

[0001] The present application relates to the field of satellite communications, and in particular to a resource allocation method and related equipment based on a deep deterministic policy gradient algorithm. Background Art

[0002] As the foundation of the national economy, the safe and stable operation of the power industry is crucial to national livelihoods and security. With the increase in distributed energy resources and remote power plants, power systems require communications technologies that can provide wide-area coverage and transmit critical information in real time to ensure efficiency and reliability.

[0003] While satellite communications offer wide-area coverage, their limited spectrum resources make it difficult to meet the power industry's demands for seamless connectivity and high-throughput services. Furthermore, as the convergence of satellite and terrestrial networks becomes a key trend in future communications networks, it also faces challenges such as frequency resource overlap, insufficient security, and difficulty ensuring reliability. Summary of the Invention

[0004] In view of this, the purpose of this application is to propose a resource allocation method and related equipment based on a deep deterministic policy gradient algorithm.

[0005] As one aspect of the present application, a resource allocation method based on a deep deterministic policy gradient algorithm is provided, which is applied to a satellite station, including:

[0006] Acquiring state parameters; the state parameters are used to characterize the channel environment state of the satellite station;

[0007] Inputting the state parameters into a deep reinforcement learning model to generate at least one action decision; the deep reinforcement learning model is trained based on a deep deterministic policy gradient algorithm and a sand cat swarm optimization algorithm; the action decision is used to indicate resource allocation and power control strategies;

[0008] Calculating an expected value of the at least one action decision, and determining the action decision with the largest expected value as a target action decision;

[0009] Based on the target action decision, the resource allocation and power control strategy is executed.

[0010] Optionally, the deep reinforcement learning model includes: a policy network, a target policy network, a Q network, and a target Q network, including:

[0011] Obtain multiple policy network individuals in the initial model;

[0012] Iteratively optimize the multiple strategy network individuals based on the sand cat swarm optimization algorithm to obtain multiple target strategy network individuals;

[0013] calculate fitness of the plurality of target policy network individuals, save a target policy network individual with the highest fitness to the initial model;

[0014] In response to the target policy network individual reaching a first preset condition, determine that the initial model is the deep reinforcement learning model; the first preset condition includes at least one of the following: the number of iterations of optimization reaches a first preset threshold, and the fitness is greater than or equal to a second preset threshold.

[0015] Optionally, the calculation of the expected value of the at least one action decision includes:

[0016] input at least one state-action pair into the Q network to obtain an expected value of the at least one state-action pair; the state-action pair includes the state parameter and the corresponding action decision;

[0017] In response to the expected value of the state-action pair being the largest, determine the action decision in the state-action pair as the target decision.

[0018] Optionally, after the resource allocation and power control strategy are executed based on the target action decision, the method includes:

[0019] obtain user data;

[0020] based on a resource expansion multiple access transmission mechanism, divide the user data into common flow data and private flow data, and send to a ground station.

[0021] Optionally, determine a common flow rate and a private flow rate; based on the common flow rate and the private flow rate, determine a reward value; the reward value is the sum of the common flow rate and the private flow rate; store the state parameter, the target action decision, the reward value, and a next state parameter as an experience sample into an experience sample database.

[0022] Optionally, in response to the number of experience samples reaching a third preset threshold, based on a priority mechanism, determine the priority of the experience samples; based on the priority of the experience samples, determine an experience sample group; based on the experience sample group, train the deep reinforcement learning model; in response to the deep reinforcement learning model reaching a second preset condition, end the training; the second preset condition includes at least one of the following: the number of training reaches a fourth preset threshold, and the expected value of the action decision output by the deep reinforcement learning model is greater than or equal to a fifth preset threshold.

[0023] Optionally, the priority mechanism is a priority mechanism based on a time difference error and a Q value;

[0024] The priority mechanism based on the timing difference error and the Q value satisfies the following expression:

[0025]

[0026] Wherein, P(i) is the priority of the i-th experience sample based on the priority mechanism, μ TD is the average value of the timing difference error of all experience samples, σ TD is the standard deviation of the timing difference error of all experience samples, P TD (i) is the priority of the i-th experience sample calculated based on the timing difference error, P Q (i) is the priority of the i-th experience sample calculated based on the Q value, and ε is a small positive value to ensure P(i)>0.

[0027] The priority of the i-th experience sample calculated based on the timing difference error satisfies the following expression:

[0028]

[0029] The priority of the i-th experience sample calculated based on the Q value satisfies the following expression:

[0030]

[0031] Wherein, the softsign function and the ISRU function are used to normalize the priority, δ i is the timing difference error of the i-th experience sample, s i is the state parameter in the i-th experience sample, a i is the target action decision corresponding to the state parameter.

[0032] As a second aspect of the present application, a resource allocation device based on a deep deterministic policy gradient algorithm is provided, applied to a satellite station, comprising: an acquisition module, a generation module, a determination module and an execution module;

[0033] The acquisition module is configured to acquire a state parameter; the state parameter is used to represent the channel environment state of the satellite station;

[0034] The generation module is configured to input the state parameter into a deep reinforcement learning model to generate at least one action decision; the deep reinforcement learning model is trained based on a deep deterministic policy gradient algorithm and a sand cat group optimization algorithm; and the action decision is used to indicate a resource allocation and power control strategy;

[0035] The determination module is configured to calculate an expected value of the at least one action decision, and determine the action decision with the largest expected value as the target action decision;

[0036] The execution module is configured to execute the resource allocation and power control strategy based on the target action decision.

[0037] Optionally, the deep reinforcement learning model includes: a policy network, a target policy network, a Q network, and a target Q network, and the resource allocation device based on the deep deterministic policy gradient algorithm further includes an optimization module and a processing module;

[0038] The acquisition module is further used to acquire multiple strategy network individuals in the initial model;

[0039] The optimization module is used to iteratively optimize the multiple policy network individuals based on the sand cat swarm optimization algorithm to obtain multiple target policy network individuals;

[0040] The processing module is used to calculate the fitness of the multiple target strategy network individuals and save the target strategy network individual with the highest fitness into the initial model;

[0041] The determination module is further used to determine that the initial model is the deep reinforcement learning model in response to the target policy network individual meeting a first preset condition; the first preset condition includes at least one of the following: the number of iterative optimizations reaches a first preset threshold and the fitness is greater than or equal to a second preset threshold.

[0042] Optionally, the determination module is specifically used to input at least one state-action pair into the Q network to obtain an expected value of the at least one state-action pair; the state-action pair includes the state parameters and the corresponding action decision; in response to the expected value of the state-action pair being the largest, determine the action decision in the state-action pair as the target decision.

[0043] Optionally, the resource allocation device based on the deep deterministic policy gradient algorithm further includes a sending module;

[0044] The acquisition module is also used to acquire user data;

[0045] The sending module is used to divide the user data into public flow data and private flow data based on the resource extended multiple access transmission mechanism, and send the data to the ground station.

[0046] Optionally, the determining module is further configured to determine a public flow rate and a private flow rate;

[0047] The determining module is further configured to determine a reward value based on the public flow rate and the private flow rate; the reward value is the sum of the public flow rate and the private flow rate;

[0048] The processing module is further configured to store the state parameter, the target action decision, the reward value, and the next state parameter as experience samples in an experience sample database.

[0049] Optionally, the resource allocation device based on the deep deterministic policy gradient algorithm further includes a training module;

[0050] The determining module is further configured to, in response to the number of the experience samples reaching a third preset threshold, determine the priority of the experience samples based on a priority mechanism;

[0051] The determining module is further configured to determine an experience sample group based on the priority of the experience sample;

[0052] The training module is used to train the deep reinforcement learning model based on the experience sample group;

[0053] The training module is also used to end training in response to the deep reinforcement learning model reaching a second preset condition; the second preset condition includes at least one of the following: the number of training times reaches a fourth preset threshold and the expected value of the action decision output by the deep reinforcement learning model is greater than or equal to a fifth preset threshold.

[0054] Optionally, the priority mechanism is a priority mechanism based on timing differential error and Q value;

[0055] The priority mechanism based on the timing difference error and the Q value satisfies the following expression:

[0056]

[0057] Wherein, P(i) is the priority of the i-th experience sample based on the priority mechanism, μ TD is the average value of the time series difference error of all empirical samples, σ TD is the standard deviation of the time series difference error of all empirical samples, P TD (i) is the priority of the i-th empirical sample calculated based on the time series difference error, P Q (i) is the priority of the i-th experience sample calculated based on the Q value, and ε is a small positive value to ensure that P(i)>0;

[0058] The priority of the i-th experience sample calculated based on the time series difference error satisfies the following expression:

[0059]

[0060] The priority of the i-th experience sample calculated based on the Q value satisfies the following expression:

[0061]

[0062] wherein the softsign function and the ISRU function are used to normalize the priority, and the δ i is a time difference error of the i-th experience sample, s i is a state parameter in the i-th experience sample, a i is the target action decision corresponding to the state parameter.

[0063] As a third aspect of the present application, an electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the above-mentioned resource allocation method based on the deep deterministic policy gradient algorithm when executing the program.

[0064] As a fourth aspect of the present application, a non-transitory computer readable storage medium is provided, which stores computer instructions for causing the computer to execute the above-mentioned resource allocation method based on the deep deterministic policy gradient algorithm provided by the present application.

[0065] From the above, it can be seen that the resource allocation method based on the deep deterministic policy gradient algorithm and the related device provided by the present application solve the problem of slow training speed caused by the scarcity of experience rewards in the early stage of training of the deep deterministic policy gradient algorithm by using the sand cat swarm optimization algorithm combined with the deep deterministic policy gradient algorithm (DDPG) to train a deep reinforcement learning model, improve the decision effect, and then input the satellite station state parameters into the deep reinforcement learning model to generate a resource allocation scheme in real time. Further, by predicting the execution effect of different action decisions, the action decision with the maximum expected value is selected as the optimal resource allocation scheme, thereby optimizing the resource allocation and power control strategy, improving the resource utilization and communication capacity of the satellite-ground network, and further improving the overall performance of the power communication system. BRIEF DESCRIPTION OF DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the present application or related art, the following will briefly introduce the drawings needed to be used in the embodiments or related art descriptions. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0067] Figure 1 A communication system schematic diagram of a resource allocation method based on a deep deterministic policy gradient algorithm provided by an embodiment of the present application;

[0068] Figure 2 A flowchart of a resource allocation method based on a deep deterministic policy gradient algorithm provided by an embodiment of the present application;

[0069] Figure 3 A flowchart of a resource allocation method based on a deep deterministic policy gradient algorithm provided by an embodiment of the present application;

[0070] Figure 4 A flowchart of a resource allocation method based on a deep deterministic policy gradient algorithm provided by an embodiment of the present application;

[0071] Figure 5 A flowchart of a resource allocation method based on a deep deterministic policy gradient algorithm provided by an embodiment of the present application;

[0072] Figure 6 A flowchart of a resource allocation method based on a deep deterministic policy gradient algorithm provided by an embodiment of the present application;

[0073] Figure 7 A flowchart of a resource allocation method based on a deep deterministic policy gradient algorithm provided by an embodiment of the present application;

[0074] Figure 8 A composition schematic diagram of a resource allocation device based on a deep deterministic policy gradient algorithm provided by an embodiment of the present application;

[0075] Figure 9 A composition schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to make the objectives, technical solutions and advantages of the present application clearer, further detailed explanations will be given below in combination with specific embodiments and with reference to the drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0077] It should be noted that, in the embodiments of the present application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or design. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a concrete way. Unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should have the usual meaning understood by people with ordinary skills in the field to which this application belongs. SUMMARY OF THE INVENTION

[0079] In related technologies, resource allocation between satellite and terrestrial networks and ratesplitting multiple access (RSMA) power control are non-deterministic polynomially hard (NP-hard) problems. Many studies have used heuristic algorithms and convex optimization methods to obtain suboptimal solutions. However, these methods rely on models and their effectiveness is heavily dependent on a comprehensive understanding of instantaneous channel state information (CSI). However, instantaneous CSI at the transmitter is difficult to obtain in real-world scenarios, especially for satellite communication systems. Due to the rapid channel time-varying and extended propagation time, CSI is often outdated. Given the dynamic and uncertain nature of the system, traditional methods cannot effectively optimize resource allocation and power control simultaneously with limited channel distribution information. Consequently, both the speed and quality of the solutions provided by these two methods are difficult to guarantee.

[0080] The inventors of this application discovered that, on the one hand, the RSMA-based power allocation method optimizes the power allocation of satellite base stations through reinforcement learning. However, because it uses a proximal strategy optimization algorithm, the training process is slow and only uses the previous round of strategy data, resulting in poor decision-making before the neural network converges. In addition, the frequency resource allocation problem between satellites and ground networks is not fully considered, and the actual application effect is limited. On the other hand, the method of using continuous convex approximation to solve non-convex optimization problems, although it can handle complex optimization problems, will introduce approximation errors, resulting in a gap between the solution and the optimal solution, and the computational complexity is high and the solution time is long, affecting the real-time performance and response speed of the system. At the same time, the convergence and stability of the algorithm vary in different environments and conditions, increasing the uncertainty of practical applications.

[0081] In order to solve the above problems, the present application provides a resource allocation method based on a deep deterministic policy gradient algorithm. A deep reinforcement learning model is obtained by using a sand cat swarm optimization algorithm combined with a deep deterministic policy gradient algorithm for training. The problem of slow training speed caused by the scarcity of experience rewards in the early stage of training of the deep deterministic policy gradient algorithm is solved, and the decision-making effect is improved. Furthermore, by inputting the satellite station state parameters into the deep reinforcement learning model, a resource allocation plan can be generated in real time. Furthermore, by predicting the execution effects of different action decisions, the action decision with the largest expected value is selected as the optimal resource allocation plan, thereby optimizing the resource allocation and power control strategies, improving the resource utilization and communication capacity of the satellite-to-ground network, and further improving the overall performance of the power communication system.

[0082] After introducing the basic principles of the present application, various non-limiting implementation methods of the present application are described in detail below.

[0083] Application Scenario Overview

[0084] Figure 1 A communication system diagram of a resource allocation method based on a deep deterministic policy gradient algorithm provided in an embodiment of the present application. Figure 1 As shown, the communication system includes a satellite station 101, a ground station 102 and a terminal 103. The satellite station 101 is connected to the ground station 102 via a wireless communication network, and the satellite station 101 is connected to the terminal 103 via a wireless communication network.

[0085] The satellite station 101 and ground station 102 mentioned here can be independent physical servers, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Optionally, the server can also be implemented on a cloud platform. For example, the cloud platform can include private clouds, public clouds, hybrid clouds, community clouds, distributed clouds, inter-clouds, and multi-clouds, etc., or any combination thereof. The embodiments of the present application are not limited to this.

[0086] In some embodiments, the terminal 103 includes but is not limited to a desktop computer, a mobile phone, a mobile computer, a tablet computer, a media player, a smart wearable device, a personal digital assistant (PDA) or other electronic devices capable of implementing the above functions.

[0087] In some embodiments, the terminal 103 is responsible for the collection and transmission of user data, such as voice, video, text information or sensor data, and converts the data into a format suitable for wireless transmission. The terminal 103 uploads user data to the satellite station 101 through a satellite communication link to realize remote communication.

[0088] In some embodiments, the satellite station 101 is responsible for the allocation of resources and the forwarding of data. The satellite station 101 receives user data from the terminal 103 and forwards the user data to the ground station 102 according to the resource allocation and power control strategy, and receives control signals and possible data updates from the ground station 102.

[0089] In some embodiments, the ground station 102 receives user data forwarded by the satellite station 101 and processes the received user data, including demodulation, decoding, verification and other steps, and then transmits the data to the final destination, such as the Internet or other communication networks. The ground station 102 can also send control instructions to the satellite station 101, including orbit adjustment, attitude control, fault diagnosis, etc., to ensure that the satellite station can operate stably and meet the communication requirements.

[0090] It should be understood that Figure 1 is an exemplary block diagram, Figure 1 The communication system shown includes a number of devices, such as a number of ground stations 102 and a number of terminals 103, which are not limited in number. In addition Figure 1 to the devices shown, Figure 1 The communication system shown can also include other devices, which are not limited.

[0091] The resource allocation method based on the deep deterministic policy gradient algorithm according to the exemplary embodiments of the present application will be described below in conjunction with the communication system Figure 1 . It should be noted that the above communication system is only shown for the purpose of facilitating the understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect. On the contrary, the embodiments of the present application can be applied to any applicable scenario.

[0092] Figure 2 A flowchart of a resource allocation method based on a deep deterministic policy gradient algorithm provided by an embodiment of the present application is shown. As Figure 2 shown, the resource allocation method based on the deep deterministic policy gradient algorithm provided by the present application can be implemented by the above satellite station 101, and specifically includes the following steps:

[0093] S201, obtaining a state parameter.

[0094] The state parameter is used to represent the channel environment state

[0095] In some embodiments, the channel environment state includes signal propagation characteristics such as path loss, Doppler shift, and shadowing, which can affect signal quality and communication system performance. State parameters can be acquired through real-time signal measurement and estimation, for example, using channel estimation techniques to obtain information such as the channel impulse response or path loss.

[0096] It should be noted that in satellite communication systems, state parameters refer to a series of measurements or data that characterize the current channel environment of a satellite station. These parameters may include, but are not limited to, signal strength, signal-to-noise ratio (SNR), bit error rate (BER), channel capacity, interference level, multipath effects, and Doppler shift. These parameters are crucial for assessing channel quality, predicting communication performance, and optimizing resource allocation and power control strategies.

[0097] In some embodiments, in a satellite fusion network, there are m terrestrial cells (T-Cells), each of which orthogonally occupies a resource block (RB) to provide services to multiple cellular users in the cell. At the same time, a low earth orbit (LEO) satellite provides downlink services to N ground stations (GS) by sharing the RBs of the m cells, where K (K) of the n GSs are connected to the ground. <n)个合并为一个GS组,并同时通过同一RB提供服务,以提高频谱效率,每个GS可以归属于多个组,并可以从多个RB上的低轨卫星那里接收服务。由于T-cell用户通常位于建筑物严重遮挡的区域,其天线增益通常会比GS低得多,因此,可以忽略来自低轨卫星的干扰,而考虑来自T-Cell基站(basestation,BS)的干扰。卫星基站需要感知当前环境状态,以便代理做出合适的决策。进一步的,将来自地面站的公共流和私有流的历史信号对干扰加噪声比(signal-to-interference-plus-noise ratio,SINR)定义为第m个RB的信道观测值,具体的,在时刻t第m个资源块的状态参数满足如下表达式:

[0098]

[0099] in, represents the common stream SINR observed by ground station n for the mth RB at time t-1, It represents the private flow SINR observed by ground station n for the mth RB at time t-1.

[0100] S202: Input the state parameters into a deep reinforcement learning model to generate at least one action decision.

[0101] Among them, the deep reinforcement learning model is trained based on the deep deterministic policy gradient algorithm and the sand cat swarm optimization algorithm; the action decision is used to indicate resource allocation and power control strategies.

[0102] It should be noted that the DDPG algorithm combines the actor-critic framework with the policy gradient method and is suitable for problems in continuous action spaces. By learning a deterministic policy, the DDPG algorithm can find the optimal action under given conditions. The sand cat swarm optimization algorithm (SCSO) is a swarm intelligence optimization algorithm inspired by the predatory behavior of sand cats. It solves optimization problems by simulating the cat's search and attack behavior, and has good optimization capabilities and fast convergence speed.

[0103] In some embodiments, state parameters are passed as input to a trained deep reinforcement learning model, which then uses a policy network to generate one or more action decisions based on the current state. These action decisions include resource allocation strategies (such as frequency band allocation and time slot allocation), power control strategies (such as adjusting transmit power), and coding and modulation scheme selection. Furthermore, the deep reinforcement learning model uses a neural network to predict the value of each action decision based on the current channel environment state and historical data, and selects the action decision that will bring the greatest expected value.

[0104] It should be understood that in communications systems, resource allocation involves allocating resources to different tasks, users, or devices based on specific rules or algorithms within limited resource constraints to optimize system performance and efficiency or meet specific requirements, such as improving throughput, reducing latency, or ensuring quality of service. Power control is the process of adjusting the power output of a device or system to meet performance requirements while reducing energy consumption, minimizing interference, and improving energy efficiency.

[0105] In some embodiments, the deep reinforcement learning model includes: a policy network, a target policy network, a Q network, and a target Q network, such as Figure 3 As shown, the resource allocation method based on the deep deterministic policy gradient algorithm provided in the embodiment of the present application also includes the following S301-S304:

[0106] S301: Acquire multiple strategy network individuals in the initial model.

[0107] In some embodiments, the policy network, the target policy network, the Q network and the target Q network of the DDPG are initialized using random parameters to obtain an initial model. The policy network (actor network) is used in deep reinforcement learning to generate actions in a given state. In practical applications, multiple policy network individuals are created by randomly initializing the weights and biases of the network, which serve as the starting point for subsequent optimization processes. Each individual has the ability to explore and learn the environment policy.

[0108] S302, based on the sand cat swarm optimization algorithm, multiple policy network individuals are iteratively optimized to obtain multiple target policy network individuals.

[0109] In some embodiments, the diversity of the SCSO algorithm can help the algorithm avoid local optima and possibly find reward signals in environments with sparse rewards. First, the policy network individuals in the SCSO algorithm are initialized, and then the exploration phase (searching for prey) and the development phase (attacking prey) of the algorithm correspond to the exploration and exploitation behaviors of the agent, respectively. By iteratively optimizing multiple policy network individuals, the SCSO algorithm continuously updates the network weights to improve the performance of the policy network and obtain multiple target policy network individuals.

[0110] S303, the fitness of multiple target policy network individuals is calculated, and the target policy network individual with the highest fitness is saved to the initial model.

[0111] In some embodiments, each target policy network individual is tested in the environment, and the cumulative reward or other performance indicators obtained by it within a certain time or a certain number of steps are recorded. Then, the fitness of all target policy network individuals is compared, the individual with the highest fitness is selected as the current optimal solution, and it is saved to the initial model to replace or supplement the original policy network.

[0112] For example, the population π pop of the SCSO algorithm includes multiple target policy network individuals. For the fitness of each target policy network individual in the population π pop , the target policy network individual is used to make action decisions a t according to the state parameters s t in an entire episode, and the reward value r t+1 obtained by the satellite station after executing the action decision is calculated as the fitness. From π popAfter obtaining the fitness of each target policy network individual, the target policy network individual with the highest fitness is selected. For each target policy network individual in the population, a random exploration angle θ is obtained using a roulette selection algorithm, and the exploration and development stage is carried out. The position of the target policy network individual is updated, i.e., the network parameters of the policy network are updated, using the position of the best target policy network individual, the position of the current target policy network individual, and the sensitivity range. Two parameter trackers are set during the iteration process of the algorithm, which are responsible for migrating the network parameters of the policy network. When the fitness satisfies the following expression, the network parameters of the policy network in the DDPG algorithm are migrated to the policy network in π pop the policy network with the minimum fitness in the network:

[0113]

[0114] where fit l is the fitness of the policy network individual in the DDPG algorithm, and N is the total number of target policy network individuals in π pop .

[0115] When the fitness satisfies the following expression, the network parameters of the policy network in π pop with the maximum fitness are migrated to the policy network in the DDPG algorithm:

[0116]

[0117] S304, in response to the target policy network individual reaching the first preset condition, determining that the initial model is a deep reinforcement learning model.

[0118] where the first preset condition includes at least one of the following: the number of iteration optimizations reaches a first preset threshold, and the fitness is greater than or equal to a second preset threshold.

[0119] In some embodiments, the target policy network is stored as an experience sample in the replay buffer of the DDPG, realizing the information flow from the evolutionary population to the neural network in the initial training stage. When the number of iteration optimizations reaches the first preset threshold or the fitness is greater than or equal to the second preset threshold, the evolutionary algorithm (EA) process is stopped, and the network parameters of the policy network in the DDPG algorithm at this time are copied to the target policy network. Then, the initial model currently containing the optimal policy network individual is determined as a deep reinforcement learning model, i.e., the initialization of the deep reinforcement learning model is completed.

[0120] S203, calculating the expected value of at least one action decision, and determining the action decision with the maximum expected value as the target action decision.

[0121] In some embodiments, the potential value of different action decisions in a given state, i.e., the expected value, is evaluated to select the action that will bring the greatest reward. The expected value calculation method may include: estimating the expected reward that the agent can obtain by following a specific strategy starting from a certain state through a state-value function; estimating the expected reward of each action by simulating random trajectories from the current state multiple times through Monte Carlo tree search; using model predictive control to use a system model to predict future states, solving an optimization problem at each step to select the action that maximizes future rewards; using a Q-network in a deep neural network to estimate the value of the action in the state by learning the value function of the state-action pair, etc.

[0122] In some embodiments, as Figure 4 As shown, S203 can be specifically implemented as follows S2031-S2032:

[0123] S2031. Input at least one state-action pair into the Q network to obtain an expected value of the at least one state-action pair.

[0124] The state-action pair includes state parameters and corresponding action decisions.

[0125] It should be noted that the Q network (action-value network) is a model that uses a neural network to approximate the Q function, which is used to estimate the expected return (expected value) that can be obtained by taking an action in a given state.

[0126] In some embodiments, each state-action pair is passed as input to the Q network, which processes these inputs through its internal neural network structure and outputs the expected value (Q value) of each state-action pair, where the Q value represents the expected reward that can be obtained by taking the action decision in the state.

[0127] For example, the satellite station inputs the current state parameters into the policy network and outputs the action decision. The action decision performed by the agent on the mth RB is a m ()=[ m (),p m ()], where α m ()=[ 1, (),…,α N,m ()] is an N-dimensional vector, which represents the distribution of the m-th RB at time step t, where α n,m ()=1 means the nth GS occupies the mth RB, otherwise α n,m ()=0. Similarly, p m ()=[ c,m (),p 1, (),…,p N,m()] is an N+1 dimensional vector, which represents the power allocation of GS on the mth RB. For n randomly generated action decisions in the action space Input them into the Q network to obtain the Q value of the state action pair {1,2,…, n}, where the action decision executed on the mth RB satisfies the following expression:

[0128]

[0129] The specific generation of random actions satisfies the following expression:

[0130] a ′ t =(a t );

[0131]

[0132] It should be understood that because DDPG enhances the algorithm's exploration capabilities by adding random perturbations to actions, the output actions may exceed the action boundary. The DDPG method sets boundary values ​​for actions that exceed the space limit, which causes the agent to repeatedly execute boundary actions, failing to effectively increase exploration. Therefore, this solution calculates the expected value of action decisions through the Q network, avoiding repeated execution of boundary actions while increasing the rationality of decision-making actions and enhancing exploration efficiency. Furthermore, for power control problems, the action space is multidimensional. Therefore, the above f(*) will be performed on each dimension, improving the performance of the DDPG algorithm in multidimensional action spaces.

[0133] S2032: In response to the expected value of the state-action pair being the largest, determine the action decision in the state-action pair as the target decision.

[0134] In some embodiments, after obtaining the expected values ​​of all state-action pairs, the expected values ​​of different action decisions are compared, and the action decision in the state-action pair with the largest expected value is selected as the target action decision to guide subsequent operations such as resource allocation and power control.

[0135] S204: Execute resource allocation and power control strategies based on the target action decision.

[0136] In some embodiments, based on the target action decision determined in S203, corresponding resource allocation and power control strategies are executed. For example, in a communication network, bandwidth allocation to a user may be increased or decreased to balance network load or improve user experience; or the power mode of a system component may be switched to a low-power mode or sleep mode to save energy or improve efficiency. Furthermore, after executing the target action decision, environmental changes and effects are continuously monitored to evaluate the effectiveness of the action decision.

[0137] The resource allocation method and related equipment based on the deep deterministic policy gradient algorithm provided in this application obtain a deep reinforcement learning model by using the sand cat swarm optimization algorithm combined with the deep deterministic policy gradient algorithm for training, thereby solving the problem of slow training speed caused by the scarcity of experience rewards in the early stage of training of the deep deterministic policy gradient algorithm, and improving the decision-making effect. Furthermore, by inputting the satellite station state parameters into the deep reinforcement learning model, a resource allocation plan can be generated in real time. Furthermore, by predicting the execution effects of different action decisions, the action decision with the largest expected value is selected as the optimal resource allocation plan, thereby optimizing the resource allocation and power control strategies, improving the resource utilization and communication capacity of the satellite-to-ground network, and further improving the overall performance of the power communication system.

[0138] In some embodiments, as Figure 5 As shown, after S204, the resource allocation method based on the deep deterministic policy gradient algorithm provided in the embodiment of the present application further includes the following S401-S402:

[0139] S401: Obtain user data.

[0140] In some embodiments, the satellite station collects user data from the terminal device through a wireless interface, such as voice, video, text information, file transfer or any other form of digital communication content. The collected user data will be temporarily stored in the local storage or cloud storage of the satellite station, waiting for further processing or transmission.

[0141] S402: Based on the resource extension multiple access transmission mechanism, the user data is divided into public flow data and private flow data, and sent to the ground station.

[0142] In some embodiments, the basic concept of Resource Extended Multiple Access (RSMA) is to divide each user's user data into two parts: public messages and private messages. Public messages of all users are encoded into a public stream data, which is decoded by multiple users, while private messages are independently encoded into private stream data, which is decoded by the corresponding user. The transmitter uses a superposition transmission method to superimpose all code streams for signal transmission. At the ground station receiving end, the total received signal is decoded. The public stream data is first decoded using the successive interference cancellation (SIC) technology, and then the other private stream data is treated as noise and its corresponding private stream data is decoded.

[0143] It should be understood that RSMA is an advanced multiple access scheme proposed for future 6G networks, which has been proven to have obvious performance advantages over non-orthogonal multiple access transmission mechanism (NOMA) and space division multiple access transmission mechanism (SDMA).

[0144] In some embodiments, as Figure 6As shown, the resource allocation method based on the deep deterministic policy gradient algorithm provided in the embodiment of the present application also includes the following S501-S503:

[0145] S501: Determine a public flow rate and a private flow rate.

[0146] In some embodiments, after the satellite station sends public and private stream data to the ground station, the ground station first decodes the public stream data using SIC technology, then subtracts it from the total received signal and considers the remaining signal to contain private stream data and noise. At this point, the ground station can decode its own private stream data and consider other private stream data to be noise. Furthermore, the public and private stream rates are calculated based on the decoded public and private stream data and the resource allocation scheme.

[0147] In some embodiments, the SINR of the public flow data and the SINR of the private flow data are first calculated. The SINR of the public flow data and the private flow data of the nth (n=1, ..., N) GS on the mth RB satisfies the following expression:

[0148] Public stream data SINR:

[0149]

[0150] Private flow data SINR:

[0151]

[0152] Among them, P T is the transmit power of T-Cell BS, σ 2 Represents the noise power spectral density. g n,m is the link gain between the nth GS and the LEO satellite on the mth RB, is the terrestrial interference link gain between the T-Cell BS using the mth RB and the nth (n=1, 2, ..., N) GS.

[0153] g n,m The rate satisfies the following expression:

[0154]

[0155] Among them, δ n,m is a small-scale fading with Rayleigh distribution, G k ,G s are the gains of GS antenna and satellite antenna respectively, f c is the carrier frequency, c is the speed of light, d s is the distance between GS and the low-Earth orbit satellite.

[0156] Satisfies the following expression:

[0157]

[0158] Among them, β n,m is the large-scale fading component, is the small-scale fading of the Rice distribution. The large-scale fading β n,m The model is: β n,m ()=-120.9-37.6log 10 ( n,m )+10log 10 (), Among them, d n,m is the distance between T-cell BS and GS (km), z is A lognormal random variable of

[0159] The private flow rate of the nth GS on the mth RB satisfies the following expression:

[0160]

[0161] The common flow rate on the mth RB satisfies the following expression:

[0162]

[0163] S502: Determine a reward value based on the public flow rate and the private flow rate.

[0164] The reward value is the sum of the public flow rate and the private flow rate.

[0165] In some embodiments, the reward value is the sum of the public flow rate and all private flow rates, and the reward value satisfies the following expression:

[0166]

[0167] in, is the common flow rate of the mth resource block, is the sum of the private flow rates of all GSs on the mth resource block.

[0168] S503: Store the state parameters, target action decision, reward value, and next state parameters as experience samples in the experience sample database.

[0169] In some embodiments, the state parameter represents the state of the current communication system or environment, which can include but is not limited to the current network conditions, resource usage, data flow queue length, etc.; the target action decision is the decision made by the communication system based on the current state, such as adjusting the priority of data transmission, allocating more bandwidth to a certain type of data flow, etc.; the reward value is the result evaluation of the current action decision; the next state parameter is the new state reached by the communication system after the current action is executed. These experience samples are stored in the experience sample database, i.e. the replay buffer or experience pool, for subsequent model training.

[0170] It should be understood that in reinforcement learning, such a storage and replay mechanism helps to break the correlation between data and improve the stability and efficiency of model training.

[0171] In some embodiments, as shown in Figure 7 The resource allocation method based on the deep deterministic policy gradient algorithm provided by the embodiments of the present application further includes the following S601-S604:

[0172] S601, in response to the number of experience samples reaching a third preset threshold, determining the priority of the experience samples based on a priority mechanism.

[0173] In some embodiments, when the number of experience samples in the experience sample database reaches the third preset threshold, in order to improve the efficiency of model learning, a priority mechanism can be introduced to score or sort each experience sample. Wherein, the priority can be based on multiple factors, such as the novelty of the sample, the reward value of the sample, the contribution of the sample to the improvement of the model, etc.

[0174] In some embodiments, the priority mechanism is a priority mechanism based on the temporal difference error and the Q value. The priority mechanism based on the temporal difference error and the Q value satisfies the following expression:

[0175]

[0176] Wherein, P(i) is the priority of the i th experience sample based on the priority mechanism, μ TD is the average value of the temporal difference error of all experience samples, σ TD is the standard deviation of the temporal difference error of all experience samples, P TD (i) is the priority of the i th experience sample calculated based on the temporal difference error, P Q (i) is the priority of the i th experience sample calculated based on the Q value, and ε is a small positive value to ensure P(i)>0;

[0177] The priority of the i th experience sample calculated based on the temporal difference error satisfies the following expression:

[0178]

[0179] The priority of the i-th experience sample calculated based on the Q value satisfies the following expression:

[0180]

[0181] Among them, the softsign function and ISRU function are used to normalize the priority, δ i is the time series difference error of the i-th experience sample, s i is the state parameter in the i-th experience sample, a i It is the target action decision corresponding to the state parameter.

[0182] It's important to note that prioritized experience replay (PER) is a technique for improving sample efficiency in reinforcement learning. It optimizes the learning process by assigning different priorities to experience samples. In PER, experience samples with larger TD errors indicate that they contain more information, so they are given higher priority and are more likely to be selected for training. However, this approach suffers from overfitting, as samples with high TD errors are oversampled. In DDPG, Q-values ​​represent the expected total reward from taking an action from the current state to the final state. Therefore, Q-values ​​can serve as another criterion for evaluating sample priority. Combining TD error and Q-values ​​to calculate priority allows for dynamic adjustment of the weight of TD error in the priority calculation. When the standard deviation of TD error is large, the weight of TD error can be reduced, and more emphasis can be placed on Q-values ​​to reduce the impact of noise and improve learning stability. Furthermore, in PER, TD error is used as a priority throughout training. However, at the beginning of training, when the TD error of experience is large, the advantages of this priority replay mechanism are not obvious. Therefore, in the early stage of training, paying more attention to the Q value of the experience sample, and paying attention to the TD error of the experience sample in the later stage is more conducive to the role of the sample.

[0183] S602: Determine an experience sample group based on the priority of the experience samples.

[0184] In some embodiments, the experience sample group is a set of samples used for training. After the priorities of all experience samples are updated, experience extraction is performed based on the priorities, and a certain number of experience samples with higher priorities are selected to form an experience sample group for subsequent model training.

[0185] S603: Based on the experience sample group, train a deep reinforcement learning model.

[0186] In some embodiments, the experience sample group selected in S602 is used to train or update the deep reinforcement learning model by inputting these experience samples into the model and adjusting the parameters of the model according to the output of the model and the actual reward.

[0187] In some embodiments, the deep reinforcement learning model is trained using gradient ascent and gradient descent methods, and the target policy network and the target Q network are updated using a soft update method, introducing a learning rate, taking a weighted average of the old network parameters and the new corresponding network parameters, and then assigning them to the target policy network and the target Q network.

[0188] S604: In response to the deep reinforcement learning model reaching a second preset condition, end training.

[0189] Among them, the second preset condition includes at least one of the following: the number of training times reaches a fourth preset threshold and the expected value of the action decision output by the deep reinforcement learning model is greater than or equal to a fifth preset threshold.

[0190] In some embodiments, the deep reinforcement learning model is iteratively trained until the network converges and the iteration ends.

[0191] In some embodiments, when the number of deep reinforcement learning model training times reaches a fourth preset threshold, or the expected value of the action decision output by the deep reinforcement learning model is greater than or equal to a fifth preset threshold, training is stopped to avoid unnecessary calculations and resource consumption.

[0192] This application provides a resource allocation method and related equipment based on a deep deterministic policy gradient algorithm. This method addresses the problem of repeated sampling of high TD error experiences in TD error-based priority evaluation mechanisms, which can easily lead to overfitting. By proposing a solution that combines Q-value calculation with priority, the method focuses more on the Q-value of the experience samples in the early stages of training and the TD error of the experience samples in the later stages. This fully utilizes the effectiveness of the samples, improves the network learning speed and algorithm performance, and enables the DDPG algorithm to function more effectively at different training stages.

[0193] Exemplarily, the resource allocation method based on the deep deterministic policy gradient algorithm provided in the embodiment of the present application includes the following steps:

[0194] The first step is to initialize the DDPG neural network, wherein the policy network and the target policy network have the same structure, including an input layer, four hidden layers, and an output layer, the hidden layers include 48 neural network nodes, use full connection, and use ReLU as the activation function. The Q network and the target Q network have the same structure, including an input layer, four hidden layers, and an output layer, the hidden layers include 48 neural network nodes, use full connection, and use ReLU as the activation function. The population size of the sand cat swarm optimization algorithm is set to 20, the maximum iteration number is set to 50, the fitness of each individual is calculated after the initialization of the individual to obtain [6.4…3.2], the sand cat swarm optimization algorithm is used for iteration, the position of the individual, that is, the parameter of the policy network, is updated each time, and the optimal policy network parameter under the current generation number is recorded, and the final optimal policy parameter [5.3…4.7] (the input layer, the hidden layer, and the output layer parameter are expanded) is obtained after the iteration is completed, which is copied to the policy network and the target policy network of the DDPG.

[0195] The second step is that in the communication system, a satellite fusion network with two LEO satellites and three T-cells is considered, that is, M=3, and there are a total of five GSs, and each RB can support three GSs to access simultaneously, that is, N=5 and K=3. The satellite station obtains the environmental state s m (t)=[8.4…7.2] is input into the actor network to obtain the action a m (t)=[0…2.3], wherein the first N dimensions represent the resource block allocation scheme, and the N+1 dimensions represent the power allocation scheme. The satellite station executes the RB allocation and power control strategy according to the action, pre-encodes the user information, and performs RSMA transmission in the same time-frequency resource. The ground station receives the total received signal, calculates the public stream and private stream SINRs respectively, and returns the results to the satellite station. The satellite station calculates the reward value r m (t+1)=24.7, further, the experience sample ([8.4…7.2], [0…2.3], 24.7, [8.3…7.5]) is stored in the replay buffer pool.

[0196] The third step is to update the priority of the samples in the experience pool, the TD error of all samples is [12.5…17.2], the Q value is [32.1…47.8], the average TD error μ TD in five times is [14.0…18.3], and the standard deviation σ TD is [2.8…2.4], the priority of each experience sample is calculated, and 16 experience groups are obtained by extracting the experience according to the priority. The network is trained using the extracted experience group, and the Q network and policy network are updated using gradient descent and gradient ascent methods respectively. The target Q and target policy network parameters are updated using a soft update method with a learning rate of τ = 0.1.

[0197] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied in a distributed scenario and performed by multiple devices working together. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the method.

[0198] It should be noted that the above description is limited to some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0199] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a resource allocation device based on a deep deterministic policy gradient algorithm.

[0200] refer to Figure 8 , the resource allocation device based on the deep deterministic policy gradient algorithm is applied to a satellite station, comprising: an acquisition module 801, a generation module 802, a determination module 803 and an execution module 804;

[0201] The acquisition module 801 is used to acquire state parameters; the state parameters are used to characterize the channel environment state of the satellite station;

[0202] The generating module 802 is configured to input the state parameters into a deep reinforcement learning model to generate at least one action decision; the deep reinforcement learning model is trained based on a deep deterministic policy gradient algorithm and a sand cat swarm optimization algorithm; the action decision is used to indicate resource allocation and power control strategies;

[0203] The determining module 803 is configured to calculate an expected value of the at least one action decision, and determine the action decision with the largest expected value as the target action decision;

[0204] The execution module 804 is configured to execute the resource allocation and power control strategy based on the target action decision.

[0205] In some embodiments, the deep reinforcement learning model includes: a policy network, a target policy network, a Q network, and a target Q network, and the resource allocation device based on the deep deterministic policy gradient algorithm further includes an optimization module 805 and a processing module 806;

[0206] The acquisition module 801 is further used to acquire multiple strategy network individuals in the initial model;

[0207] The optimization module 805 is used to iteratively optimize the multiple policy network individuals based on the sand cat swarm optimization algorithm to obtain multiple target policy network individuals;

[0208] The processing module 806 is used to calculate the fitness of the multiple target strategy network individuals and save the target strategy network individual with the highest fitness into the initial model;

[0209] The determination module 803 is further used to determine that the initial model is the deep reinforcement learning model in response to the target strategy network individual meeting a first preset condition; the first preset condition includes at least one of the following: the number of iterative optimizations reaches a first preset threshold and the fitness is greater than or equal to a second preset threshold.

[0210] In some embodiments, the determination module 803 is specifically used to input at least one state-action pair into the Q network to obtain an expected value of the at least one state-action pair; the state-action pair includes the state parameters and the corresponding action decision; in response to the expected value of the state-action pair being the largest, determine the action decision in the state-action pair as the target decision.

[0211] In some embodiments, the resource allocation device based on the deep deterministic policy gradient algorithm further includes a sending module 807;

[0212] The acquisition module 801 is also used to acquire user data;

[0213] The sending module 807 is used to divide the user data into public flow data and private flow data based on the resource extended multiple access transmission mechanism, and send the data to the ground station.

[0214] In some embodiments, the determining module 803 is further configured to determine a public flow rate and a private flow rate;

[0215] The determining module 803 is further configured to determine a reward value based on the public flow rate and the private flow rate; the reward value is the sum of the public flow rate and the private flow rate;

[0216] The processing module 806 is further configured to store the state parameter, the target action decision, the reward value, and the next state parameter as experience samples in an experience sample database.

[0217] In some embodiments, the resource allocation apparatus based on the deep deterministic policy gradient algorithm further includes a training module 808;

[0218] The determining module 803 is further configured to, in response to the number of the experience samples reaching a third preset threshold, determine the priority of the experience samples based on a priority mechanism;

[0219] The determining module 803 is further configured to determine an experience sample group based on the priority of the experience sample;

[0220] The training module 808 is used to train the deep reinforcement learning model based on the experience sample group;

[0221] The training module 808 is also used to end training in response to the deep reinforcement learning model reaching a second preset condition; the second preset condition includes at least one of the following: the number of training times reaches a fourth preset threshold and the expected value of the action decision output by the deep reinforcement learning model is greater than or equal to a fifth preset threshold.

[0222] In some embodiments, the priority mechanism is a priority mechanism based on timing differential error and Q value;

[0223] The priority mechanism based on the timing difference error and the Q value satisfies the following expression:

[0224]

[0225] Wherein, P(i) is the priority of the i-th experience sample based on the priority mechanism, μ TD is the average value of the time series difference error of all empirical samples, σ TD is the standard deviation of the time series difference error of all empirical samples, P TD (i) is the priority of the i-th empirical sample calculated based on the time series difference error, P Q (i) is the priority of the i-th experience sample calculated based on the Q value, and ε is a small positive value to ensure that P(i)>0;

[0226] The priority of the i-th experience sample calculated based on the time series difference error satisfies the following expression:

[0227]

[0228] The priority of the i-th experience sample calculated based on the Q value satisfies the following expression:

[0229]

[0230] wherein the softsign function and the ISRU function are used to normalize the priority, and the delta i is a time difference error of the i-th experience sample, s i is a state parameter in the i-th experience sample, a i is the target action decision corresponding to the state parameter.

[0231] For the convenience of description, the above apparatus is described in various modules in terms of functions. Of course, the functions of the modules can be implemented in one or more software and / or hardware in the implementation of the present application.

[0232] The apparatus of the above embodiment is used to implement the resource allocation method based on the deep deterministic policy gradient algorithm in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described here.

[0233] Based on the same inventive concept, corresponding to any of the above embodiment methods, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the resource allocation method based on the deep deterministic policy gradient algorithm according to any of the above embodiments.

[0234] Figure 9 A more specific hardware structure of an electronic device is shown, which can include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 for communication within the device.

[0235] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present application.

[0236] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0237] The input / output interface 1030 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0238] The communication interface 1040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0239] The bus 1050 comprises a pathway for transmitting information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0240] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0241] The electronic device of the above embodiment is used to implement the corresponding resource allocation method based on the deep deterministic policy gradient algorithm in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0242] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the resource allocation method based on the deep deterministic policy gradient algorithm as described in any of the above embodiments.

[0243] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0244] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the resource allocation method based on the deep deterministic policy gradient algorithm as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0245] Based on the same inventive concept, corresponding to the resource allocation method based on the deep deterministic policy gradient algorithm described in any of the above embodiments, the present disclosure also provides a computer program product, which includes a computer program. In some embodiments, the computer program is executable by one or more processors so that the processors execute the resource allocation method based on the deep deterministic policy gradient algorithm. Corresponding to the execution subject corresponding to each step in each embodiment of the resource allocation method based on the deep deterministic policy gradient algorithm, the processor that executes the corresponding step may belong to the corresponding execution subject.

[0246] The computer program product of the above embodiment is used to enable the processor to execute the resource allocation method based on the deep deterministic policy gradient algorithm as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0247] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. Within the scope of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0248] In addition, for simplicity of description and discussion, and in order not to make the embodiment of the application difficult to understand, the known power supply / ground connection with integrated circuit (IC) chip and other components may or may not be shown in the accompanying drawings provided. In addition, the device can be shown in the form of a block diagram to avoid making the embodiment of the application difficult to understand, and this also takes into account the following fact, that is, the details of the embodiment of these block diagram devices are highly dependent on the platform to be implemented in the embodiment of the application (that is, these details should be fully within the scope of understanding of those skilled in the art). When specific details (for example, circuit) are set forth to describe exemplary embodiments of the application, it will be apparent to those skilled in the art that the embodiment of the application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0249] Although the present invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may utilize the embodiments discussed.

[0250] The embodiments of the present application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of this application.

Claims

1. A resource allocation method based on a deep deterministic policy gradient algorithm, characterized in that: Applied to a satellite station, the method includes: Acquiring state parameters; the state parameters are used to characterize the channel environment state of the satellite station; Inputting the state parameters into a deep reinforcement learning model to generate at least one action decision; the deep reinforcement learning model is trained based on a deep deterministic policy gradient algorithm and a sand cat swarm optimization algorithm; the action decision is used to indicate resource allocation and power control strategies; Calculating an expected value of the at least one action decision, and determining the action decision with the largest expected value as a target action decision; executing the resource allocation and power control strategy based on the target action decision; Get user data; Based on a resource extended multiple access transmission mechanism, the user data is divided into public stream data and private stream data, and sent to a ground station; Determine public flow rate and private flow rate; Determining a reward value based on the public flow rate and the private flow rate; the reward value is the sum of the public flow rate and the private flow rate; storing the state parameter, the target action decision, the reward value, and the next state parameter as experience samples in an experience sample database; In response to the number of the experience samples reaching a third preset threshold, determining the priority of the experience samples based on a priority mechanism; determining an experience sample group based on the priorities of the experience samples; Training the deep reinforcement learning model based on the experience sample group; In response to the deep reinforcement learning model meeting a second preset condition, ending training; the second preset condition includes at least one of the following: the number of training times reaches a fourth preset threshold and the expected value of the action decision output by the deep reinforcement learning model is greater than or equal to a fifth preset threshold; Wherein, the priority mechanism is a priority mechanism based on timing differential error and Q value; The priority mechanism based on the timing difference error and the Q value satisfies the following expression: Wherein, P(i) is the priority of the i-th experience sample based on the priority mechanism, μ TD is the average value of the time series difference error of all empirical samples, σ TD is the standard deviation of the time series difference error of all empirical samples, P TD (i) is the priority of the i-th empirical sample calculated based on the time series difference error, P Q (i) is the priority of the i-th experience sample calculated based on the Q value, and ε is a small positive value to ensure that P(i)>0; The priority of the i-th experience sample calculated based on the time series difference error satisfies the following expression: The priority of the i-th experience sample calculated based on the Q value satisfies the following expression: Among them, the softsign function and ISRU function are used to normalize the priority, δ i is the time series difference error of the i-th empirical sample, s i is the state parameter in the i-th experience sample, a i The target action decision is made for the state parameter.

2. The method according to claim 1, characterized in that The deep reinforcement learning model includes: a policy network, a target policy network, a Q network, and a target Q network. The method further includes: Obtain multiple policy network individuals in the initial model; Iteratively optimize the multiple strategy network individuals based on the sand cat swarm optimization algorithm to obtain multiple target strategy network individuals; Calculating the fitness of the multiple target strategy network individuals, and saving the target strategy network individual with the highest fitness into the initial model; In response to the target strategy network individual meeting a first preset condition, determining that the initial model is the deep reinforcement learning model; the first preset condition includes at least one of the following: the number of iterative optimizations reaches a first preset threshold and the fitness is greater than or equal to a second preset threshold.

3. The method according to claim 2, characterized in that Calculating the expected value of the at least one action decision and determining the action decision with the largest expected value as the target action decision includes: Inputting at least one state-action pair into the Q network to obtain an expected value of the at least one state-action pair; the state-action pair includes the state parameter and the corresponding action decision; In response to the expected value of the state-action pair being the largest, the action decision in the state-action pair is determined to be the target action decision.

4. A resource allocation device based on a deep deterministic policy gradient algorithm, characterized in that: Applied to a satellite station, the device includes: an acquisition module, a generation module, a determination module, a sending module, an execution module, a processing module, and a training module; The acquisition module is used to acquire state parameters; the state parameters are used to characterize the channel environment state of the satellite station; The generation module is used to input the state parameters into a deep reinforcement learning model to generate at least one action decision; the deep reinforcement learning model is trained based on a deep deterministic policy gradient algorithm and a sand cat swarm optimization algorithm; the action decision is used to indicate resource allocation and power control strategies; The determination module is configured to calculate an expected value of the at least one action decision, and determine the action decision with the largest expected value as the target action decision; The execution module is used to execute the resource allocation and power control strategy based on the target action decision; The acquisition module is also used to acquire user data; The sending module is used to divide the user data into public flow data and private flow data based on the resource extended multiple access transmission mechanism, and send the data to the ground station; The determining module is further configured to determine a public flow rate and a private flow rate; The determining module is further configured to determine a reward value based on the public flow rate and the private flow rate; the reward value is the sum of the public flow rate and the private flow rate; The processing module is used to store the state parameter, the target action decision, the reward value, and the next state parameter as experience samples in an experience sample database; The determining module is further configured to, in response to the number of the experience samples reaching a third preset threshold, determine the priority of the experience samples based on a priority mechanism; The determining module is further configured to determine an experience sample group based on the priority of the experience sample; The training module is used to train the deep reinforcement learning model based on the experience sample group; The training module is further configured to terminate training in response to the deep reinforcement learning model meeting a second preset condition; the second preset condition comprising at least one of the following: the number of training times reaching a fourth preset threshold and the expected value of the action decision output by the deep reinforcement learning model being greater than or equal to a fifth preset threshold; Wherein, the priority mechanism is a priority mechanism based on timing differential error and Q value; The priority mechanism based on the timing difference error and the Q value satisfies the following expression: Wherein, P(i) is the priority of the i-th experience sample based on the priority mechanism, μ TD is the average value of the time series difference error of all empirical samples, σ TD is the standard deviation of the time series difference error of all empirical samples, P TD (i) is the priority of the i-th empirical sample calculated based on the time series difference error, P Q (i) is the priority of the i-th experience sample calculated based on the Q value, and ε is a small positive value to ensure that P(i)>0; The priority of the i-th experience sample calculated based on the time series difference error satisfies the following expression: The priority of the i-th experience sample calculated based on the Q value satisfies the following expression: Among them, the softsing function and ISRU function are used to normalize the priority, δ i is the time series difference error of the i-th empirical sample, s i is the state parameter in the i-th experience sample, a i The target action decision is made for the state parameter.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 3 when executing the program. 6 . A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to claim 1 .

Images