Post-disaster wireless communication network resource allocation method

Abstract

The invention belongs to the technical field of wireless communication, and particularly relates to a post-disaster wireless communication network resource allocation method. The method comprises the following steps: arranging a corresponding strategy network for each agent, training each strategy network by adopting a multi-agent reinforcement learning model comprising each strategy network to obtain an optimal network parameter of each strategy network, and deploying each strategy network configured with the optimal network parameter to the corresponding agent. Therefore, resource allocation in the network is guided; according to the method, algorithm convergence is accelerated, single-step decision delay is reduced, the network throughput is finally improved, and real-time and reliable service quality guarantee is provided for users.

Description

Technical Field

[0001] This invention belongs to the field of wireless communication technology, specifically relating to a method for allocating wireless communication network resources after a disaster. Background Technology

[0002] Ensuring the Quality of Service (QoS) requirements of users in wireless communication networks is an unavoidable fundamental requirement in the development of any advanced 6G communication technology; at the same time, resources within wireless communication networks are usually limited. Therefore, how to rationally allocate limited resources within the network (such as base station transmit power, user access bandwidth, macro base station backhaul bandwidth, etc.) to ensure the QoS requirements of all access users within the network while maximizing service quality and providing a good user communication experience is one of the common resource allocation problems in wireless communication networks. However, due to the inherent characteristics of wireless channels, electromagnetic signals often experience large-scale and small-scale fading from the transmitter to the receiver, especially when the signal transmission distance is long, severely degrading the quality of the received signal. Furthermore, the random distribution of obstacles can lead to the obstruction of direct links between the base station and access users, making it difficult to guarantee good access link quality. Due to the openness of the wireless communication environment, legitimate signals are easily attacked by interferers, hindering normal communication processes. Channel conditions are easily affected by environmental fluctuations, user mobility, and climate change; therefore, the Channel Condition Information (CSI) of wireless communication networks is usually dynamically changing, and CSI fluctuations exist even in indoor spaces.

[0003] In summary, the aforementioned phenomena pose challenges to the resource allocation problem in wireless communication networks. Many agent-based deep reinforcement learning (MADRL) algorithms (such as MADDPG, MATD3, and MAPPO) can learn dynamic information from the environment through the interaction between the agent and the environment; they can address the highly coupled optimization variable problem with high-dimensionality, high-class, discrete, or continuous variables by deploying a policy network for each agent; and they can solve the performance evaluation problem of the current resource allocation results by deploying a global criticism network. These solutions overcome the limitations of traditional mathematical optimization algorithms in solving mixed-integer nonlinear resource optimization problems, such as highly coupled optimization variables and the inability to model stochastic environmental information with precise statistical models.

[0004] The MAPPO algorithm, employing Centralized Training Distributed Execution (CTDE), promotes collaboration among agents by deploying a global critique network for all agents during the training phase and sharing local observations through a shared channel. Simultaneously, during the deployment phase, agents only need to consider the observations and parameters of their policy network to take actions, thus improving decision-making efficiency. Furthermore, its unique policy update mechanism using a pruning technique further enhances the stability of the training process by preventing excessively large policy update steps. However, because simply deploying a global critique network and sharing local observations during the training phase makes it difficult to explicitly model the interdependencies between agents, it can easily lead to insufficient or inefficient collaboration. This reduces the algorithm's convergence speed, increases the agent's single-step decision time, and, when mapped to resource allocation problems in wireless communication networks, may struggle to provide real-time user QoS guarantees, resulting in user information loss and a poor user communication experience.

[0005] In summary, while the MAPPO algorithm can alleviate the non-stationarity of multi-agent training and improve the stability of policy updates, its slow convergence speed and long single-step decision time stem from its reliance on a single global critique network to promote cooperation among agents. This lack of explicit modeling of agent dependencies leads to issues such as slow convergence and long single-step decision times, impacting real-time resource allocation and QoS assurance in wireless communication networks. Therefore, explicitly modeling agent dependencies to promote sufficiency in collaboration, thereby improving convergence speed and reducing decision time, is crucial for guiding real-time QoS assurance and rational resource allocation within wireless communication networks. Summary of the Invention

[0006] To overcome the aforementioned problems, this invention provides a method for allocating wireless communication network resources after a disaster. This method is particularly suitable for long-distance communication environments where random obstacles and inaccurate channel state information exist near ground access users. It is also applicable to short-distance or indoor communication scenarios. Existing methods based on Multi-Agent Proximity Policy Optimization (MAPPO) fail to explicitly consider the global historical dependencies in the policy update process of each agent when optimizing network resource allocation, resulting in slow convergence and insufficient real-time decision-making. Therefore, this invention introduces a shared cooperative perception module for each agent within the MAPPO framework. This module explicitly fuses historical global policy state information through recurrent neural networks, multilayer perceptrons, and attention mechanisms, guiding agents to coordinate exploration and utilization behaviors during policy updates. This accelerates algorithm convergence, reduces single-step decision latency, ultimately improves network throughput, and provides users with real-time and reliable quality of service assurance.

[0007] A method for allocating resources in a post-disaster wireless communication network includes the following: S1, deploy the corresponding policy network for each agent; S2, a multi-agent reinforcement learning model including each policy network is used to train each policy network to obtain the optimal network parameters of each policy network, and each policy network with the optimal network parameters is deployed to the corresponding agent. S3, the global state information composed of the local observation information of all agents is input into each policy network. Each policy network generates the corresponding local action according to its current optimal network parameters and sends it to the corresponding agent. Each agent executes the corresponding action, thereby guiding the allocation of network resources through the coordinated actions of multiple agents. The multi-agent reinforcement learning model includes a shared cooperative perception network, a policy network deployed separately for each agent, and a global criticism network; wherein: The recurrent neural network comprises two hidden layers and one output layer. Each hidden layer contains 128 hidden units, and the output layer uses the tanh activation function. The network parameters are as follows: ; The multilayer perceptron includes two hidden layers and one output layer. Each hidden layer contains 128 hidden units, and the output layer uses the Rectified Linear Init (ReLU) activation function. The network parameters are as follows: ; Each policy network has the same structure, consisting of three hidden layers and one output layer. Each hidden layer contains 128 hidden units, and the output layer uses the ReLU activation function. The network parameters of the policy network for each agent are: ; The global critique network comprises three hidden layers and one output layer. Each hidden layer contains 128 hidden units, and the output layer has no activation function. The network parameters are as follows: .

[0008] The shared collaborative perception network also employs an attention mechanism. Each policy network inputs the obtained historical policy entropy sequence into a recurrent neural network. The recurrent neural network outputs a temporal feature vector, which is then input into a multilayer perceptron. The multilayer perceptron outputs a historical policy embedding vector. The historical policy embedding vector is processed by the attention mechanism to obtain the collaborative context vector of each agent. The collaborative context vector is concatenated with the global state information and then input into the policy network of the corresponding agent. The local actions output by each policy network constitute the joint action. The global state information and the joint action are used as input to the global critique network, which is used to evaluate the quality of the joint action.

[0009] The specific details of training each policy network using a multi-agent reinforcement learning model are as follows: Step 1: Intelligent Agent The policy network extracts the historical local action results of the corresponding agent and calculates its corresponding historical policy entropy sequence; where: exist In the next iteration, the agent policy entropy for: ; in It refers to the first In the next iteration, for the intelligent agent The deployed policy network is for intelligent agents The strategy adopted is as follows: the local observation information of all agents constitutes the global state information. intelligent agent Policy network acquisition And based on the current network parameters The output is used to guide the agent. Local Actions ; in , For the set of all intelligent agents, For the first In the next iteration, the agent Local observation information; Then in the In the next iteration, the agent Historical strategy entropy sequence for: ; in For the first In the next iteration, the agent The strategy entropy, to , The preset historical observation length; Step 2: Obtain the temporal feature vector; For intelligent agents The deployed policy network will enable agents The historical strategy entropy sequence is input into a recurrent neural network, from which the recurrent neural network extracts the agent. Temporal feature vectors Specifically: ; in These are the network parameters of a recurrent neural network; Step 3: Obtain the historical policy embedding vector of the agent; The temporal feature vector output by the recurrent neural network The input is a multilayer perceptron, which processes the temporal feature vector. Mapping to a high-dimensional feature space yields the intelligent agent. Historical strategy embedding vector : ; in These are the network parameters of the multilayer perceptron; Step 4: Construct normalized attention weights for each agent among the historical policy embedding vectors of all other agents; details are as follows: Step 4.1: Connect the intelligent agent Historical strategy embedding vector As a query vector, the historical policies of all agents are embedded into the vector. As the key vector, dot product attention is used to measure its value as the query vector. Define the agent based on its correlation with any vector in the key vector. For intelligent agents Historical strategy embedding vector The scoring function is : ; in, and These are the query weight matrix and the key weight matrix, respectively. It is the dimension of the key vector, and T is the transpose; Step 4.2: Normalize the score function using the Softmax function to obtain the agent. For intelligent agents Normalized attention weights of historical strategy embedding vectors : ; in For intelligent agents The scoring function for the historical policy embedding vector of agent k; Step 4.3: Embed the historical policy of agent i into the vector As a value vector, and with normalized attention weights Perform a weighted summation to obtain the intelligent agent. Collaboration context vector : ; in, It is a value weight matrix; Step 5: Concatenate the cooperative context vector with the global state information and input it into the policy network of the corresponding agent; the specific content is as follows: global state information With intelligent agents Collaboration context vector By splicing them together, an intelligent agent is formed. Enhanced feature vectors intelligent agents of Input the corresponding policy network, and the policy network will... Output it as an intelligent agent Selected local action and transmit it to the corresponding intelligent agent. ; Step 6: The local actions output by each policy network constitute the joint action. ; global state information With joint actions The inputs to the global critical network are shared, and the global critical network is used to evaluate joint actions. The advantages and disadvantages; Step 7: Update the network parameters of the policy network Network parameters of recurrent neural networks Network parameters of multilayer perceptron and the network parameters of the global critique network ; Step 7.1: Determine the network parameters used to update the policy network. The objective function is as follows: Each agent executes the local action output by the corresponding policy network. Then, through interaction with the environment, the immediate reward for this iteration is obtained. Global state information for the next iteration And calculate long-term discount rewards. ,in It is a discount factor. Indicates the number of iterations from the current iteration. Begin, look ahead. step; For the first The immediate reward at the next iteration; and the rewards generated in each iteration. These are called experience pairs and are stored in the experience replay pool. middle; Global critique networks employ common Q-functions and V-functions to evaluate the joint actions of all agents. An evaluation was conducted to obtain the state-action value function related to the long-term reward of the joint action. With state value function The formula is as follows: ; in , for the first In the next iteration, each policy network adopts a policy for its corresponding agent. A combined strategy formed by piecing together elements; This represents the expected value of long-term discount rewards; This indicates that the network parameters are At that time, all agents are in the global state information The following joint actions were taken. Then continue to follow the current joint strategy in the long term. The expected long-term discount reward value that can be obtained will This is called the Q value; This indicates that the network parameters are At that time, all agents are in the global state information The following follows the current joint strategy The expected long-term discount reward that can be obtained will This is called the V value; Global Criticism Network estimates the global dominance function using a truncated generalized dominance estimation method. Used to evaluate the global state information of all agents. The following joint actions were taken. The benefits brought by time, namely The degree of superiority or inferiority is defined as follows: ; Network parameters of the guiding strategy network Updated target function Defined as: ; in, Indicates the replay of experiences from the pool Global state information obtained from mid-sample Helian Combined action Find the expected value; , representing intelligent agents The ratio of sampling probabilities between the old and new strategies; This represents the agent at the (t-1)th iteration. The corresponding policy network is the agent. The strategy adopted; Indicates the shearing parameter; Step 7.2, Intelligent Agent Network parameters of the strategy network Network parameters of recurrent neural networks Network parameters of multilayer perceptron and the network parameters of the global critique network The update process is as follows: (1) Network parameters of the policy network Update process: Maximize the agent Objective function of policy update Thus, the first In the next iteration, the agent The optimal network parameters for the strategy network : ; intelligent agent The network parameter update process for the policy network is as follows: ; in For intelligent agents The learning rate of the policy network; To solve using the Adam optimizer Compared to The partial derivative; (4) Network parameters of recurrent neural networks Network parameters of multilayer perceptron The update process is as follows: ; in, It is the learning rate of recurrent neural networks and multilayer perceptrons; Describe the objective function For recurrent neural network parameters gradient, Describe the objective function Multilayer perceptron parameters The gradient; (5) Network parameters of the global critique network Update process: For a global critique network, parameters are updated by minimizing the mean squared error loss function, where the mean squared error loss function... Defined as: ; in This indicates that the expression within the parentheses refers to the experience replay pool. The empirical data obtained from sampling Find the average; This indicates that the mean square error of the expression within the parentheses is calculated. Indicates the current network parameters The Q value below; For all agents in global state information The following joint actions were taken. Then continue to follow the current joint strategy in the long term. The expected long-term discount reward value that can be obtained For global state information Based on the current joint strategy The joint action obtained at the (t+1)th iteration; In the Optimal global criticism network parameters at the next iteration The definition is as follows: ; The network parameter update process of the global critique network is as follows: ; in The learning rate of the global critique network; Step 7.3: Repeat step 7.2 for multiple iterations until the agent receives an immediate reward through interaction with the environment. Value convergence, intelligent agent The strategy network finds the optimal network parameters. .

[0010] The beneficial effects of this invention are: This invention, while maintaining the stability of agent policy updates, explicitly models the policy distribution state relationship between agents through dual encoders (Recurrent Neural Network (RNN) and Multilayer Perceptron (MLP), promoting agent cooperation at the global level, thereby accelerating algorithm convergence, reducing single-step decision latency, and providing an effective solution to the resource allocation problem that requires real-time response in communication networks.

[0011] This invention requires no additional hardware support, resulting in lower implementation costs. Compared to methods that introduce complex relationship modeling such as hypergraph neural networks, this method has a simpler structure and is easier to implement. Furthermore, due to its adaptability to environmental dynamics, this algorithm also possesses stronger robustness and scalability. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of a wireless communication scenario according to the present invention.

[0013] Figure 2 This is a system functional block diagram of the present invention.

[0014] Figure 3 This is a diagram of the strategy network structure of the present invention.

[0015] Figure 4 The figure shows the experimental results of Embodiment 2 of the present invention. Detailed Implementation

[0016] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it.

[0017] Example 1 First, the problem of resource integration and allocation within the network is constructed as an optimization problem constrained by meeting the basic Quality of Service (QoS) requirements of access users and aimed at maximizing network throughput. In this problem, the resource allocation decisions of each communication entity are considered its actions, and each communication entity responsible for resource allocation acts as an agent, continuously iteratively updating its resource allocation strategy through a policy network until the algorithm converges or training is complete.

[0018] The core improvement of this invention lies in the following: based on optimizing network resources using the MAPPO algorithm, a shared cooperative perception module is added to the entire policy network when training the policy network of each agent. This module is composed of a recurrent neural network, a multilayer perceptron, and an attention mechanism. Its function is to explicitly calculate the agent's perception result of the historical global policy state and introduce this result as an additional input into the agent's policy network, thereby affecting the update of policy network parameters from the perspective of global cooperation, so as to accelerate the convergence of the algorithm.

[0019] Furthermore, the "historical global policy state" mainly refers to the policy entropy of each agent, which reflects whether the agent is more inclined to explore new policies or utilize existing policies during the policy update process. This invention, through a shared collaborative perception module, aggregates the policy state information of all agents, thereby guiding each agent to rationally adjust its exploration and utilization tendencies from a global perspective. This guidance information is not inherent in the original MAPPO algorithm but is derived through the design of this method. Therefore, this method can explicitly model the interdependencies of agents during the policy update process, thereby guiding the parameter updates of their policy network and effectively promoting cooperation among agents. Specifically: A method for allocating resources in a post-disaster wireless communication network includes the following: First, we establish a long-distance wireless communication scenario where access users are uniformly distributed in a two-dimensional plane, obstacles are randomly distributed near the access users, and the base station is located at the center of the two-dimensional plane. Figure 1 As shown.

[0020] Due to significant path loss and obstacle penetration loss in user access links, a RIS-UAV system, consisting of a UAV equipped with a smart reflective panel, is deployed at a certain height at the center of a two-dimensional plane. This system forms a cascaded link between the base station, RIS-UAV, and the access user, using the smart reflective panel (RIS) mounted on a UAV, to assist information transmission. A time-division (TS) protocol is deployed on the RIS components, dividing the unit of time into an information transmission (IT) phase and an energy harvesting (EH) phase: the IT phase is used to complete signal transmission, while the EH phase utilizes the energy from the received signal to provide power for the UAV's hovering, maintaining its continuous operation.

[0021] Based on the aforementioned system, an optimization problem is constructed with the goal of maximizing network throughput and constraints set on the minimum QoS requirements of access users and the minimum energy collection of the RIS (Resource Identifier). This problem involves the joint optimization of base station transmit power resources, RIS element phase shifts, and information transmission phase durations. To this end, the base station and RIS are treated as agents, and the proposed method is used to train policy networks deployed separately for each agent. By explicitly modeling the mutual influence of policy distributions among agents, cooperation between agents is promoted, thereby improving the algorithm's convergence speed. Furthermore, while ensuring user service quality requirements, the algorithm fully optimizes network resource allocation, providing users with a superior communication experience.

[0022] A multi-agent reinforcement learning model based on the MAPPO algorithm is deployed with a centralized training and distributed execution framework. The multi-agent reinforcement learning model includes a shared cooperative perception network, a policy network deployed for each agent, and a global critique network. The number of policy networks is the same as the number of agents, that is, a corresponding policy network is deployed for each agent. The model sets a shared cooperative perception network and a global critique network before and after all policy networks to train the policy networks.

[0023] The shared cooperative perception network explicitly calculates the agent's perception result of historical global policy state guidance, and concatenates this result with global observation information before inputting it into the corresponding agent's policy network; the shared cooperative perception network includes a recurrent neural network (RNN), a multilayer perceptron (MLP), and an attention mechanism, wherein: The recurrent neural network comprises two hidden layers and one output layer. Each hidden layer contains 128 hidden units, and the output layer uses the tanh activation function. The network parameters of the recurrent neural network are as follows: ; The multilayer perceptron includes two hidden layers and one output layer. Each hidden layer contains 128 hidden units, and the output layer uses the Rectified Linear Init, ReLU activation function. The network parameters of the multilayer perceptron are as follows: ; An attention mechanism is applied to the historical policy embedding vectors of each agent, enabling each agent to perceive the global historical policy state and obtain the perception result, i.e., the agent's state. Collaboration context vector .

[0024] Each policy network has the same structure, consisting of three hidden layers and one output layer. Each hidden layer contains 128 hidden units, and the output layer uses the ReLU activation function. The network parameters of the i-th policy network are... ; The global critique network comprises three hidden layers and one output layer. Each hidden layer contains 128 hidden units, the output layer has no activation function, and the network parameters of the global critique network are as follows: .

[0025] A multi-agent reinforcement learning model is employed to train a policy network deployed separately for each agent. The training process is as follows: Figure 2-3 As shown: Step 1: For the intelligent agent The deployed policy network extracts the historical local action results of its corresponding agent and calculates its corresponding historical policy entropy sequence; where: Calculate using the following formula In the next iteration, the agent policy entropy (Used to represent the distribution of current local actions): ; in It refers to the first In the next iteration, for the intelligent agent The deployed policy network is for intelligent agents The strategy adopted is as follows: This involves using the global state information observed by all agents during this iteration. Input to intelligent agent When deploying a policy network, the policy network is based on the current network parameters. Output guidance agent Local actions to perform actions ; in , For the set of all intelligent agents, For the first In the next iteration, the agent The local observation information includes the location information of all its access users and the maximum transmit power constraint of the base station agent, while for the intelligent reflector RIS agent, the local observation information includes the location information of all its access users and the minimum energy harvesting constraint of the RIS. The joint observation information is obtained by combining the local observation information of all agents. That is, global state information ,Right now ; Then in the In the next iteration, the agent Historical strategy entropy sequence It is determined by the policy entropy generated from the nearest number of historical iterations. ; in For the first The policy entropy generated in the next iteration. to , The preset historical observation length; Step 2: Obtain the temporal feature vector; In the MAPPO algorithm using Centralized Training Distributed Execution (CTDE), global state information is defined as the joint local observation information of all agents; that is, the global state is equivalent to the joint observation. The deployed policy network inputs the agent's historical policy entropy sequence into the recurrent neural network, from which the recurrent neural network extracts the agent's... Temporal feature vectors Specifically, the temporal feature code of the intelligent agent: ; in These are the network parameters of a recurrent neural network; Step 3: Obtain the historical policy embedding vector of the agent; In the In the next iteration, the temporal feature vector output by the recurrent neural network will be... The input is a multilayer perceptron, which processes the temporal feature vector. Mapping to a high-dimensional feature space yields the intelligent agent. Historical strategy embedding vector : ; in These are the network parameters of the multilayer perceptron; Step 4: Construct normalized attention weights for each agent among the historical policy embedding vectors of all other agents; details are as follows: Step 4.1: Connect the intelligent agent Historical strategy embedding vector As a query vector, the historical strategies of all agents are embedded into the vector. As key vectors, dot product attention is used to measure their performance as query vectors. The correlation between the agent and any vector in the key vector; defining the agent. For intelligent agents Historical strategy embedding vector The scoring function is : ; in, and These are the learnable query weight matrix and the key weight matrix, respectively. It is the dimension of the key vector, and T is the transpose; Step 4.2: Normalize the score function using the Softmax function to obtain the agent. For intelligent agents Normalized attention weights of historical strategy embedding vectors : ; in For intelligent agents The scoring function for the historical policy embedding vector of agent k; This weight measures the agent's performance in the current state. For intelligent agents The importance of understanding historical strategy status information.

[0026] Step 4.3: Embed the historical policy of agent i into the vector As a value vector, and together with the normalized attention weights Perform a weighted summation to obtain the intelligent agent. Collaboration context vector : ; in, It is a learnable value weight matrix; this vector explicitly defines the agent. The degree of attention paid to the historical policy state information of all agents, i.e., the agent Based on a unique understanding of the historical global policy state from the perspective of local policy state, this vector influences the agent from a global perspective. The strategy update should lean towards either continuing to explore new strategies or utilizing existing strategies.

[0027] Step 5: Concatenate the cooperative context vector and joint observation information, and input the result into the policy network of the corresponding agent; the specific steps are as follows: The global state information observed by all agents, i.e., the joint observation information. With intelligent agents Collaboration context vector By splicing them together, an intelligent agent is formed. Enhanced feature vectors intelligent agents of Input its corresponding policy network (i.e., ...) As a new input to the policy network, the corresponding policy network is based on... Output it as an intelligent agent Selected local action and transmit it to the corresponding intelligent agent. ; Step 6: For the MAAPPO algorithm, the local actions output by each policy network constitute the joint action. ;Right now An agent observes the enhanced feature vector. And according to the current policy network Selected local action This constitutes a joint action; Global state information observed by all agents With joint actions The joint actions serve as input to the global critique network, and the output of the global critique network is used to evaluate the joint actions. The value of combining advantages and disadvantages in a strategy ; Step 7: Update the network parameters of the policy network Network parameters of recurrent neural networks Network parameters of multilayer perceptron and the network parameters of the global critique network ; Step 7.1, Intelligent Agent Network parameters of the corresponding policy network The updated objective function has been determined, as detailed below: Each agent performs its local action according to the output of its corresponding policy network. After performing an action, the system interacts with the environment to obtain global state information for the next iteration. Instant rewards for this iteration and long-term discount rewards ,in It is a discount factor. Indicates the number of iterations from the current iteration. Begin, look ahead. step; For the first The immediate reward at the next iteration; this formula is used to calculate the expected reward if the current resource allocation result is continued. Simultaneously, the above interaction process generates... These are called experience pairs and are stored in the experience replay pool. middle; The cooperative context vector for the next iteration is obtained by updating the historical policy entropy sequence stored by each agent using a sliding window update method. Global status information and the enhanced feature vector ; The combined actions of the agent are controlled using common Q-functions and V-functions. An evaluation was conducted to obtain the state-action value function related to the long-term reward of the joint action. With state value function The calculation formula is as follows: ; in, For the first In the next iteration, each policy network adopts a policy for its corresponding agent. A combined strategy formed by piecing together elements; It represents the expected value of the long-term discounted reward obtained by all agents interacting with the environment using the current joint policy; Represents the global state information of all agents. The following joint actions were taken. Then continue to follow the current joint strategy in the long term. (That is, the expected long-term discount reward value that can be obtained next time) This is called the Q value; Represents the global state information of all agents. The following follows the current joint strategy The expected long-term discount reward that can be obtained will This is called the V value; Estimating the global dominance function using the truncated generalized dominance estimation (GAE) method Used to evaluate the global state information of all agents. The following joint actions were taken. The benefits brought by this are equivalent to the global state information. The following follows the current joint strategy The gains obtained are the average performance. The degree of superiority or inferiority is defined as follows: ; The MAPPO algorithm uses a clipping mechanism to constrain the magnitude of agent policy network updates, thus ensuring the stability of the training process. Network parameters of the strategy network Updated target function The definition is as follows: ; in, Indicates the replay of experiences from the pool Global state information obtained from mid-sample and joint actions Find the expected value; , representing intelligent agents The ratio of sampling probabilities between the old and new strategies; This represents the agent at the (t-1)th iteration. The corresponding policy network is the agent. The strategy adopted; This represents the shearing parameter, which restricts the sampling probability ratio to a range. This prevents excessively large policy update cycles.

[0028] Step 7.1, Intelligent Agent Network parameters of the strategy network Network parameters of recurrent neural networks Network parameters of multilayer perceptron and the network parameters of the global critique network The update process is as follows: (1) Intelligent agent Network parameters of the strategy network Update process: Maximize the agent Objective function of policy update Thus, the first In the next iteration, the agent The optimal network parameters for the strategy network : ; Therefore, the Adam optimizer is used to solve the problem. Compared to partial derivatives ,Right now For intelligent agents The gradients of the network parameters of the policy network are calculated; and the agent is updated using the stochastic gradient ascent method. The network parameters of the policy network are as follows: ; in For intelligent agents The learning rate of the policy network; (2) Network parameters of recurrent neural networks Network parameters of multilayer perceptron Update process: Network parameters of recurrent neural networks Network parameters of multilayer perceptron The updates all rely on gradient information backpropagated from the policy network; due to the shared nature of recurrent neural networks and multilayer perceptrons, both receive gradient information from all agents, and their update direction is the aggregated mean of all gradient information: ; in, It is the learning rate of recurrent neural networks and multilayer perceptrons; Represents intelligent agents Objective function of policy update For recurrent neural network parameters gradient, Represents intelligent agents Objective function of policy update Multilayer perceptron parameters The gradient (i.e.) To solve using the Adam optimizer Compared to The partial derivative, To solve using the Adam optimizer Compared to (partial derivatives) (3) Network parameters of the global critique network Update process: For a global critique network, parameters are updated by minimizing the mean squared error loss function, where the mean squared error loss function... Defined as: ; in This indicates that the expression within the parentheses refers to the experience replay pool. The empirical data obtained from sampling Find the average; This indicates that the mean square error of the expression within the parentheses is calculated. Indicates the current network parameters The Q value below; Indicates the old network parameters before the update. The Q-value represents the global state information of all agents. The following joint actions were taken. Then continue to follow the current joint strategy in the long term. The expected long-term discount reward value that can be obtained For global state information Based on the current joint strategy The joint action obtained at the (t+1)th iteration; In the Optimal global criticism network parameters at the next iteration The definition is as follows: ; Therefore, the network parameter update process of the global critique network using the stochastic gradient descent method is as follows: ; in The learning rate of the global critique network; The agent observes global state information and collaborative context vectors Local actions are selected based on the policy network to form joint actions. Then, instant reward values ​​are obtained through interaction with the environment. Global state information for the next iteration , forming experience pairs and put it into the experience replay pool. In the process, a rolling window update mechanism is used to update the historical policy entropy value storage sequence for use in the next iteration. Then, update the network parameters according to the above-mentioned network parameter update rules; Step 7.3, repeat step 7.2, until the agent receives an immediate reward through interaction with the environment after each iteration. The value gradually converges and reaches the predetermined number of iterations, at which point the agent... The strategy network finds approximately optimal network parameters. The optimal parameters for other networks were also obtained: the network parameters of the recurrent neural network. Network parameters of multilayer perceptron and the network parameters of the global critique network .

[0029] During each iteration, the agent adjusts its actions based on the experience samples obtained from the experience pool. How should the parameters of the policy network and the critique network be updated? After the parameters are updated, they can guide the agent in allocating resources within the network. Furthermore, the agent can obtain information from the global critique network. Evaluation of the current resource allocation results: This indicates that the current resource allocation strategy is better than the historical average and should be strengthened; conversely, the current resource allocation strategy should be suppressed if it is not. Through continuous iteration, updating, and evaluation, when the reward value gradually converges, it can be said that the agent has found the near-optimal policy distribution parameters.

[0030] Set the network parameters of the policy network for each agent to the network parameters of the corresponding optimal policy network obtained above, and then deploy the configured policy networks to the corresponding agents. This will allow the global state information, composed of the local observation information of all agents, to be considered. When input is given to each policy network, each policy network uses its current optimal network parameters. Output the corresponding local action To the corresponding intelligent agent; each intelligent agent executes the corresponding... This allows for the allocation of network resources to be guided through the coordinated actions of multiple intelligent agents.

[0031] Example 2 against Figure 1 The constructed post-disaster communication network model has the following environmental parameters: it includes three base stations connected to a macro base station via wireless backhaul links, with a maximum wireless backhaul capacity of 300Mbps. Each base station is associated with 10 randomly distributed access users within its coverage area. The base stations are evenly distributed over a distance of 1000 m. A 1000 m ground area is covered with obstacles that can be considered as cubes, and their distribution is modeled as a Poisson point process with a density of 0.15 obstacles / m. The length and width are uniformly distributed in the range [1,2] m, and the height is uniformly distributed in the range [1.25,2] m. Near each base station, there is a RIS (Radio Receptor System) with 36 elements carried by a UAV, with a height of 200 m. Its horizontal position is determined by a density-aware strategy that senses the horizontal position of the ground access user. The minimum data rate requirement for each access user is 2.5 Mbps, the maximum available transmit power of the base station is 300 W, the minimum energy consumption of the UAV in hovering state is 80 W, and the power of additive white Gaussian noise is -114 dBm.

[0032] In this wireless communication network model, the three base stations are labeled as base station-1, base station-2, and base station-3, and their corresponding RIS are also labeled as RIS-1, RIS-2, and RIS-3, respectively. The location information of each corresponding base station-RIS and its 10 associated users is as follows. This information is concatenated to form the global state information. : (1) Base station-1 is located at coordinates (250, 250, 0) m, and the coordinates of its 10 associated ground users are (245.3, 260.7, 0), (280.1, 240.5, 0), (230.8, 220.9, 0), (260.4, 290.2, 0), (210.5, 250.6, 0), (290.7, 270.3, 0), (240.2, 210.8, 0), (270.9, 230.4, 0), (220.3, 280.1, 0), (300.0, 250.0, 0) m; the maximum transmit power of this base station is 300 W and the access bandwidth is 2 MHz. The corresponding RIS-1 is located at (250, 250, 200) m, and its observed user coordinates are exactly the same as those of base station 1, and it has a minimum energy harvesting requirement of 40 W.

[0033] (2) Base station-2 is located at coordinates (500, 500, 0) m, and its associated user coordinates are (510.2, 490.5, 0), (480.3, 520.7, 0), (530.8, 470.9, 0), (460.4, 540.2, 0), (550.5, 480.6, 0), (470.7, 530.3, 0), (520.2, 460.8, 0), (490.9, 510.4, 0), (540.3, 450.1, 0), (450.0, 550.0, 0) m; its maximum transmit power is 300 W and its access bandwidth is 2 MHz. The corresponding RIS-2 is located at (500, 500, 200) m, with the same user coordinates as base station 2, and a minimum energy harvesting requirement of 40 W.

[0034] (3) Base station-3 is located at coordinates (750, 750, 0) m, and its associated user coordinates are (745.3,760.7, 0), (780.1, 740.5, 0), (730.8, 720.9, 0), (760.4, 790.2, 0), (710.5,750.6, 0), (790.7, 770.3, 0), (740.2, 710.8, 0), (770.9, 730.4, 0), (720.3,780.1, 0), (800.0, 750.0, 0) m; the maximum transmit power is 300 W and the access bandwidth is 2 MHz. The corresponding RIS-3 is located at (750, 750, 200) m, with the same user coordinates as base station 3, and a minimum energy harvesting requirement of 40 W.

[0035] Each base station and each RIS can be considered as an agent: the base station agent controls the phase and amplitude of the signal it transmits to each associated user, while the RIS agent controls the phase shift of each element on its surface and the time used for information transmission. By controlling these resources, the network throughput (defined as the data transmission rate for all users within the network) is maximized; therefore, the reward function at each iteration... Defined as network throughput.

[0036] Regarding the aforementioned communication network model and optimization problem, the training method described in Example 1 is used to train the policy network of each agent, with the following parameter settings: learning rate. Shear parameters Discount Factor Batch sampling size is 128, experience pool Size is 1,000,000, number of iterations is 6 10 With the parameters set above, the policy network is trained using the training method of Example 1 to achieve resource optimization within the network.

[0037] The policy networks of each agent were trained using the CMAPPO algorithm proposed in this invention and the traditional MAPPO algorithm. The experimental results are as follows: Figure 4 As shown, the horizontal axis represents the number of training iterations, and the vertical axis represents the instantaneous reward during each iteration. The convergence point is defined as follows: the moving average instantaneous reward calculated over the past 100 iterations over 500 consecutive training iterations. An increase of less than 0.5% is considered an immediate reward. It has converged; and the shaded area represents the 95% confidence interval around the immediate reward value calculated based on a sliding window (size 50 iterations) during training.

[0038] It is evident that the traditional CMAPPO algorithm, compared to the MAPPO algorithm, achieves a certain degree of improvement in convergence speed due to its explicit modeling of the agent's perception process guided by historical global states, which enhances the cooperation process among agents. Furthermore, the improved understanding of the interdependencies between agents' policy distribution states also leads to a certain degree of improvement in reward convergence. The results show that after a certain number of training iterations, the reward value reaches a stable convergence state, at which point the joint action output by all agents... This can be considered as achieving a near-optimal result. Therefore, by obtaining the joint actions... Mapped to resource allocation within the network, a near-optimal resource allocation strategy was ultimately obtained. Table 1 first presents the resource allocation results generated for each base station using this method, including refined beamforming amplitude (dimensionless), phase (unit: radians), and transmit power (unit: watts, W) for each associated user: Table 1 Base Station Resource Allocation Results User order Base Station-1 Base Station-2 Base Station-3 1 5.55,-1.120,30.80 4.75,-1.243,22.56 5.83,-1.354,34.00 2 3.24,1.833,10.50 5.89,1.992,34.69 4.96,1.932,24.60 3 6.37,0.253,40.59 3.23,0.261,10.44 5.14,0.245,26.42 4 5.45,-2.179,29.70 5.16,-2.678,26.63 4.63,-2.074,21.44 5 5.16,-0.458,26.63 6.33,-0.232,40.07 4.88,-0.394,23.81 6 5.46,1.422,29.81 6.38,1.517,40.70 5.91,1.476,34.93 7 4.15,2.792,17.22 5.52,2.950,30.47 5.22,2.834,27.25 8 3.81,0.927,14.52 3.78,0.722,14.29 3.83,0.837,14.67 9 5.40,-2.985,29.16 4.92,-2.460,24.21 4.93,-2.651,24.30 10 6.18,-1.515,38.20 5.40,-1.421,29.16 6.39,-1.404,40.83 total — , — , 267.13 W — , — , 273.22 W — , — , 272.25 W

[0039] The table shows the amplitude, phase (rad), and allocated power (W) in sequence. The numerical results provide precise guidance for the amplitude and phase of the base station's transmitted signal for any associated user. For example, base station-1 allocates resources of 6.37, 0.253, and 40.59 for user 3, indicating that the user's beamforming amplitude is 6.37, the phase is 0.253 rad, and the power of base station-1's transmitted signal for user 3 is 40.59 W. Meanwhile, base station-1 allocates resources of 3.81, 0.927, and 14.52 for user 8, with significantly lower power allocation values, demonstrating the method's ability to prioritize resources based on user channel conditions and the overall network status. Overall, the total transmit power of each base station is 267.13 W, 273.22 W, and 272.25 W, respectively, all strictly meeting the maximum power constraint of no more than 300 W, verifying the feasibility and reliability of this method under complex constraints.

[0040] Secondly, the resource allocation results generated by the method of this invention for all elements of each RIS are shown, including phase shifts (unit: radians) and time proportions (dimensionless) for the information transmission phase, as follows: For RIS-1, the phase shift configuration values ​​for its 36 reflective elements are as follows: 0.10, -0.50, 0.75, -0.20, 0.60, -0.40, 0.30, -0.70, 0.15, -0.55, 0.80, -0.35, -0.23, 0.67, -0.45, 0.33, -0.78, 0.12, -0.56, 0.89, -0.34, 0.25, -0.72, 0.50, 0.45, -0.12, 0.78, -0.33, 0.22, -0.67, 0.41, -0.25, 0.63, -0.48, 0.37, -0.60. These phase shift values ​​are distributed in the interval [-π, π]. By adjusting the phase of each reflecting element, the wavefront of the reflected signal can be changed, thereby focusing the signal energy onto the target user. Meanwhile, the time proportions of the 36 components in the RIS-1 during the information transmission phase are as follows: 0.60, 0.55, 0.70, 0.45, 0.65, 0.40, 0.75, 0.30, 0.80, 0.35, 0.85, 0.50, 0.55, 0.60, 0.65, 0.70, 0.40, 0.75, 0.45, 0.80, 0.35, 0.90, 0.25, 0.95, 0.30, 0.85, 0.40, 0.75, 0.50, 0.65, 0.60, 0.55, 0.70, 0.45, 0.80, 0.35. These time proportions, ranging from 0.25 to 0.95, represent the percentage of time each component dedicates to information transmission per unit of time, with the remaining time used for energy harvesting. By rationally allocating the time proportion of each component, information transmission efficiency can be maximized while meeting the RIS's energy harvesting requirements. For RIS-2, the phase shift configuration values ​​of its 36 reflective elements are as follows: -0.12, 0.78, -0.23, 0.67, -0.34, 0.25, -0.56, 0.33, -0.78, 0.45, -0.67, 0.22, 0.45, -0.67, 0.33, -0.56, 0.89, -0.12, 0.78, -0.45, 0.22, -0.33, 0.41, -0.78, -0.33, 0.22, -0.45, 0.12, -0.78, 0.41, -0.25, 0.67, -0.12, 0.89, -0.56, 0.33.These phase shift values ​​also cover the entire interval [-π, π], but the specific distribution pattern is different from that of RIS-1. This reflects that the algorithm can generate customized phase configuration schemes for the specific user group location and channel conditions served by each RIS. The time allocation for each RIS-2 component during the information transmission phase is as follows: 0.70, 0.45, 0.80, 0.55, 0.60, 0.65, 0.75, 0.40, 0.85, 0.50, 0.90, 0.35, 0.95, 0.30, 0.65, 0.70, 0.45, 0.75, 0.40, 0.80, 0.55, 0.60, 0.65, 0.70, 0.75, 0.35, 0.80, 0.45, 0.85, 0.50, 0.55, 0.60, 0.65, 0.70, 0.75, 0.40. These ratios range from 0.30 to 0.95, with some components (such as the 24th component) achieving a time ratio as high as 0.95, indicating that these components play a more critical role in building a high-quality reflection link and are therefore allocated more communication time. For RIS-3, the phase shift configuration values ​​of its 36 reflective elements are as follows: 0.45, -0.33, 0.78, -0.45, 0.33, -0.78, 0.22, -0.67, 0.12, -0.89, 0.56, -0.41, -0.12, 0.89, -0.56, 0.41, -0.25, 0.67, -0.33, 0.12, -0.78, 0.45, -0.67, 0.22, 0.78, -0.45, 0.33, -0.78, 0.22, -0.12, 0.89, -0.56, 0.41, -0.25, 0.33, -0.78. This configuration further validates the algorithm's ability to generate diverse phase shift modes, with each RIS's phase matrix being an optimized response to its specific service scenario. The time allocation for each RIS-3 component during the information transmission phase is as follows: 0.65, 0.70, 0.45, 0.80, 0.55, 0.60, 0.75, 0.40, 0.85, 0.50, 0.90, 0.35, 0.95, 0.30, 0.65, 0.70, 0.45, 0.75, 0.40, 0.80, 0.55, 0.60, 0.65, 0.70, 0.75, 0.35, 0.80, 0.45, 0.85, 0.50, 0.55, 0.60, 0.65, 0.70, 0.75, 0.40.Similar to RIS-2, the time ratio is dynamically allocated between 0.30 and 0.95, ensuring an optimal balance between energy harvesting and information transmission.

[0041] Guided by the above resource allocation strategy, the actual data transmission rates of the 10 users associated with each base station were used as performance indicators to evaluate the resource allocation results, as shown in Table 2: Table 2 User speeds (Mbps) for each base station base station User rates (in order) Total base station speed (Mbps) Base Station 1 3.2, 4.1, 2.8, 5.0, 3.5, 4.3, 2.9, 3.8, 4.5, 3.0 37.1 Base Station 2 3.0, 4.5, 2.7, 4.8, 3.8, 5.2, 3.1, 4.0, 3.3, 4.1 38.5 Base Station 3 3.5, 4.2, 2.9, 4.7, 3.6, 5.1, 3.2, 4.3, 3.8, 4.9 40.2

[0042] The final results show that the total network throughput is 115.8 Mbps, which meets the wireless backhaul capacity constraint; the total transmit power of each base station meets the constraint of its maximum transmit power; and the rate of all users is higher than the minimum transmission rate requirement, which meets all resource constraints within the network.

[0043] A MAPPO algorithm-driven method for post-disaster communication network resource allocation uses historical policy entropy sequences instead of current policy entropy values, avoiding the circular dependency problem between policy network inputs and outputs. At the same time, it enhances the algorithm's ability to perceive temporal cooperation patterns by capturing the temporal dynamic evolution of cooperation relationships through recurrent neural networks.

[0044] In summary, this invention improves the convergence speed of the algorithm and reduces the single-step decision time of the agents by explicitly modeling the mutual influence of the policy distribution states among the agents from a global perspective, thus providing more real-time QoS guarantees for access users.

[0045] The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the scope of protection of the present invention is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present invention, any person skilled in the art can make equivalent substitutions or changes based on the technical solution and inventive concept of the present invention within the scope of the technology disclosed in the present invention. These simple modifications are all within the scope of protection of the present invention.

Claims

1. A method for allocating resources in a post-disaster wireless communication network, characterized in that, Includes the following: S1, deploy the corresponding policy network for each agent; S2, a multi-agent reinforcement learning model including each policy network is used to train each policy network to obtain the optimal network parameters of each policy network, and each policy network with the optimal network parameters is deployed to the corresponding agent. S3, the global state information composed of the local observation information of all agents is input into each policy network. Each policy network generates the corresponding local action according to its current optimal network parameters and sends it to the corresponding agent. Each agent executes the corresponding action, thereby guiding the allocation of network resources through the coordinated actions of multiple agents. The multi-agent reinforcement learning model includes a shared cooperative perception network, a policy network deployed separately for each agent, and a global criticism network; wherein: The recurrent neural network comprises two hidden layers and one output layer. Each hidden layer contains 128 hidden units, and the output layer uses the tanh activation function. The network parameters are as follows: ; The multilayer perceptron includes two hidden layers and one output layer. Each hidden layer contains 128 hidden units, and the output layer uses the ReLU activation function. The network parameters are as follows: ; Each policy network has the same structure, consisting of three hidden layers and one output layer. Each hidden layer contains 128 hidden units, and the output layer uses the ReLU activation function. The network parameters of the policy network for each agent are: ; The global critique network comprises three hidden layers and one output layer. Each hidden layer contains 128 hidden units, and the output layer has no activation function. The network parameters are as follows: .

2. The method for allocating resources in a post-disaster wireless communication network according to claim 1, characterized in that, The shared collaborative perception network also employs an attention mechanism. Each policy network inputs the obtained historical policy entropy sequence into a recurrent neural network. The recurrent neural network outputs a temporal feature vector, which is then input into a multilayer perceptron. The multilayer perceptron outputs a historical policy embedding vector. The historical policy embedding vector is processed by the attention mechanism to obtain the collaborative context vector of each agent. The collaborative context vector is concatenated with the global state information and then input into the policy network of the corresponding agent. The local actions output by each policy network constitute the joint action. The global state information and the joint action are used as input to the global critique network, which is used to evaluate the quality of the joint action.

3. The method for allocating resources in a post-disaster wireless communication network according to claim 2, characterized in that, The specific details of training each policy network using a multi-agent reinforcement learning model are as follows: Step 1: Intelligent Agent The policy network extracts the historical local action results of the corresponding agent and calculates its corresponding historical policy entropy sequence; where: exist In the next iteration, the agent policy entropy for: ； in It refers to the first In the next iteration, for the intelligent agent The deployed policy network is for intelligent agents The strategy adopted is as follows: the local observation information of all agents constitutes the global state information. intelligent agent Policy network acquisition And based on the current network parameters The output is used to guide the agent. Local Actions ; in , For the set of all intelligent agents, For the first In the next iteration, the agent Local observation information; Then in the In the next iteration, the agent Historical strategy entropy sequence for: ； in For the first In the next iteration, the agent The strategy entropy, to , The preset historical observation length; Step 2: Obtain the temporal feature vector; For intelligent agents The deployed policy network will enable agents The historical strategy entropy sequence is input into a recurrent neural network, from which the recurrent neural network extracts the agent. Temporal feature vectors Specifically: ； in These are the network parameters of a recurrent neural network; Step 3: Obtain the historical policy embedding vector of the agent; The temporal feature vector output by the recurrent neural network The input is a multilayer perceptron, which processes the temporal feature vector. Mapping to a high-dimensional feature space yields the intelligent agent. Historical strategy embedding vector : ； in These are the network parameters of the multilayer perceptron; Step 4: Construct normalized attention weights for each agent among the historical policy embedding vectors of all other agents; details are as follows: Step 4.1: Connect the intelligent agent Historical strategy embedding vector As a query vector, the historical policies of all agents are embedded into the vector. As the key vector, dot product attention is used to measure its value as the query vector. Define the agent based on its correlation with any vector in the key vector. For intelligent agents Historical strategy embedding vector The scoring function is : ； in, and These are the query weight matrix and the key weight matrix, respectively. It is the dimension of the key vector, and T is the transpose; Step 4.2: Normalize the score function using the Softmax function to obtain the agent. For intelligent agents Normalized attention weights of historical strategy embedding vectors : ； in For intelligent agents The scoring function for the historical policy embedding vector of agent k; Step 4.3: Embed the historical policy of agent i into the vector As a value vector, and with normalized attention weights Perform a weighted summation to obtain the intelligent agent. Collaboration context vector : ； in, It is a value weight matrix; Step 5: Concatenate the cooperative context vector with the global state information and input it into the policy network of the corresponding agent; the specific content is as follows: global state information With intelligent agents Collaboration context vector By splicing them together, an intelligent agent is formed. Enhanced feature vectors intelligent agents of Input the corresponding policy network, and the policy network will... Output it as an intelligent agent Selected local action and transmit it to the corresponding intelligent agent. ; Step 6: The local actions output by each policy network constitute the joint action. ; global state information With joint actions The inputs to the global critical network are shared, and the global critical network is used to evaluate joint actions. The advantages and disadvantages; Step 7: Update the network parameters of the policy network Network parameters of recurrent neural networks Network parameters of multilayer perceptron and the network parameters of the global critique network ; Step 7.1: Determine the network parameters used to update the policy network. The objective function is as follows: Each agent executes the local action output by the corresponding policy network. Then, through interaction with the environment, the immediate reward for this iteration is obtained. Global state information for the next iteration And calculate long-term discount rewards. ,in It is a discount factor. Indicates the number of iterations from the current iteration. Begin, look ahead. step; For the first The immediate reward at the next iteration; and the rewards generated in each iteration. These are called experience pairs and are stored in the experience replay pool. middle; Global critique networks employ common Q-functions and V-functions to evaluate the joint actions of all agents. An evaluation was conducted to obtain the state-action value function related to the long-term reward of the joint action. With state value function The formula is as follows: ； in , for the first In the next iteration, each policy network adopts a policy for its corresponding agent. A combined strategy formed by piecing together elements; This represents the expected value of long-term discount rewards; Global Criticism Network estimates the global dominance function using a truncated generalized dominance estimation method. Used for evaluation The degree of superiority or inferiority is defined as follows: ； Network parameters of the guiding strategy network Updated target function Defined as: ； in, Indicates the replay of experiences from the pool Global state information obtained from mid-sample and joint actions Find the expected value; , representing intelligent agents The ratio of sampling probabilities between the old and new strategies; This represents the agent at the (t-1)th iteration. The corresponding policy network is the agent. The strategy adopted; Indicates the shearing parameter; Step 7.2, Intelligent Agent Network parameters of the strategy network Network parameters of recurrent neural networks Network parameters of multilayer perceptron and the network parameters of the global critique network The update process is as follows: (1) Network parameters of the policy network Update process: Maximize the agent Objective function of policy update Thus, the first In the next iteration, the agent The optimal network parameters for the strategy network : ； intelligent agent The network parameter update process for the policy network is as follows: ； in For intelligent agents The learning rate of the policy network; To solve using the Adam optimizer Compared to The partial derivative; (2) Network parameters of recurrent neural networks Network parameters of multilayer perceptron The update process is as follows: ； in, It is the learning rate of recurrent neural networks and multilayer perceptrons; Describe the objective function For recurrent neural network parameters gradient, Describe the objective function Multilayer perceptron parameters The gradient; (3) Network parameters of the global critique network Update process: For a global critique network, parameters are updated by minimizing the mean squared error loss function, where the mean squared error loss function... Defined as: ； in This indicates that the expression within the parentheses refers to the experience replay pool. The empirical data obtained from sampling Find the average; This indicates that the mean square error of the expression within the parentheses is calculated. For all agents in global state information The following joint actions were taken. Then continue to follow the current joint strategy in the long term. The expected long-term discount reward value that can be obtained For global state information Based on the current joint strategy The joint action obtained at the (t+1)th iteration; In the Optimal global criticism network parameters at the next iteration The definition is as follows: ； The network parameter update process of the global critique network is as follows: ； in The learning rate of the global critique network; Step 7.3: Repeat step 7.2 for multiple iterations until the agent receives an immediate reward through interaction with the environment. Value convergence, intelligent agent The strategy network finds the optimal network parameters. .

Images