Game confrontation-oriented multi-agent air-ground collaborative decision-making system

Abstract

A game confrontation-oriented multi-agent air-ground collaborative decision-making system belongs to the field of game confrontation and artificial intelligence, and comprises a command and control module used for acquiring information of a collaborative target sensing module, a collaborative information monitoring module and a collaborative communication interference module, calling embedded technologies and algorithms, and completing service registration and service requests; the collaborative target sensing module is used for outputting sensing information, receiving control information of the command and control module and executing collaborative target sensing service; the collaborative information monitoring module is used for outputting monitoring information, receiving control information of the command and control module and executing collaborative information monitoring service; and the cooperative communication interference module is used for outputting interference information, receiving control information of the command and control module and executing cooperative communication interference service. According to the invention, an effective interference strategy can be rapidly specified according to the target situation information to complete the cooperative interference problem.

Description

Technical Field

[0001] The present invention belongs to the technical field of the intersection of game confrontation, artificial intelligence and simulation exercises, and specifically relates to a multi-agent air-ground collaborative decision-making system for game confrontation. Background Art

[0002] Faced with complex and unknown communication environments, a large amount of real-time information from the physical environment, and a variety of cross-domain resources, achieving the positioning and interference of target communication signals is a thorny problem often encountered in related research at this stage.

[0003] Traditional resource allocation mainly relies on communication interference schemes manually formulated by expert experience, which are affected by objective human factors. In addition, facing complex environments, it is no longer applicable to rely solely on manually formulated resource allocation schemes. Of course, there are also heuristic search algorithms, such as genetic algorithms, particle swarm algorithms, ant colony algorithms, etc. Although such algorithms can quickly allocate resources in simple scenarios, when the environment state is complex and there are many location parameters, it is difficult to quickly adjust the interference scheme according to changes in the environment.

[0004] Therefore, it is of great significance to achieve precise positioning and formulate effective interference plans in complex and unknown environments, large amounts of real-time information and multiple resources. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention provides a multi-agent air-ground collaborative decision-making system for game confrontation, which can accurately detect targets, precisely locate targets and obtain target situation information through collaboration, and then quickly allocate interference resources.

[0006] The technical solution adopted by the present invention to solve the above problems is: A multi-agent air-ground collaborative decision-making system for game confrontation, wherein the system interacts with an information management platform for data generation, and the information management platform is used to generate simulation operation data for game confrontation; the system comprises: The command and control module is used to obtain information from the collaborative target perception, collaborative information monitoring and collaborative communication interference modules, call embedded technologies and algorithms, and complete service registration and service requests; A collaborative target perception module, used to output perception information; and used to receive information from the information management platform and control information from the command and control module, and perform collaborative target perception services; A collaborative information monitoring module, used to output monitoring information; and used to receive information from the information management platform and control information from the command and control module, and perform collaborative information monitoring services; The collaborative communication interference module is used to output interference information; and to receive control information from the command and control module to perform collaborative communication interference services.

[0007] Furthermore, the collaborative target perception module has collaborative frequency band search function, collaborative direction finding function, collaborative positioning function and collaborative target confirmation function; according to the service request, the command and control module calls the service governance technology to generate target perception behavior; the collaborative target perception module executes the target perception service and returns the perception information to the command and control module; the command and control module calls the collaborative positioning algorithm to calculate the target's location coordinates.

[0008] Furthermore, the collaborative information monitoring module has collaborative tracking and monitoring functions and collaborative situational awareness functions; based on service requests, the command and control module calls service governance technology to generate information monitoring behavior; the collaborative information monitoring module executes information monitoring services and periodically returns monitoring information to the command and control module; based on the acquired periodic target information, the command and control module calls the fusion algorithm to obtain the target's situation information.

[0009] Furthermore, the collaborative communication interference module has a collaborative interference execution function; according to the target situation information and service request, the command and control module calls the collaborative interference algorithm to generate interference action, and then calls the service governance technology to generate interference behavior; the collaborative communication interference module executes communication interference service; the command and control module performs interference effect evaluation, and updates and optimizes the interference behavior according to the current effect evaluation result.

[0010] The service governance technology includes three parts: service registration, service request and service execution. The process is as follows: (1) Service registration: When a service provider starts up, it registers its service name and identity ID in the registration center; (2) Service request: The command and control module makes a request for the required service based on the service demand. After receiving the service request, the command and control module queries the registration center for available service addresses based on the distributed fast retrieval algorithm. After obtaining the list of all available service addresses, the command and control module selects a suitable service provider to initiate a service call request. If the requested service is found to be unavailable, the command and control module periodically calls service discovery until an available service is found or the service request is exited. (3) Service execution: The command and control module initiates a service call request to the service provider. The service provider executes the corresponding service according to the service content and returns the service execution result.

[0011] Furthermore, the collaborative interference algorithm includes the following steps: Step 1: Construct a Markov decision process and adopt the decision-making method of "sequential decision-making and centralized training". The command and control module regards each jamming platform as an independent intelligent agent and decides on jamming actions in sequence through the same network. After the current jamming platform makes a decision, it executes the jamming action and inputs the changed target state information to the next intelligent agent. The centralized training method is used to update the decision network parameters as a whole. Step 2: Construct the state space S, action space A and interference effect judgment conditions of the interference platform; Step 3: Determine the reward function R according to the optimization goal determined by the jammer, that is, the optimization goal of minimizing the number of jamming platforms and the consumption of power resources as much as possible under the premise of achieving overall suppression of the target frequency hopping signal; Step 4: Based on the constructed state space and action space, establish a deep reinforcement learning neural network based on the Actor-Critic method, which includes a strategy execution network composed of Actor modules and a value evaluation network composed of Critic networks; set the number of network training times , the number of training steps in each training round , initialize the parameters of the strategy execution network and the value evaluation network, and initialize the state space; set up data cache B, and put the interaction between each interference platform and the environment in data cache B; Step 5: Start the first round of training in the established deep reinforcement learning neural network. According to the initial network parameter settings, start from the initial environment state implement , through interaction in the environment, that is, obtaining the next moment's environmental state based on the calculation of the interference effect , and the reward value corresponding to the action is calculated through the reward function R , and store the data obtained in the process in data cache B, and then store the environment state Set as the starting state of the next moment, and repeat the above process to obtain the trajectories of all interference platforms; if the actual number of training steps in step 4 reaches the set , or the set termination condition is reached, the current round of training is stopped; Step 6: When training the network, input all the trajectory data in cache B into the network, update the value assessment network parameters and strategy execution network parameters; enter the next round of training, repeat steps 5 and 6 until the number of training rounds reaches After completing the training.

[0012] Further, in step 2, the state space S is defined as , where h represents the number of frequencies of a target frequency hopping signal that is not interfered with; the action space definition A is , where a represents the jamming action, which is the action encoding of the jamming platform, jamming direction angle and jamming power; the jamming effect is expressed by the interference-to-signal ratio.

[0013] Furthermore, in step 3, the reward function includes three parts, namely, basic reward, interference effectiveness reward and interference power reward; the reward function is designed as follows: (3.1) Basic reward: This reward is the direct goal of coordinated interference, that is, to interfere with all targets as much as possible; (3.2) Jamming effectiveness reward: This reward is mainly used to guide the jamming platform to jam as many targets as possible; (3.3) Interference power reward: This reward is mainly used to guide the interference platform to implement interference with the smallest interference power possible, thereby reducing interference resources.

[0014] Furthermore, in step 4, the Actor-Critic method includes an Actor network based on a policy function method and a Critic network based on a value function method, which is a combination of the advantages of policy-based and value-based reinforcement learning algorithms, making up for the shortcoming that the value function-based method cannot display policy generation, and greatly reducing the variance of the value estimation updated by Monte Carlo sampling in the policy function-based method; In the Actor network, it decides an action a based on the current state of the environment s of the agent. After interacting with the environment, it obtains an immediate reward through reward function calculation. Finally, the Critic network evaluates the value of the action and updates the strategy function. Parameters , maximize the current policy function, ; in, represents the stationary distribution of the strategy, represents the Actor network, and represents the Critic network; In the Critic network, it evaluates the value of action a made by the Actor network in state s based on the state value function, and updates the neural network parameters through the temporal difference error established by the immediate reward. , ; in represents the target network, whose parameters are updated in the same way as the value function-based method; Correspondingly, the Actor-Critic network parameters and Updated to, ; ; in, and are the learning rates of the Actor network and the Critic network respectively.

[0015] The beneficial effects of the present invention are: 1. The present invention designs a collaborative positioning algorithm and a fusion algorithm to achieve collaborative and accurate target discovery, precise target positioning, and acquisition of target situation information; 2. The present invention designs a service governance technology that can quickly discover services and complete service calls based on service requests; 3. The communication node of the present invention is provided with a frequency hopping communication anti-interference method, which can effectively interfere with this communication scenario; 4. The present invention introduces a reinforcement learning model with sequential decision-making and centralized training, which solves the problem of non-stationary multi-agent environment and obtains good training convergence performance; 5. The present invention can provide the optimal interference resource allocation scheme while ensuring the interference effect, greatly saving interference resources and reducing resource waste; BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the game confrontation scenario; Figure 2 is a system block diagram of the present invention; Figure 3 It is a schematic diagram of service governance technology; Figure 4 This is the implementation flow chart of the Actor-Critic based collaborative interference algorithm; Figure 5 It is a schematic diagram of the collaborative decision-making method of "sequential decision-making and centralized training". DETAILED DESCRIPTION

[0017] The present embodiment is described in detail below with reference to the accompanying drawings.

[0018] Example 1 Reference Figure 1~Figure 5 , a multi-agent air-ground collaborative decision-making system for game confrontation, which is based on Figure 1 The simulation is carried out in the game confrontation scenario described in the paper. This scenario is a scenario in which the interference party's multiple platforms interfere with the communication nodes of the communication party through collaborative decision-making. It is assumed that the airspace communication party uses frequency hopping signals to send information, and the transmission power remains unchanged during the information transmission. It is assumed that the interference party deploys an air-to-ground multi-platform network that supports different services. The command platform performs search, reconnaissance and strike tasks by collaboratively controlling the search platform, direction finding platform, directional platform, and interference platform, thereby destroying the communication of the communication system. Among them, the search platform, direction finding platform, and directional platform are aerial drones and ground radar equipment, which support collaborative target perception services and collaborative information monitoring services. The interference platform is a ground jammer, which uses comb spectrum interference to suppress the target frequency hopping signal.

[0019] Example 2 Reference Figure 1~Figure 5, a multi-agent air-ground collaborative decision-making system for game confrontation, the simulation operation data required by it is generated by the information management platform, including spectrum information data and location information data. Figure 1 In the game confrontation scenario, the result data returned when executing the target perception and information monitoring tasks is input into the collaborative target perception module and the collaborative information monitoring module as information of the information management platform. Figure 2 As shown, the simulation system includes:

[0020] The command and control module is used to obtain information from the collaborative target perception, collaborative information monitoring and collaborative communication interference modules, call embedded technologies and algorithms, and complete service registration and service requests; A collaborative target perception module, used to output perception information; and used to receive information from the information management platform and control information from the command and control module, and perform collaborative target perception services; A collaborative information monitoring module, used to output monitoring information; and used to receive information from the information management platform and control information from the command and control module, and perform collaborative information monitoring services; The collaborative communication interference module is used to output interference information; and to receive control information from the command and control module to perform collaborative communication interference services.

[0021] The collaborative target perception module of the multi-agent air-ground collaborative decision-making system for game confrontation described in this embodiment is configured with a frequency band search platform equipped with a spectrum analyzer, a direction finding platform equipped with a direction finder, and a target confirmation platform equipped with a radar, and has collaborative frequency band search functions, collaborative direction finding functions, collaborative positioning functions, and collaborative target confirmation functions; according to the service request, the command and control module calls the service governance technology to generate target perception behavior; the collaborative target perception module executes the target perception service and returns the perception information to the command and control module, including the spectrum information monitored by the spectrum analyzer, the deflection information monitored by the direction finder, and the coordinate information monitored by the radar device; the command and control module calls the collaborative positioning algorithm to calculate the position coordinates of the target; The collaborative positioning algorithm adopts the classic three-point positioning algorithm. It constructs a set of equations through the deflection information returned by three direction-finding platforms, and uses the least squares method to solve the set of equations to obtain the position coordinates of the target.

[0022] The collaborative information monitoring module of the multi-agent air-ground collaborative decision-making system for game confrontation described in this embodiment has collaborative tracking and monitoring functions and collaborative situation awareness functions; according to the service request, the command and control module calls the service governance technology to generate information monitoring behavior; the collaborative information monitoring module executes the information monitoring service and periodically returns the monitoring information to the command and control module, including the spectrum information monitored by the spectrum analyzer, the deflection information monitored by the direction finder, and the coordinate information monitored by the radar equipment; according to the periodic target information obtained, the command and control module calls the fusion algorithm to obtain the situation information of the target; On the one hand, the fusion algorithm performs track fusion, fusing related tracks to form a more accurate fused track; on the other hand, it fuses the target's spectral characteristics (frequency, number of frequency points), enemy, friend, country and other attributes; and finally forms the target's situation information in the command and control module.

[0023] The collaborative communication interference module of a multi-agent air-ground collaborative decision-making system for game confrontation described in this embodiment is equipped with a jammer-loaded jammer platform and has a collaborative interference execution function; according to target situation information and service requests, the command and control module calls the collaborative interference algorithm to generate interference actions, and then calls the service governance technology to generate interference behaviors; the collaborative communication interference module executes interference services; the command and control module evaluates the interference effect, and updates and optimizes the interference behavior based on the current effect evaluation results.

[0024] Example 3 A multi-agent air-ground collaborative decision-making system for game confrontation, serving governance technologies such as Figure 3 As shown, it includes service registration, service request and service execution. The process is as follows: (1) Service registration: When a service provider starts up, it registers its service name and identity ID in the registration center; (2) Service request: The command and control module makes a request for the required service based on the service demand. After receiving the service request, the command and control module queries the registration center for available service addresses based on the distributed fast retrieval algorithm. After obtaining the list of all available service addresses, the command and control module selects a suitable service provider to initiate a service call request. If the requested service is found to be unavailable, the command and control module periodically calls service discovery until an available service is found or the service request is exited. (3) Service execution: The command and control module initiates a service call request to the service provider. The service provider executes the corresponding service according to the service content and returns the service execution result.

[0025] Example 4 A multi-agent air-ground collaborative decision-making system for game confrontation, the collaborative interference algorithm is as follows Figure 4 As shown, the following steps are included: Step 1: Construct a Markov decision process and adopt the collaborative decision-making method of "sequential decision-making and centralized training"; Figure 5 As shown in the figure, each interference platform is regarded as an independent intelligent agent, and the interference actions are decided in turn through the same network; after the current interference platform makes a decision, it executes the interference action and inputs the changed target state information to the next intelligent agent; the centralized training method is used to update the decision network parameters from the whole; Step 2: Construct the state space S, action space A and interference effect judgment conditions of the interference platform of the interference party; In step 2, the state space S is defined as , where h represents the number of frequencies of a target frequency hopping signal that is not interfered with; the action space definition A is , where a represents the jamming action, which is the action code of the jamming platform, jamming direction angle and jamming power; the jamming effect is expressed by the interference-signal ratio, and the mathematical expression is, ; in, is the product of the antenna gains, Signal propagation distance, represents the effective interference power, The indicator value indicating whether the interference frequency band of the interference station is aligned with the target frequency band is expressed as, ; Step 3: Determine the reward function R according to the optimization goal determined by the jammer, that is, the optimization goal of minimizing the number of jamming platforms and the consumption of power resources as much as possible under the premise of achieving overall suppression of the frequency hopping signal; In step 3, the reward function includes three parts, namely, basic reward, interference effectiveness reward and interference power reward; the reward function is designed as follows: (3.1) Basic reward: This reward is the direct goal of coordinated interference, that is, to interfere with all targets as much as possible; therefore, the basic reward is defined as follows, ; (3.2) Jamming effectiveness reward: This reward is mainly used to guide the jamming platform to jam as many targets as possible; therefore, the jamming effectiveness reward is defined as follows: ; in is the number of targets that have been interfered, The number of targets successfully interfered by the current interference platform; (3.3) Interference power reward: This reward is mainly used to guide the interference platform to implement interference with as little interference power as possible, thereby reducing interference resources; therefore, the interference power reward is defined as follows: ; in is the weight factor, is the interference power of the current interference platform; In summary, the complete reward function is defined as, ; Step 4: Based on the constructed state space and action space, establish a deep reinforcement learning neural network based on the Actor-Critic method, which includes a strategy execution network composed of Actor modules and a value evaluation network composed of Critic networks; set the number of network training times , the number of training steps in each training round , initialize the parameters of the strategy execution network and the value evaluation network, and initialize the state space; set up data cache B, and put the interaction between each interference platform and the environment in data cache B; In step 4, the Actor-Critic method includes an Actor network based on a policy function method and a Critic network based on a value function method, which is a combination of the advantages of policy-based and value-based reinforcement learning algorithms, making up for the shortcoming of the value function-based method that cannot display policy generation, and greatly reducing the variance of the value estimation updated by Monte Carlo sampling in the policy function-based method; In the Actor network, it decides an action a based on the current state of the environment s of the agent. After interacting with the environment, it obtains an immediate reward through reward function calculation. Finally, the Critic network evaluates the value of the action and updates the strategy function. Parameters , maximize the current policy function, ; in, represents the stationary distribution of the strategy, represents the Actor network, and represents the Critic network; In the Critic network, it evaluates the value of action a made by the Actor network in state s based on the state value function, and updates the neural network parameters through the temporal difference error established by the immediate reward. , ; in represents the target network, whose parameters are updated in the same way as the value function-based method; Correspondingly, the Actor-Critic network parameters and Updated to, ; ; in, and They are the learning rates of the Actor network and the Critic network respectively; Step 5: Start the first round of training in the established deep reinforcement learning neural network. According to the initial network parameter settings, start from the initial environment state implement , through interaction in the environment, that is, obtaining the next moment's environmental state based on the calculation of the interference effect , and the reward value corresponding to the action is calculated through the reward function R , and store the data obtained in the process in data cache B, and then store the environment state Set as the starting state of the next moment, and repeat the above process to obtain the trajectories of all interference platforms; if the actual number of training steps in step 4 reaches the set , or the set termination condition is reached, the current round of training is stopped; Step 6: When training the network, input all the trajectory data in cache B into the network, update the value assessment network parameters and strategy execution network parameters; enter the next round of training, repeat steps 5 and 6 until the number of training rounds reaches After completing the training.

[0026] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Obviously, for professionals in this field, after understanding the content and principle of the present invention, they may make various simple modifications and equivalent substitutions in details without departing from the content of the technical solution of the present invention. However, these modifications and substitutions based on the present invention are still within the protection scope of the technical solution of the present invention.

Claims

1. A multi-agent air-ground collaborative decision-making system for game confrontation, wherein the system interacts with an information management platform for data generation, and the information management platform is used to generate simulation operation data for game confrontation; characterized in that: The system comprises: The command and control module is used to obtain information from the collaborative target perception, collaborative information monitoring and collaborative communication interference modules, call embedded technologies and algorithms, and complete service registration and service requests; A collaborative target perception module, used to output perception information; and used to receive information from the information management platform and control information from the command and control module, and perform collaborative target perception services; A collaborative information monitoring module, used to output monitoring information; and used to receive information from the information management platform and control information from the command and control module, and perform collaborative information monitoring services; The collaborative communication interference module is used to output interference information; and to receive control information from the command and control module to perform collaborative communication interference services.

2. The multi-agent air-ground collaborative decision-making system for game confrontation as claimed in claim 1, characterized in that: The collaborative target perception module has collaborative frequency band search function, collaborative direction finding function, collaborative positioning function and collaborative target confirmation function; according to the service request, the command and control module calls the service governance technology to generate target perception behavior; The collaborative target perception module performs target perception services and returns perception information to the command and control module; the command and control module calls the collaborative positioning algorithm to calculate the target's position coordinates.

3. The multi-agent air-ground collaborative decision-making system for game confrontation as claimed in claim 1, characterized in that: The collaborative information monitoring module has collaborative tracking and monitoring functions and collaborative situation awareness functions; according to the service request, the command and control module calls the service governance technology to generate information monitoring behavior; the collaborative information monitoring module executes the information monitoring service and periodically returns the monitoring information to the command and control module; according to the acquired periodic target information, the command and control module calls the fusion algorithm to obtain the target's situation information.

4. The multi-agent air-ground collaborative decision-making system for game confrontation as claimed in claim 1, characterized in that: The collaborative communication interference module has a collaborative interference execution function; according to the target situation information and service request, the command and control module calls the collaborative interference algorithm to generate interference action, and then calls the service governance technology to generate interference behavior; the collaborative communication interference module executes communication interference service; the command and control module evaluates the interference effect, and updates and optimizes the interference behavior according to the current effect evaluation result.

5. A multi-agent air-ground collaborative decision-making system for game confrontation as claimed in any one of claims 2 to 4, characterized in that: The service governance technology includes service registration, service request and service execution. The process is as follows: (1) Service registration: When a service provider starts up, it registers its service name and identity ID in the registration center; (2) Service request: The command and control module makes a request for the required service based on the service demand. After receiving the service request, the command and control module queries the registration center for available service addresses based on the distributed fast search algorithm. After obtaining the list of all available service addresses, it selects the appropriate service provider to initiate a service call request. If the requested service is found to be unavailable, the command and control module periodically calls service discovery until an available service is found or the service request is exited; (3) Service execution: The command and control module initiates a service call request to the service provider. The service provider executes the corresponding service according to the service content and returns the service execution result.

6. The multi-agent air-ground collaborative decision-making system for game confrontation as claimed in claim 4, characterized in that: The collaborative interference algorithm comprises the following steps: Step 1: Construct a Markov decision process and adopt the decision-making method of "sequential decision-making and centralized training". The command and control module regards each jamming platform as an independent intelligent agent and decides on jamming actions in sequence through the same network. After the current jamming platform makes a decision, it executes the jamming action and inputs the changed target state information to the next intelligent agent. The centralized training method is used to update the decision network parameters as a whole. Step 2: Construct the state space S, action space A and interference effect judgment conditions of the interference platform; Step 3: Determine the reward function R according to the optimization goal determined by the jammer, that is, the optimization goal of minimizing the number of jamming platforms and the consumption of power resources as much as possible under the premise of achieving overall suppression of the target frequency hopping signal; Step 4: Based on the constructed state space and action space, establish a deep reinforcement learning neural network based on the Actor-Critic method, which includes a strategy execution network composed of Actor modules and a value evaluation network composed of Critic networks; set the number of network training times , the number of training steps in each training round , initialize the parameters of the strategy execution network and the value evaluation network, and initialize the state space; set up data cache B, and put the interaction between each interference platform and the environment in data cache B; Step 5: Start the first round of training in the established deep reinforcement learning neural network. According to the initial network parameter settings, start from the initial environment state implement , through interaction in the environment, that is, obtaining the next moment's environmental state based on the calculation of the interference effect , and the reward value corresponding to the action is calculated through the reward function R , and store the data obtained in the process in data cache B, and then store the environment state Set as the starting state of the next moment, and repeat the above process to obtain the trajectories of all interference platforms; if the actual number of training steps in step 4 reaches the set , or the set termination condition is reached, the current round of training is stopped; Step 6: When training the network, input all the trajectory data in cache B into the network, update the value assessment network parameters and strategy execution network parameters; enter the next round of training, repeat steps 5 and 6 until the number of training rounds reaches After completing the training.

7. The multi-agent air-ground collaborative decision-making system for game confrontation as claimed in claim 6, characterized in that: In step 2, the state space S is defined as , where h represents the number of frequencies of a target frequency hopping signal that are not interfered with; The action space A is defined as , where a represents the jamming action, which is the action encoding of the jamming platform, jamming direction angle and jamming power; the jamming effect is expressed by the interference-to-signal ratio.

8. The multi-agent air-ground collaborative decision-making system for game confrontation as claimed in claim 6, characterized in that: In step 3, the reward function includes three parts, namely, basic reward, interference effectiveness reward and interference power reward; the specific reward function is designed as follows: (3.1) Basic reward: This reward is the direct goal of coordinated interference, that is, to interfere with all targets as much as possible; (3.2) Jamming effectiveness reward: This reward is mainly used to guide the jamming platform to jam as many targets as possible; (3.3) Interference power reward: This reward is mainly used to guide the interference platform to implement interference with the smallest interference power possible, thereby reducing interference resources.

9. The multi-agent air-ground collaborative decision-making system for game confrontation as claimed in claim 6, characterized in that: In the step 4, the Actor-Critic method includes an Actor network based on a policy function method and a Critic network based on a value function method, which is a combination of the advantages of policy-based and value-based reinforcement learning algorithms, making up for the shortcoming of the value function-based method that cannot display policy generation, and greatly reducing the variance of the value estimation updated by Monte Carlo sampling in the policy function-based method; In the Actor network, it decides an action a based on the current state of the environment s of the agent. After interacting with the environment, it obtains an immediate reward through reward function calculation. Finally, the Critic network evaluates the value of the action and updates the strategy function. Parameters , maximize the current policy function, ； in, represents the stationary distribution of strategies, represents the Actor network, and represents the Critic network; In the Critic network, it evaluates the value of action a made by the Actor network in state s based on the state value function, and updates the neural network parameters through the temporal difference error established by the immediate reward. , ； in represents the target network, whose parameters are updated in the same way as the value function-based method; Actor-Critic Network Parameters and Updated to, ； ； in, and are the learning rates of the Actor network and the Critic network respectively.

Images