Multi-agent collaborative exploration method and system based on collaborative target network

Abstract

The invention discloses a multi-agent collaborative exploration method and system based on a collaborative target network, and aims to solve the problems that in multi-agent collaborative exploration, the exploration efficiency is low due to sparse rewards, and random network distillation is difficult to adapt to a multi-agent scene. According to the method, a collaborative exploration environment is constructed in a continuous space, collaborative target networks matched with intelligent agents in number are designed, so that each intelligent agent corresponds to a randomly fixed exclusive target network, a reference distribution cluster is formed, and a trainable global predictor is combined. In the aspect of a reward mechanism, a mean error reward is adopted to guide exploration in the early stage, and a false count reward is utilized to relieve the sparse reward problem in the later stage. The overall framework is based on centralized training and distributed execution, and a value network and a distributed strategy network are fused to optimize a collaborative exploration strategy. According to the method, multiple agents can realize efficient cooperative exploration only by depending on local observation and limited communication, and the integrity and task suitability of exploration in a continuous space are improved.

Description

Technical Field

[0001] This invention belongs to the field of multi-agent cooperative control and reinforcement learning technology, and particularly relates to a multi-agent cooperative exploration method and system that solves the sparse reward problem in continuous space through a cooperative target network mechanism of "one-to-one correspondence between agent and target network" and is based on a centralized training-distributed execution architecture. Background Technology

[0002] In multi-agent continuous spatial collaborative exploration tasks (such as UAV swarm area mapping and ground robot team environmental exploration), sparse rewards are the core bottleneck restricting exploration efficiency. In existing technologies, although intrinsic rewards can be generated through random network distillation (RND) mechanisms to alleviate the sparsity problem, traditional RND schemes generally use a single target network for exploration. The target network cannot match the exploration perspective of each agent, resulting in a lack of specificity in the baseline distribution. The exploration behavior of some agents is difficult to be effectively guided, leading to "repeated exploration" or "regional omission".

[0003] Meanwhile, existing multi-agent reinforcement learning methods, if employing a fully distributed architecture, lack a global collaborative perspective, easily leading to overlapping exploration by agents; if employing a fully centralized architecture, they rely on high-bandwidth real-time communication, resulting in poor robustness in actual deployment. These problems collectively lead to low exploration efficiency and slow training convergence of multi-agents in continuous space, making it difficult to meet the requirements of practical tasks. Summary of the Invention

[0004] To address the inefficiency of multi-agent cooperative exploration caused by sparse rewards and insufficient target network-agent adaptability, this invention proposes a multi-agent cooperative exploration method and system based on a cooperative target network. The core innovation lies in: designing… Each agent corresponds to A collaborative target network cluster with "a dedicated fixed target network" is used to generate targeted intrinsic rewards by combining a random network distillation mechanism. At the same time, relying on the CTDE architecture, "centralized optimization of collaborative strategies and distributed execution of exploration tasks" are realized, ultimately improving the exploration efficiency and robustness of multi-agents in continuous space.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A multi-agent cooperative exploration method based on cooperative target networks includes the following steps: S101, Constructing includes The decision-making environment is explored in a continuous spatial collaborative manner by multiple agents, and the attributes of each agent are initialized. S102. Define the local observation information of each agent, construct an independent state space based on local observation for each agent, and define the continuous action space of the decision level. S103. Utilize the random network distillation mechanism to build a system containing... A cluster of dedicated fixed-target networks and A trainable prediction network generates an exploration reward by calculating the mean square error of the prediction network's output and that of a randomly sampled dedicated fixed target network on the global state observations; the global state is generated by splicing together the local observations of all agents. S104. The total reward signal is composed of the fusion of exploration rewards and external environment rewards. The collaborative exploration strategy is optimized through a multi-agent reinforcement learning algorithm with centralized training and distributed execution. S105. Using the trained policy network, each agent autonomously generates decision actions based on its own local observations, driving the multi-agent group to perform continuous spatial collaborative exploration tasks.

[0006] In an embodiment of the present invention, the attributes of the agent in S101 include maximum speed. With perception radius The continuous space is a two-dimensional planar region, and the environmental dimensions can be configured according to exploration needs, for example, as follows: During initialization, Several agents are randomly distributed within an effective region of continuous space, with the initial position spacing not less than the sensing radius. .

[0007] In an embodiment of the present invention, in S102, the local observation of a single agent is... Dimensional vector (in this embodiment) ), including its own absolute position Self-velocity component and other things within the perception range The relative positions of the agents State space: The state space of a single agent is its own. 2D local observation; continuous action space: the action space of a single agent is defined as a two-dimensional velocity vector. Satisfying speed constraints ,in , The agents are respectively in , The velocity component in the direction.

[0008] In an embodiment of the present invention, S103: Each agent corresponds to a dedicated fixed-target network: There are three identical fully connected networks, whose inputs are all global states (with dimensions of 1). The output is a fixed dimension (e.g.) The feature vector is 1000 dimensional; all parameters of the dedicated fixed target network are randomly initialized and remain fixed throughout the training process; the trainable prediction network has the same structure as the dedicated fixed target network, the input is the global state, the output dimension is the same as the dedicated fixed target network, the parameters are trained by gradient descent, and the optimization objective is to minimize the mean square error between the feature vector and the randomly sampled output feature vector of the dedicated fixed target network; the global state is: ... The local observation vectors of each agent are concatenated to generate the data. ,in For the first Local observation of an agent.

[0009] In an embodiment of the present invention, in S103, the prediction network is trained by minimizing a loss function, which is: ,in, To predict the network's output vector, From A dedicated fixed target network Random sampling One output feature vector, This is the global state. To predict the network parameters (updated through training). for The parameters of a dedicated fixed-target network are fixed. Exploration rewards are calculated by comparing the prediction network with... The generation of statistical differences in the output of a dedicated fixed-target network includes: (1) Calculation The mean of the output of a dedicated fixed-target network ; (2) Calculation The second-order central moment output by a dedicated fixed target network ; (3) Generate two-stage exploration rewards: early exploration rewards Later exploration rewards ; (4) Final exploration reward ,in For fixed weighting coefficients (in this embodiment) ).

[0010] In an embodiment of the present invention, in S104: external environmental reward Includes: positive rewards for exploring unvisited areas (in this embodiment) Negative rewards for collisions between intelligent agents (in this embodiment) ), negative rewards for task timeout (in this embodiment) ), and complete the preset exploration coverage target (such as regional coverage). sparse positive rewards (in this embodiment) Total reward signal ,in The intrinsic reward weighting coefficient (range of values) This embodiment ).

[0011] In an embodiment of the present invention, the multi-agent reinforcement learning algorithm used in S104 is the Multi-Agent Dual-Delay Deep Deterministic Policy Gradient (MATD3) algorithm, whose centralized training process includes: evaluating the value of joint actions using multiple centralized critic networks: the input to the critic network is the global state. and The joint action of individual agents ( ,, For the first The actions of each agent are updated by minimizing the temporal difference error; a distributed actor network is used to generate decision actions for each agent: each actor network relies only on the local observations of its corresponding agent. The goal is to maximize the evaluation value of the critic network through policy gradient updates. After training converges, only the lightweight executor network of each agent is retained. During the task execution phase, each agent inputs its local observations into the executor network to directly generate decision actions.

[0012] Furthermore, during the training process, Historical data generated by the interaction between an agent and its environment (including global state) Joint actions Total Rewards Next global state Task completion marker ) is stored in a shared experience replay pool, from which batch data is sampled (in this embodiment, the batch size is ). Update the network of executives and the network of critics.

[0013] This invention also provides a multi-agent cooperative exploration system based on a cooperative target network, comprising: Environment initialization module: used to build the environment initialization module. A continuous spatial collaborative exploration and decision-making environment for multiple intelligent agents, defining the physical attributes (maximum speed) of each agent. Perception radius ) and initial distribution area; simultaneously initialize A fixed target network specific to each agent; A trainable prediction network, and a centralized commentator network required for multi-agent reinforcement learning. A distributed executor network, with network parameters initialized using Xavier; The learning and training module includes an exploration reward generation unit and a strategy optimization unit; the exploration reward generation unit is used to optimize the strategy based on a random network distillation mechanism. The output error between a dedicated fixed-target network and the prediction network is used to generate exploration rewards in real time. The strategy optimization unit is used for fusion. External environmental rewards We employ an offline training and collaborative exploration strategy based on the MATD3 algorithm with a CTDE architecture. Task execution module: Used to load and deploy the trained agent network, receive real-time local observation information from each agent, output decision-making action instructions, and drive the process. Individual agents perform collaborative exploration tasks in a distributed manner.

[0014] The data interaction between the learning and training module and the task execution module of this invention strictly follows the core principles of centralized training and distributed execution: In the training phase, the policy optimization unit can fully access global information such as the global state (generated by splicing together local observations from all agents), joint actions of multiple agents, and total rewards. It evaluates the value of joint actions through a centralized critic network, accurately guides the parameter updates of the distributed executor network, and ensures that the globally optimal collaborative exploration strategy is learned. In the execution phase, each agent decision-making unit in the task execution module relies only on the local observation information obtained by its own sensors (such as its own position, speed, and the relative position of other agents within its perception range). It does not need to communicate with other agents in real time and directly generates decision actions through the deployed lightweight executor network, realizing a completely decentralized exploration task execution.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention overcomes the limitations of traditional random network distillation, which relies on "adapting a single target network to multiple agents," and designs a new method... Each agent corresponds to A collaborative target network cluster with "each dedicated fixed target network" combined with a two-stage exploration reward system (early mean error reward driving global coverage, and later pseudo-count reward guiding precise supplementation) transforms state "novelty" into dense reward signals tailored to each agent's exploration perspective. This design effectively avoids the "repeated exploration" or "regional omission" problems caused by the "lack of specificity in the baseline distribution" in traditional schemes. Compared to traditional RND schemes, multi-agent regional coverage efficiency is significantly improved. The percentage of unexplored areas has decreased to [amount missing]. the following.

[0016] This invention, based on the CTDE architecture, addresses the architectural pain points of existing technologies through a "centralized training optimization collaboration and distributed execution reduced dependency" model: During training, it utilizes global information (global state, joint actions) to optimize strategies, avoiding exploration overlap caused by the "lack of collaborative perspective" in fully distributed architectures; during execution, it eliminates the need for real-time communication, relying only on local observations and decisions, reducing dependence on high-bandwidth communication and overcoming the "poor anti-interference capability" of fully centralized architectures. In actual deployment, even if a single agent fails, the remaining... An individual intelligent agent can still complete With the above-mentioned coverage, the system's fault tolerance and robustness are significantly improved.

[0017] This invention generates targeted intrinsic rewards through a dedicated target network cluster, providing continuous and effective training feedback to each agent, thus avoiding the training stagnation caused by lack of feedback in reinforcement learning algorithms in sparse reward environments. Simultaneously, the combination of a shared experience replay pool (storing global state, joint actions, total rewards, etc.) and the MATD3 algorithm further improves training stability and data utilization efficiency. Compared to existing multi-agent exploration schemes, the cooperative strategy of this invention achieves faster convergence. Furthermore, all networks adopt a basic fully connected structure with a small parameter size (a single executor network). It supports embedded device deployment and is easy to implement in engineering projects.

[0018] This invention supports The number of agents can be flexibly expanded (e.g. Adapted for dual-agent collaborative exploration, (Adapts to multi-drone swarm mapping), only the global state dimension needs to be adjusted ( Local observation dimensions and the number of dedicated target networks do not require reconstructing the algorithm flow or network structure; simultaneously, continuous spatial dimensions (such as...) The physical attributes of the intelligent agent (maximum speed, perception radius) can be configured according to task requirements, adapting to different carriers such as drone swarms and ground robot teams, as well as various collaborative exploration scenarios such as regional surveying and post-disaster exploration. Its versatility is significantly better than traditional solutions with fixed scale.

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0020] The accompanying drawings, which constitute a part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention. In the drawings: Figure 1 Flowchart for a multi-agent collaborative exploration method.

[0021] Figure 2 A schematic diagram of a simulation environment for multi-agent collaborative exploration.

[0022] Figure 3 This is a schematic diagram of the collaborative target network structure.

[0023] Figure 4 The reward convergence curve for multi-agent collaborative exploration. Detailed Implementation

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of the invention described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] Figure 1 This is a flowchart illustrating a multi-agent cooperative exploration method based on a cooperative target network as described in this invention. The flowchart clearly shows the complete process from environment initialization to final task execution, as follows: Figure 1 As shown, the method includes the following steps: S101, Constructing includes The system enables agents to collaboratively explore and make decisions in a continuous spatial environment and initialize the attributes of each agent.

[0028] Specifically, a two-dimensional continuous planar space is constructed as the collaborative exploration and decision-making environment. The size of the environment can be configured according to task requirements (20m×20m in this embodiment). The environmental boundary is set to a "bounceable" type to prevent the agent from exceeding the exploration range. There are no dynamic targets or static obstacles in the environment; only the focus is on... One intelligent agent (in this embodiment) The collaborative exploration logic between agents. A unified physical attribute is set for each agent: maximum speed. Perception radius (Can only perceive other agents within this radius); the motion of all agents follows a discrete-time dynamic model, at time... intelligent agent Location The updated formula is ,in For intelligent agents exist The velocity vector of motion at any given moment. This is the simulation step size. During the initialization phase... Several agents are randomly distributed within the effective area of ​​the environment, and the initial distance between any two agents is not less than the perception radius. (This embodiment) (to avoid initial collision).

[0029] S102. Define the local observation information of each agent, construct an independent state space based on local observation for each agent, and define the continuous action space of the decision level.

[0030] Specifically, for each agent, local observation, state space, and continuous action space are defined: Local observation for Dimensional vector (in this embodiment) ), including intelligent agents its own absolute position ( (dimensional) and its own velocity components ( (dimensional), and other within the perception range The relative positions of the agents (in this embodiment, relative to the agents) relative position ( (dimensional), and intelligent agents relative position ( (dimensional), and the average distance of the agent within the perception range ( (Dimension, used to assist in determining collaborative density); if an agent exceeds its perception range, the corresponding relative position dimension is assigned a value. Each agent's state space is its own. Local observation This eliminates the need to store global information, thus reducing local computational burden. The action space is defined as a two-dimensional continuous velocity vector. And satisfy the speed constraint ,in , The agents are respectively in , The velocity component in the direction directly corresponds to the agent's movement speed in the action output, without the need for discretization.

[0031] S103. Utilize the random network distillation mechanism to build a system containing... A cluster of dedicated fixed-target networks and A trainable prediction network generates an exploration reward by calculating the mean square error of the global state observations of the prediction network and a randomly sampled dedicated fixed target network; the global state is generated by splicing together the local observations of all agents.

[0032] Specifically, a reward generation model is constructed based on a random network distillation mechanism: First, the global state is defined. ,Depend on The local observation vectors of each agent are concatenated to generate the data. (In this embodiment, the dimension is) ). Setup A cluster consisting of dedicated fixed-target networks, with each agent corresponding to... There are several target networks, all with the same structure (all are "input layer → ...). Neuron ReLU layer → A fully connected network with a "ReLU layer of neurons" as input. Global state The output is 3D feature vectors; parameters of each target network are independently and randomly initialized (following...) The distribution is initialized and then fixed to simulate the exploration perspective of each agent. Simultaneously, a system is built... A trainable prediction network, whose structure is identical to the target network, has parameters denoted as follows: The network is trained using gradient descent, and the optimization objective is to minimize the mean square error of the target network's output feature vectors obtained from random sampling. The loss function is... ,in From Individual target network Random sampling Each output uses the Adam optimizer (learning rate). ) Update parameters. The exploration reward generation process is as follows: first calculate Mean of the output of each target network With second moment ( To (Squaring each element); then generating a two-stage reward, early exploration reward. (Drives global uniform exploration), later exploration rewards (Based on pseudo-counting to guide the supplementation of unexplored areas); final fusion exploration rewards (This embodiment) ).

[0033] S104. The total reward signal is composed of the fusion of exploration rewards and external environment rewards. The collaborative exploration strategy is optimized through a multi-agent reinforcement learning algorithm with centralized training and distributed execution. Specifically, a total reward signal is constructed and the MATD3 algorithm is used to optimize the strategy: external environmental reward. It includes four categories: intelligent agents entering "near" Positive coverage reward when no agent stays in the unvisited area within the step Distance between two agents Negative collision penalty at (collision determination radius) Environmental coverage Sparse positive task rewards Task Exceeding Negative timeout penalty when a step is not completed The total reward signal is ,in The intrinsic reward weighting coefficient (in this embodiment) The MATD3 algorithm is optimized using a centralized training, distributed execution (CTDE) architecture: the centralized commentator network input is... Global state and Joint Operations ( A smart agent Dimensional action, Output global value By minimizing the time-series difference error update, the loss function is: , where the target value ( As a discount factor, (For target commentator parameters); the distributed executive network is configured for each agent. One, input is Local observation Output Wei Action Through policy gradient updates, the objective is to maximize the reviewer network evaluation value, and the loss function is... During training, historical data (global state) generated by the agent's interaction with the environment are used. Joint actions Total Rewards Next global state Task completion marker ) stored in a capacity of A shared experience replay pool, each sampling The data is updated in the network to avoid the influence of sample correlation.

[0034] S105. Using the trained policy network, each agent autonomously generates decision actions based on its own local observations, driving the multi-agent group to perform continuous spatial collaborative exploration tasks.

[0035] Specifically, the strategy deployment and execution process is as follows: After training converges, only retain... A lightweight executor network of agents (each parameter scale) During the task execution phase, the system operates in a fully distributed mode: each agent acquires local observations in real time through its own sensors. (Without needing to communicate with other intelligent agents), will Inputting the onboard executive network directly generates a two-dimensional velocity vector. The intelligent agent is based on a dynamic model. Movement is achieved through implicit division of labor in the trained policy (as in the agent in this embodiment). Explore the left side of the environment, intelligent agent Explore the right side, intelligent agent (Supplementing the intermediate area) enables efficient collaborative exploration without centralized scheduling, ensuring the real-time performance and robustness of task execution.

[0036] Figure 2 This is a schematic diagram of the multi-agent cooperative exploration simulation environment described in this invention. Figure 2 As shown, this instance is in a two-dimensional continuous space, containing The task involves multiple intelligent agents (large blue circles) and several feature points to be explored (small black circles), with no static obstacles or dynamic targets. The core objective is to "collaboratively explore and cover spatial regions containing feature points." To adapt to the requirements of collaborative exploration, The key parameters of each agent are set completely uniformly: the maximum speed is also uniform. The sensing radius is (It can perceive other agents and black feature points within the radius in real time for collision avoidance and exploration guidance). The decision logic for the exploration task is: when an agent moves to the vicinity of a black feature point and the distance between the feature point and the feature point... When the feature point is successfully covered, it is considered "successfully covered"; if Each intelligent agent jointly covers The above black feature points, with a pairwise distance of not less than [missing information] If the collision detection radius is reached (to avoid interference between agents), it is considered a successful cooperative exploration. The motion of the agent (blue circle) follows a discrete-time dynamics model. ( The simulation step size is used, and the interaction with the environmental boundary (the implicit constraint being the continuous spatial range within which the agent can move freely) is only "bounce-like" (to prevent the agent from leaving the exploration area without restriction). This environment accurately simulates the collaborative exploration process of multiple agents on discretely distributed feature points in a cooperative exploration scenario, and can intuitively verify the effectiveness of the method of this invention in the core task of "collision avoidance-dispersion-feature point coverage".

[0037] Figure 3 This is a schematic diagram of the cooperative target network structure used in this invention. Figure 2 The diagram illustrates the core framework for intrinsic reward generation in multi-agent collaborative exploration. The diagram includes elements related to... One-to-one correspondence between intelligent agents A dedicated fixed target network, and Both the training predictive network and the target network take the global state, composed of spliced ​​local observations from all agents, as input. A dedicated, fixed target network with fixed parameters provides a baseline distribution of exploration perspectives for each agent. The predictive network is trained by learning the error between the target network's output and randomly sampled data, and then combines this error with the statistical characteristics of the target network's output to generate a two-stage exploration reward, providing dense intrinsic incentives for the multiple agents. This structure effectively alleviates the sparse reward problem through a targeted reward mechanism, supporting efficient collaborative exploration by multiple agents within a "centralized training-distributed execution" framework.

[0038] Figure 4 This is a convergence curve of the team's average reward during the multi-agent collaborative exploration training process described in this invention. Figure 4 As shown in the figure, the horizontal axis represents the number of training steps, and the vertical axis represents the number of agents in that round (in this embodiment). The average cumulative reward value of the multi-agent team clearly reflects the complete learning process from "inefficient exploration" to "efficient collaboration" as shown by the curve. In the early stages of training, the multi-agent team has not yet formed an effective collaborative strategy, and external rewards in the environment, such as positive rewards for "covering unvisited areas," are extremely sparse. The average team reward remains at a low level. At this time, the reward mainly consists of exploration rewards generated based on the collaborative target network. The error between a dedicated fixed target network and the prediction network provides the agent with intensive incentives to "explore new regions," driving it to break free from "local stagnation." As training progresses, guided by continuous exploration rewards, the agent gradually learns a "distributed and complementary" exploration mode: leveraging... Each dedicated target network characterizes the baseline distribution from different perspectives. Agents spontaneously avoid overlapping exploration areas, unvisited areas are rapidly covered, and external coverage rewards begin to be steadily acquired. This stage corresponds to the rapid rise of the curve, marking the initial formation of the "exploration-coverage" collaborative strategy, and the transition of multiple agents from "blind movement" to "collaborative exploration." In the later stages of training, the agents' collaborative exploration strategy matures: through a lightweight actor network, each agent can autonomously achieve collaborative behavior of "collision avoidance, dispersion, and efficient coverage" relying only on local observations; external rewards and exploration rewards are deeply integrated, and the team's average reward curve tends to converge smoothly and remain in a high value range. This result verifies the effectiveness of the invention "collaborative target network cluster combined with the MATD3 algorithm," which not only solves the problem of inefficient exploration caused by sparse rewards in continuous space, but also enables a multi-agent group to learn a robust collaborative exploration strategy through an architecture of "centralized training to optimize collaboration and distributed execution to reduce dependence." The convergence curve visually demonstrates the evolution of multi-agent collaborative exploration capabilities, proving that the method of this invention can support multi-agents to stably achieve efficient collaborative exploration in continuous space, providing technical support for practical scenarios such as UAV swarm area mapping and ground robot team environmental exploration.

[0039] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-agent cooperative exploration method and system based on cooperative target networks, characterized in that, Includes the following steps: S101, Constructing includes The decision-making environment is explored in a continuous spatial collaborative manner by multiple agents, and the attributes of each agent are initialized. S102. Define the local observation information of each agent, construct an independent state space based on local observation for each agent, and define the continuous action space of the decision level. S103. Utilize the random network distillation mechanism to build a system containing... A cluster of dedicated fixed-target networks and a trainable prediction network are used to generate exploration rewards by calculating the mean square error of the global state observations of the prediction network and the randomly sampled dedicated fixed-target networks; the global state is generated by splicing together the local observations of all agents. S104. The total reward signal is composed of the fusion of exploration rewards and external environment rewards. The collaborative exploration strategy is optimized through a multi-agent reinforcement learning algorithm with centralized training and distributed execution. S105. Using the trained policy network, each agent autonomously generates decision actions based on its own local observations, driving the multi-agent group to perform continuous spatial collaborative exploration tasks.

2. The multi-agent cooperative exploration method based on cooperative target network according to claim 1, characterized in that, In step S101, the agent's attributes include maximum speed. With perception radius ; The continuous space is a two-dimensional planar region, and the environmental dimensions are configured according to exploration requirements; during initialization... Several agents are randomly distributed within an effective region of continuous space, with the initial position spacing not less than the sensing radius. .

3. The multi-agent cooperative exploration method based on cooperative target network according to claim 1, characterized in that, In S102: Local observations of a single agent A dimensional vector, including its own absolute position. Self-velocity component and other things within the perception range The relative positions of the agents ; The state space of a single agent is its own. Dimensional local observation; The continuous action space of a single agent is defined as a two-dimensional velocity vector. Satisfying speed constraints ,in , The agents are respectively in , The velocity component in the direction.

4. The multi-agent cooperative exploration method based on cooperative target network according to claim 1, characterized in that, In S103: Dedicated fixed-target networks refer to networks where each agent corresponds to a specific target network. Three identical fully connected networks, each taking the global state as input, with dimension 1. The output is a feature vector of fixed dimensions; all parameters of the dedicated fixed target network are randomly initialized and remain fixed throughout the training process. The structure of the trainable prediction network is the same as that of the dedicated fixed target network. The input is the global state, and the output dimension is the same as that of the dedicated fixed target network. The parameters are trained by gradient descent. The optimization objective is to minimize the mean square error between the output feature vector of the randomly sampled dedicated fixed target network and the network itself. The global state is determined by The local observation vectors of each agent are concatenated to generate the following representation: ,in For the first Local observation of an agent.

5. The multi-agent cooperative exploration method based on a cooperative target network according to claim 1 or 4, characterized in that, In step S103, the prediction network is trained by minimizing a loss function, which is: ,in, To predict the network's output vector, From A dedicated fixed target network Random sampling One output feature vector, This is the global state. To predict the network parameters, they are updated through training. for The parameters of a dedicated fixed target network remain unchanged.

6. The multi-agent cooperative exploration method based on cooperative target network according to claim 5, characterized in that, In step S103, the exploration reward is calculated by a prediction network and... The statistical differences generated from the output of a dedicated fixed-target network include: (1) Calculation The mean of the output of a dedicated fixed-target network ; (2) Calculation The second-order central moment output by a dedicated fixed target network ; (3) Generate two-stage exploration rewards: early exploration rewards Later exploration rewards ; (4) Final exploration reward ,in These are fixed weighting coefficients.

7. The multi-agent cooperative exploration method based on cooperative target network according to claim 6, characterized in that, In S104: external environmental rewards This includes: positive rewards for exploring unvisited areas, negative rewards for collisions between agents, negative rewards for task timeouts, and sparse positive rewards for completing preset exploration coverage objectives; total reward signal. ,in To explore the reward weighting coefficient.

8. The multi-agent cooperative exploration method based on cooperative target network according to claim 1, characterized in that, In S104: the multi-agent reinforcement learning algorithm is a multi-agent dual-delay deep deterministic policy gradient algorithm, and its centralized training process includes: evaluating the value of joint actions using multiple centralized critic networks: the input to the critic network is the global state. and The joint action of individual agents , For the first The actions of each agent are updated by minimizing the temporal difference error; a distributed network of agents is used to generate decision actions for each agent: each agent network relies only on the local observations of its corresponding agent. The goal is to maximize the evaluation value of the critic network through policy gradient updates. After training converges, only the lightweight executor network of each agent is retained. During the task execution phase, each agent inputs its local observations into the executor network to directly generate decision actions.

9. The multi-agent cooperative exploration method based on cooperative target network according to claim 8, characterized in that, During the training process, Historical data generated from interactions between agents and the environment is stored in a shared experience replay pool. Batch data is sampled from this pool to update the agent network and the critic network. The historical data includes the global state. Joint actions Total Rewards Next global state and task completion marker .

10. A multi-agent cooperative exploration system based on a cooperative target network, characterized in that, include: Environment initialization module: used to build the environment initialization module. A continuous spatial collaborative exploration and decision-making environment for multiple agents is established, defining the physical attributes and initial distribution area of ​​each agent; simultaneously, initialization is performed. A fixed target network specific to each agent; A trainable prediction network, and a centralized commentator network required for multi-agent reinforcement learning. A distributed executor network, with network parameters initialized using Xavier, including physical attributes such as maximum speed. and perception radius ; The learning and training module includes an exploration reward generation unit and a strategy optimization unit; the exploration reward generation unit is used to optimize the strategy based on a random network distillation mechanism. The output error between a dedicated fixed-target network and the prediction network is used to generate exploration rewards in real time. ; The strategy optimization unit is used for fusion. External environmental rewards We employ an offline training and collaborative exploration strategy based on the MATD3 algorithm with a CTDE architecture. Task execution module: Used to load and deploy the trained agent network, receive real-time local observation information from each agent, output decision-making action instructions, and drive the process. Individual agents perform collaborative exploration tasks in a distributed manner.

Images