Multi-agent cooperative control method and device based on reinforcement learning and medium

Abstract

The invention relates to the technical field of intelligent control, and discloses a multi-unmanned-vehicle cooperative control method and device based on reinforcement learning and a medium, and the method comprises the following steps: building a first space, and generating an escape strategy in the first space; obtaining interaction information, and generating a second space and a reinforcement learning framework based on the interaction information; collecting state information, establishing a dual-network architecture based on the first space and the second space, performing training based on the reinforcement learning framework, and performing gradient optimization on the escape strategy; and executing the escape strategy in a distributed manner. According to the cooperative control method, the stability of the control process is improved.

Description

Technical Field

[0001] This application relates to the field of intelligent control technology, and in particular to a multi-vehicle cooperative control method based on reinforcement learning. Background Technology

[0002] With the rapid development of multi-agent systems and autonomous control technologies, the collaborative combat capabilities of unmanned ground vehicles (UAVs) in complex mission scenarios have attracted increasing attention. Among various mission types, the encirclement and control problem is a typical example with high challenge and practical value. Encirclement and control missions typically require multiple UAVs to coordinate and surround a target, restricting its free movement, and ultimately achieving the mission objective of capturing the target. Traditional encirclement and control methods mostly rely on manual rules, geometric strategies, or centralized scheduling schemes. Although they can achieve certain results in ideal static environments, they suffer from the following main problems when facing real-world scenarios with high uncertainty in target movement, high requirements for cooperation, and high environmental complexity: the collaborative strategy lacks flexibility and is difficult to adapt to target escape behavior and sudden interference; high dependence on multi-vehicle communication and poor coordination lead to instability in the encirclement process; the control strategy lacks self-learning ability and is difficult to generalize to different scenarios; and the model structure is highly dependent, making training and deployment complex and difficult to implement in practice.

[0003] In the process of collaborative control of multiple devices, the stability of collaborative control is an issue that cannot be ignored. Therefore, how to improve the stability of collaborative control is a technical problem that needs to be solved. Summary of the Invention

[0004] This application provides a multi-vehicle cooperative control method based on reinforcement learning, which improves the stability of cooperative control.

[0005] To achieve the above objectives, the main technical solutions adopted in this application include:

[0006] In a first aspect, embodiments of this application provide a multi-unmanned vehicle cooperative control method based on reinforcement learning, including the following steps: establishing a first space and generating an escape strategy within the first space;

[0007] Acquire interaction information and generate a second space and reinforcement learning framework based on the interaction information;

[0008] Collect state information, establish a dual network architecture based on the first and second spaces, train it based on a reinforcement learning framework, and perform gradient optimization on the escape strategy;

[0009] Distributed execution escape strategy.

[0010] This application proposes a multi-vehicle cooperative control method based on reinforcement learning. It establishes a first space to generate basic escape strategies, constructs a second space and reinforcement learning framework based on interaction information, collects state information to establish a dual-network architecture and optimizes the strategy through gradient optimization, and finally executes the strategy in a distributed manner. Its advantages include: hierarchical modeling achieves the fusion of static rules and dynamic interactions, improving adaptability to complex environments; the dual-network architecture combines prior knowledge and real-time data, enhancing decision-making accuracy and strategy iteration capabilities through gradient optimization; distributed execution reduces communication dependencies and single-point failure risks, supporting flexible system expansion; and the reinforcement learning framework, based on data-driven continuous evolution, significantly improves the robustness and generalization performance of multi-vehicle cooperation in scenarios such as emergency obstacle avoidance and target acquisition, significantly improving the stability of cooperative control.

[0011] Optionally, training based on a reinforcement learning framework includes:

[0012] The autonomous vehicle's actions, status, and reward data are stored in batches into the experience playback buffer pool.

[0013] Optimize network weights by sampling collaborative experience data from multiple agents in a buffer pool.

[0014] In the above scheme, the autonomous vehicle's actions, states, and reward data are stored in batches into an experience replay buffer, and collaborative experience data is sampled to optimize network weights. The advantages are as follows: batch storage enables the reuse of historical experience, alleviating the problems of sample sparsity and the curse of dimensionality in multi-agent scenarios, and improving data utilization efficiency; random sampling breaks the temporal correlation of data, reduces the risk of training bias and parameter oscillation, and enhances training stability; explicit use of multi-vehicle collaborative experience optimizes the joint strategy, accelerates collaborative learning between agents, and supports offline training and parallel computing, reducing dependence on real-time communication and improving algorithm training efficiency and engineering deployment flexibility.

[0015] Optionally, gradient optimization of the escape strategy includes:

[0016] Based on the state-action value (Q value) evaluation results provided by the dual-network architecture, the strategy parameters are dynamically adjusted.

[0017] In the above scheme, when performing gradient optimization on the escape strategy, the policy parameters are dynamically adjusted based on the Q-value evaluation results of the dual-network architecture. The advantages are as follows: Q-value quantification of the long-term reward of actions provides a clear gradient direction for policy optimization, avoiding blind trial and error and improving optimization accuracy; the decoupled design of centralized Critic evaluation of global collaborative effect and distributed Actor execution of local policy balances global optimization and communication efficiency; time-aware Q-value evaluation combined with models such as LSTM can capture historical state dependencies and enhance adaptability to dynamic environments; dynamic parameter adjustment supports adaptive learning rate and exploration-exploitation balance, optimizing policy stability in non-steady-state scenarios, while modular design simplifies scenario migration and engineering deployment, improving the algorithm's scalability and practicality.

[0018] Optionally, establishing a first space and generating escape strategies within that first space includes:

[0019] Establish a two-dimensional space for collaborative capture by multiple unmanned vehicles;

[0020] Acquire information on obstacles in a two-dimensional space and the position and angle information of each unmanned vehicle in a collaborative manner;

[0021] Parameterize obstacle information and the position and angle information of each unmanned vehicle;

[0022] Generate target autonomous vehicle escape strategies based on parameterized information.

[0023] In the above scheme, the advantages of establishing a first space and generating a parameterized escape strategy are as follows: Two-dimensional spatial modeling simplifies complex environments, reduces computational dimensionality, and improves algorithm efficiency; parameterization transforms physical information into quantifiable mathematical expressions, supporting accurate collision detection and path planning, and enhancing the interpretability and controllability of the strategy; a unified coordinate system and parameter standards promote multi-vehicle information sharing, reducing communication errors and bandwidth requirements; modular design makes the strategy reusable and easily expandable, adaptable to different obstacle types and task scenarios, while also being compatible with existing sensor hardware, accelerating simulation verification and engineering implementation, and providing basic physical rule support and efficient environmental characterization capabilities for multi-unmanned vehicle collaborative encirclement.

[0024] Optionally, acquiring interaction information and generating a second space based on that interaction information, along with the reinforcement learning framework, includes:

[0025] Acquire the interaction information of the autonomous vehicle and model the interaction information of the autonomous vehicle into a Markov decision model;

[0026] Generate the action space, observation space, and reward function for a multi-unmanned vehicle cooperative encirclement task based on a Markov decision model.

[0027] In the above scheme, the interaction information of autonomous vehicles is modeled as a Markov decision model and an action / observation / reward space is generated. The advantages are as follows: the formal definitions of state, action, and reward are clearly defined through the mathematical framework of MDP, providing a rigorous theoretical basis for policy optimization; the action space is adapted to the continuous motion characteristics of autonomous vehicles, and the physical constraints are accurately modeled by combining second-order kinematic equations; the observation space focuses on key features such as relative distance and angle, filtering redundant information and improving decision-making efficiency; the reward function explicitly encodes the target (such as distance reduction and angle balancing), guiding the agent to learn cooperative strategies; the combination of distributed training and centralized evaluation balances communication costs and global optimization capabilities, enabling the algorithm to efficiently handle the dynamic interaction of multiple vehicles and adapt to the real-time decision-making needs of complex task scenarios.

[0028] Optionally, acquiring the interaction information of the autonomous vehicle and modeling this interaction information into a Markov decision model includes:

[0029] The task is formalized as a tuple, which includes the global state space, joint observation space, joint action set, state transition function, reward function set, and discount factor;

[0030] Define the continuous action space of the agent and describe the agent's motion state based on the second-order kinematic equations.

[0031] In the above scheme, the interaction information of autonomous vehicles is modeled as a Markov decision model. The state is described by formalizing the task definition with tuples and second-order kinematic equations. The advantages are as follows: Tuple modeling clearly defines the core elements such as global state, action, and reward, and builds a rigorous mathematical framework for the algorithm, which is convenient for theoretical analysis and policy optimization; the continuous action space combined with second-order kinematic equations accurately describes the dynamic characteristics of autonomous vehicles such as speed and acceleration, avoids discretization errors, and makes the policy output more in line with actual physical constraints; joint observation and action set explicitly capture the interaction relationship of multiple vehicles, support the combination of distributed execution and centralized training, and improve the collaborative efficiency; the state transition function and discount factor consider environmental changes and long-term rewards, enhance the adaptability of the algorithm to dynamic scenarios, and thus help improve the stability of collaborative control.

[0032] Optionally, the collected status information includes:

[0033] Acquire status observation information of the unmanned vehicle used for the encirclement and capture;

[0034] Acquire observation information about the target unmanned vehicle's own status;

[0035] The status information is obtained by stitching together the status observation information of the capture unmanned vehicle and the status observation information of the target unmanned vehicle.

[0036] In the above scheme, when collecting state information, the state observation information of the encirclement vehicle and the target vehicle is obtained and stitched together. The advantages are as follows: by integrating the states of both parties, the algorithm can construct a complete task situation map, effectively eliminate information asymmetry, and accurately predict the target's action intentions; the stitched state information supports explicit modeling of the adversarial relationship between encirclement and escape, which is convenient for optimizing multi-agent cooperative strategies and dynamically adjusting the encirclement layout; redundant state data enhances the system's fault tolerance, and even if some sensors fail, the state can still be inferred based on other observations; at the same time, this method simplifies the design of the reward function, can more intuitively evaluate the encirclement progress, accelerate the convergence of strategy training, and is compatible with multi-source heterogeneous sensor data, improving the system's adaptability and robustness in complex environments.

[0037] Optional multi-vehicle cooperative control methods also include:

[0038] The input local state is processed by extracting features through a fully connected layer, and then the temporal dependency is modeled through an LSTM module. The attention mechanism is used to filter key features, and finally the agent's action is output.

[0039] Input the joint state and joint action, extract features through a fully connected layer and LSTM module, capture the interaction relationship between agents using a multi-head attention mechanism, and output the joint Q value.

[0040] In the above scheme, the multi-autonomous vehicle cooperative control method processes local states through an LSTM module and an attention mechanism. Combined with multi-head attention to calculate the joint Q-value, it can accurately capture the temporal dependence characteristics of autonomous vehicle motion, adaptively focus on key decision information, and improve the response speed and accuracy to dynamic environmental changes (such as sudden target turning or the appearance of obstacles). The multi-head attention mechanism deeply mines the interaction patterns between agents, optimizes multi-vehicle cooperation strategies, and enhances the cooperative efficiency in the encirclement task. At the same time, the architecture supports end-to-end training, reduces feature engineering dependence, and reduces system development and maintenance costs through modular design, significantly improving the decision accuracy, robustness, and engineering practicality of the algorithm in complex scenarios.

[0041] Secondly, embodiments of this application propose an electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor; when the processor runs the computer program, it implements the multi-unmanned vehicle cooperative control method as described in any of the above embodiments.

[0042] Thirdly, embodiments of this application propose a storage medium containing a computer program that, when run by a processor, implements a multi-unmanned vehicle cooperative control method as described in any of the above embodiments. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0044] Figure 1 Here are some block diagrams illustrating the principle of the multi-unmanned vehicle cooperative encirclement method in some embodiments of this application;

[0045] Figure 2 This is a schematic diagram of the method steps in some embodiments of this application;

[0046] Figure 3 This is a schematic diagram illustrating a successful capture in some embodiments of this application;

[0047] Figure 4 This is a schematic diagram of one stage of the 3V1 cooperative capture trajectory process in some embodiments of this application;

[0048] Figure 5 This is a schematic diagram of another stage of the 3V1 cooperative capture trajectory process in some embodiments of this application;

[0049] Figure 6 This is a schematic diagram of another stage of the 3V1 cooperative capture trajectory process in some embodiments of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used in the description of this application is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms "comprising" and "having," and any variations thereof, in the description, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the description, claims, or accompanying drawings of this application are used to distinguish different objects, not to describe a specific order or hierarchy.

[0052] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0053] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," and "attachment" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0054] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, in this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0055] In this application, "multiple" refers to two or more (including two), and similarly, "multiple groups" refers to two or more (including two), and "multiple pieces" refers to two or more (including two).

[0056] With the rapid development of multi-agent systems and autonomous control technologies, the collaborative combat capabilities of unmanned ground vehicles (UAVs) in complex mission scenarios have attracted increasing attention. Among various mission types, the encirclement and control problem is a typical example with high challenge and practical value. Encirclement missions typically require multiple UAVs to cooperate in surrounding a target, restricting its free movement, and ultimately achieving the mission objective of capturing the target.

[0057] Traditional encirclement and control methods mostly rely on manual rules, geometric strategies, or centralized scheduling schemes. While they can achieve certain results in ideal static environments, they suffer from the following main problems when facing real-world scenarios with high uncertainty in target movement, high requirements for cooperation, and high environmental complexity: the cooperative strategies lack flexibility and are difficult to adapt to target escape behavior and sudden interference; high dependence on multi-workshop communication and poor coordination lead to instability in the encirclement process; the control strategies lack self-learning capabilities and are difficult to generalize to different scenarios; and the model structure is highly dependent, making training and deployment complex and difficult to implement in practice.

[0058] In recent years, multi-agent reinforcement learning (MADDPG) has demonstrated strong autonomous learning capabilities and distributed control advantages in multi-vehicle cooperative control, becoming a powerful tool for solving the trapping control problem. However, MADDPG still faces the following bottlenecks in practical applications: weak cooperation among agents and dispersed policy learning lead to unstable trapping effects; the network structure cannot effectively utilize historical observations, and policy decisions lack temporal memory; the high dimensionality of the state space results in low training efficiency and is prone to learning non-convergence.

[0059] In view of this, this application proposes a multi-agent cooperative control method based on reinforcement learning. Specifically, it discloses a multi-agent reinforcement learning method that introduces long short-term memory networks and attention mechanisms for encirclement and capture mission scenarios. This method can be widely applied to the cooperative control needs of ground unmanned vehicles in tasks such as intelligent tracking, target encirclement and capture, and security patrol, and the cooperative control is more stable.

[0060] The following is in conjunction with the appendix Figure 1 - Appendix Figure 6 Specifically, this application discloses a multi-unmanned vehicle cooperative control method based on reinforcement learning, which includes the following steps: S1, establishing a first space and generating an escape strategy within the first space;

[0061] S2. Obtain interaction information and generate a second space and reinforcement learning framework based on the interaction information;

[0062] S3. Collect state information, establish a dual network architecture based on the first and second spaces, train it based on the reinforcement learning framework, and perform gradient optimization on the escape strategy.

[0063] S4, Distributed execution escape strategy.

[0064] This application proposes a multi-vehicle cooperative control method based on reinforcement learning. By establishing a first space to generate escape strategies, the basic obstacle avoidance behaviors (such as steering and deceleration) of unmanned vehicles can be designed in advance for common obstacles and static environmental constraints (such as road boundaries) to ensure safe passage in simple environments. Based on real-time interactive information (such as the positions of other unmanned vehicles, dynamic obstacle trajectories, traffic signal changes, etc.), a second space is generated, enabling the system to dynamically adjust strategies to adapt to complex dynamic environments (such as multi-vehicle games at intersections and sudden pedestrian crossings), thereby improving the environmental robustness of multi-vehicle cooperation.

[0065] By using a reinforcement learning framework to optimize escape strategies through gradients, autonomous vehicles can learn optimal behavior patterns autonomously through trial and error, leveraging a large amount of simulation or real-world driving data. For example, in multi-vehicle intersection scenarios, reward functions can guide the optimization of yielding order and speed adjustment strategies for each vehicle (e.g., reducing collision risk and improving traffic efficiency), avoiding the limitations of traditional rule-based algorithms.

[0066] By combining the first space (static strategy) and the second space (dynamic interaction) to establish a dual-network architecture, it is possible to simultaneously process prior knowledge (such as road rules) and real-time data (such as sensor perception information) to achieve coordinated control of "global planning + local response".

[0067] For example, in a highway scenario, the first spatial strategy ensures that the vehicle stays in its lane, while the second spatial strategy dynamically adjusts the vehicle's speed based on the lane-changing intentions of adjacent vehicles to avoid collisions while maintaining smooth traffic flow.

[0068] Reinforcement learning frameworks can continuously train dual networks by collecting new state information (such as driving data under different weather and road conditions), enabling the algorithm to adapt to diverse scenarios (such as wet and slippery roads in rainy weather and low light environments at night), thus avoiding the problem of insufficient generalization ability of traditional manually tuned algorithms.

[0069] For example, by simulating extreme scenarios (such as a sudden tire blowout) through data augmentation, algorithms can learn more robust emergency avoidance strategies, improving the survivability of autonomous vehicles in rare but high-risk scenarios.

[0070] In other embodiments, training based on a reinforcement learning framework includes:

[0071] The autonomous vehicle's actions, status, and reward data are stored in batches into the experience playback buffer pool.

[0072] Optimize network weights by sampling collaborative experience data from multiple agents in a buffer pool.

[0073] In the above scheme, the actions, states, and reward data of multiple vehicles are uniformly stored in a buffer pool to avoid the "use and discard" of single interaction data and realize the reuse of historical experience. Especially in multi-agent scenarios, the collaborative experience generated by a single round of training (such as obstacle avoidance interaction and path coordination between vehicles) can be sampled multiple times, which significantly reduces the dependence on real-time interaction data and improves training efficiency.

[0074] The state space and action space of multi-vehicle cooperative scenarios have extremely high dimensions (such as the positions, speeds, and heading angles of multiple vehicles). Direct training can easily lead to sparse samples (i.e., insufficient experience in effective cooperation). Experience replay fills the experience gaps in sparse areas by repeatedly sampling historical data, enabling the algorithm to learn more general cooperative strategies from limited data.

[0075] In reinforcement learning, the real-time interaction data between the agent and the environment are usually highly correlated (e.g., similar state changes between consecutive actions). Direct training can easily lead to network parameter oscillations or getting stuck in local optima. Experience replay breaks the temporal correlation by randomly sampling historical data (i.e., "shuffling" the data order), making the training data closer to independent and identically distributed (i.i.d.), thereby improving the stability of network convergence.

[0076] Reward signals for multi-agent collaboration (such as collision penalties and path efficiency rewards) may be noisy or delayed due to the complexity of the scenario.

[0077] By storing reward data from multiple rounds, the buffer pool can reduce the impact of single-round reward fluctuations during sampling through batch statistics such as averaging, making the gradient update direction more accurate.

[0078] In other embodiments, gradient optimization of the escape strategy includes:

[0079] Based on the state-action value (Q value) evaluation results provided by the dual-network architecture, the strategy parameters are dynamically adjusted.

[0080] In the above scheme, the Q-value (state-action value function) directly reflects the expected long-term reward of taking a certain action in a specific state. By adjusting the policy parameters based on the Q-value evaluation results, the algorithm can clearly know "which actions are better," thereby efficiently optimizing the policy along the gradient direction and avoiding blind trial and error. For example, in encirclement tasks, the Q-value can quantify the advantages and disadvantages of different encirclement angles, guiding the autonomous vehicle to adjust towards directions that yield higher long-term rewards (such as forming an encirclement faster).

[0081] In the early stages of training, the algorithm can identify the action space that has not been fully explored by regions with low Q values, thus encouraging "exploration". In the later stages of training, regions with high Q values ​​have been verified as effective strategies, and the algorithm can favor "utilizing" these mature actions.

[0082] This dynamic equilibrium mechanism makes the strategy optimization process more directional, avoiding getting trapped in local optima or overexploration.

[0083] In the dual-network architecture, the Critic network (centralized) calculates the joint Q-value through global state to evaluate the collaborative effect of multiple agents; while the Actor network (distributed) only needs to generate actions based on its own local observations.

[0084] This design allows agents to remain distributed during execution (reducing communication dependencies), but learns the globally optimal cooperative strategy through centralized Q-score evaluation during training, balancing efficiency and consistency.

[0085] For example, in multi-vehicle obstacle avoidance scenarios, the Critic network can comprehensively evaluate the trajectory safety of all vehicles, guide each vehicle to adjust its speed and direction, and avoid the risk of global collision caused by local optimal decisions.

[0086] Dual-network architectures typically incorporate time-series models such as LSTM, enabling Q-value evaluation to consider historical state sequences (e.g., the movement trends of target vehicles). Based on this time-aware Q-value adjustment strategy, it can better cope with dynamically changing environments (e.g., a target vehicle suddenly accelerating to escape). For example, in pursuit missions, policy parameters can predict the target vehicle's next move based on its historical trajectory, allowing for advance adjustments to the encirclement formation.

[0087] In other embodiments, establishing a first space and generating an escape strategy within the first space includes:

[0088] Establish a two-dimensional space for collaborative capture by multiple unmanned vehicles;

[0089] Acquire information on obstacles in a two-dimensional space and the position and angle information of each unmanned vehicle in a collaborative manner;

[0090] Parameterize obstacle information and the position and angle information of each unmanned vehicle;

[0091] Generate target autonomous vehicle escape strategies based on parameterized information.

[0092] In the above scheme, by establishing a two-dimensional space for multi-vehicle collaborative capture, the three-dimensional physical world is simplified into a planar coordinate system, reducing the dimension of the state space (such as ignoring vehicle height), significantly reducing computational complexity, and transforming information such as obstacle position and vehicle coordinates / angles into mathematical parameters (such as coordinate points and geometric shapes), enabling the algorithm to perform accurate collision detection and path planning through numerical calculations.

[0093] For example, parameterizing a circular obstacle as "(x,y,r)" (center point coordinates and radius) can quickly determine whether a vehicle has entered a dangerous area.

[0094] The parameterized information (such as vehicle spacing and relative angle) can be directly mapped to physical rules (such as collision avoidance distance threshold and steering angle limit), so that the escape strategy conforms to the actual kinematic constraints.

[0095] All autonomous vehicles describe the environment based on the same two-dimensional spatial coordinate system and parameterization method, eliminating coordinate transformation errors and improving collaboration accuracy.

[0096] For example, during convoy driving, each vehicle can quickly build global situational awareness by sharing parameterized information (such as "vehicle A is at position (10,20) with a heading angle of 30°").

[0097] Parameterized information (such as coordinate points and angle values) is lighter than raw sensor data (such as images and point clouds), reducing the amount of communication between vehicles.

[0098] In other embodiments, acquiring interaction information and generating a second space and reinforcement learning framework based on the interaction information includes:

[0099] Acquire the interaction information of the autonomous vehicle and model the interaction information of the autonomous vehicle into a Markov decision model;

[0100] Generate the action space, observation space, and reward function for a multi-unmanned vehicle cooperative encirclement task based on a Markov decision model.

[0101] In the above scheme, the interaction information is modeled as a Markov Decision Process (MDP). It assumes that the current state fully contains historical information, allowing the algorithm to predict the future by focusing only on the current state, significantly reducing modeling complexity. For example, in vehicle obstacle avoidance scenarios, only the relative position and speed at the current moment need to be considered, without needing to trace the complete historical trajectory.

[0102] The tuple representation of MDP (state space, action space, transition probabilities, reward function) provides a rigorous mathematical foundation for the algorithm, and convergence and optimality can be proven using theoretical tools such as dynamic programming and policy gradient.

[0103] By clearly defining the action space (e.g., linear velocity, angular velocity), the continuous physics control problem is transformed into a discrete / continuous decision-making problem, making it suitable for reinforcement learning algorithms. When designing the observation space (e.g., relative distance, angle, velocity difference), irrelevant data (such as sensor noise) can be filtered out, highlighting task-relevant features. For example, in a capture mission, only the geometric relationship between the target vehicle and the encirclement needs to be considered, without needing to perceive global environmental details.

[0104] By carefully designing reward functions (such as distance rewards, angle rewards, and collision penalties), task objectives can be explicitly encoded, guiding agents to learn cooperative strategies. For example, in multi-vehicle platooning scenarios, simultaneously optimizing "maintaining spacing" (distance reward) and "aligning headings" (angle reward) can achieve efficient collaboration.

[0105] The MDP framework naturally supports multi-agent interaction modeling. By defining joint state and joint action spaces, it can explicitly capture policy dependencies between vehicles. For example, in an intersection scenario, vehicle A's "accelerate through" action needs to consider vehicle B's possible reaction (slow down to give way). The joint state space can simultaneously represent the states of both vehicles.

[0106] In other embodiments, acquiring the autonomous vehicle's interaction information and modeling this interaction information into a Markov decision model includes:

[0107] The task is formalized as a tuple, which includes the global state space, joint observation space, joint action set, state transition function, reward function set, and discount factor;

[0108] Define the continuous action space of the agent and describe the agent's motion state based on the second-order kinematic equations.

[0109] In the above scheme, the control inputs of the autonomous vehicle (such as throttle and steering) are essentially continuous values. By defining a continuous motion space, physical control quantities can be directly mapped, avoiding the accuracy loss and motion space explosion problems caused by discretization. For example, representing the steering angle as a continuous interval of [-30°, 30°] enables the vehicle to perform more refined trajectory planning.

[0110] Compared to first-order kinematic models (which only consider position and velocity), second-order equations introduce acceleration / angular acceleration, which more accurately reflects vehicle inertia and dynamic constraints. For example, in high-speed driving scenarios, vehicles cannot change direction instantaneously, and second-order models can simulate this physical characteristic more realistically through acceleration constraints.

[0111] Continuous action spaces allow policy gradient methods (such as DDPG and TD3) to directly optimize action parameters, avoiding the inefficient search in continuous space by value function-based methods. For example, the optimal acceleration and steering angle can be directly output through the Actor network, rather than selecting approximate actions from a discrete action set.

[0112] State representation based on kinematic equations makes policy outputs more consistent with physical laws and reduces control oscillations. For example, in platooning, vehicles can predict the impact of their actions on subsequent states based on a second-order model, avoiding frequent acceleration and deceleration; trajectory planning naturally satisfies the vehicle's minimum turning radius constraint.

[0113] Furthermore, although the joint Q-value is calculated using the global state during training, each vehicle only needs local observations (such as the relative positions of adjacent vehicles) during the execution phase, reducing communication requirements and the risk of single-point failures. The parameterized representation of the continuous action space (such as the output of a neural network) enables the algorithm to quickly adapt to different types of unmanned vehicles (such as cars and trucks) or task requirements (such as encirclement and transportation).

[0114] In other embodiments, collecting status information includes:

[0115] Acquire status observation information of the unmanned vehicle used for the encirclement and capture;

[0116] Acquire observation information about the target unmanned vehicle's own status;

[0117] The status information is obtained by stitching together the status observation information of the capture unmanned vehicle and the status observation information of the target unmanned vehicle.

[0118] In the above scheme, by simultaneously acquiring the states (such as position, speed, and acceleration) of both the encircling party and the target party, the algorithm can construct a complete battlefield situation map, avoiding decision-making biases caused by local observations. For example, relying solely on the observations of the encircling vehicle itself may misjudge the target's escape intention, while combining the state of the target vehicle can more accurately predict its next move.

[0119] Furthermore, each vehicle only needs to upload its own state to the central controller (or distributed shared memory), and global observations are generated by concatenating these states, reducing the frequency of real-time communication. For example, vehicle A does not need to know the specific actions of vehicle B; it only needs to share its own state, and the algorithm implicitly derives the cooperative strategy.

[0120] Simultaneously, the observation information of both the encircling vehicle and the target vehicle is retained. Even if some sensors fail (such as the target vehicle losing its GPS signal), the status can still be inferred from the relative observations of the encircling vehicle, improving system reliability. The stitched status can flexibly include time-varying factors (such as changes in obstacle positions and terrain features), enabling the algorithm to quickly adapt to environmental changes. For example, in urban road scenarios, if a temporary obstacle suddenly appears, the algorithm can replan the encirclement path based on the status of all vehicles; when the target vehicle adopts an unconventional escape strategy, the encircling vehicle can adjust its response based on the real-time status.

[0121] Complete state information allows the reward function to more directly assess task progress (such as the tightness of the encirclement and the probability of target escape), accelerating policy convergence. For example, it can directly calculate the "encirclement completion rate" based on the spliced ​​state, avoiding the use of indirect indicators (such as the average distance from the target).

[0122] Furthermore, different vehicles may be equipped with different types of sensors (such as LiDAR and cameras), and state stitching allows for the flexible integration of multi-source heterogeneous data. For example, vehicle A uses LiDAR to obtain high-precision location information, while vehicle B uses visual SLAM to estimate the target's pose, and both are represented uniformly through state stitching; the target vehicle's own state can be directly obtained through onboard sensors or indirectly inferred from the observations of the vehicle.

[0123] In other embodiments, the multi-unmanned vehicle cooperative control method further includes:

[0124] The input local state is processed by extracting features through a fully connected layer, and then the temporal dependency is modeled through an LSTM module. The attention mechanism is used to filter key features, and finally the agent's action is output.

[0125] Input the joint state and joint action, extract features through a fully connected layer and LSTM module, capture the interaction relationship between agents using a multi-head attention mechanism, and output the joint Q value.

[0126] In the above scheme, the motion decisions of autonomous vehicles have obvious temporal correlations (such as the current steering angle affecting future position). LSTM effectively models long-term sequence dependencies through a gating mechanism, avoiding the gradient vanishing problem of traditional RNNs. For example, in high-speed obstacle avoidance scenarios, LSTM can memorize the vehicle's historical trajectory, predict the future position of obstacles, and generate smoother avoidance actions.

[0127] Attention mechanisms can adaptively filter temporal features, highlighting the information segments most important for the current decision. For example, in intersection scenarios, the model can automatically focus on recent traffic light changes and the acceleration and deceleration behavior of adjacent vehicles, ignoring irrelevant background information; in target escape scenarios, it strengthens the response weights to abnormal actions of the target vehicle (such as sudden braking and steering).

[0128] Multi-head attention mechanisms allow models to simultaneously focus on multiple interaction dimensions in a joint state, capturing complex vehicle-to-vehicle cooperative patterns. For example, in platooning, different attention heads can focus on multiple cooperative objectives such as "maintaining distance," "aligning headings," and "synchronizing speeds." In encirclement missions, the model can learn a division of labor strategy where "the flanking vehicle needs to focus on the target's escape direction, and the blocking vehicle needs to focus on the positions of other encirclement vehicles."

[0129] After extracting joint features through fully connected layers and LSTM, multi-head attention can automatically discover implicit relationships between agents without the need for manual feature design. For example, it can automatically identify cooperative patterns such as "when vehicle A decelerates, vehicle B needs to accelerate to fill the gap"; and in dynamic obstacle scenarios, it can infer the obstacle avoidance intentions of other vehicles and adjust its own strategy accordingly.

[0130] The feature filtering function of attention mechanisms can suppress the influence of sensor noise and abnormal observations. For example, when a vehicle's GPS signal is briefly lost, the model can reduce the impact of the vehicle's state through attention weights and continue to make decisions based on other sensor data; in dense traffic scenes, it can automatically ignore interference information from distant, irrelevant vehicles.

[0131] LSTM's memory capability enables the model to quickly adapt to dynamic environmental changes. For example, when a new obstacle suddenly appears, the model can predict its movement trend based on historical observations and generate a more reasonable obstacle avoidance path; when traffic rules change (such as temporary traffic restrictions), it can capture the changing patterns through temporal features and adjust its strategy accordingly.

[0132] Secondly, embodiments of this application propose an electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor; when the processor runs the computer program, it implements the multi-unmanned vehicle cooperative control method as described in any of the above embodiments.

[0133] Thirdly, embodiments of this application propose a storage medium containing a computer program that, when run by a processor, implements a multi-unmanned vehicle cooperative control method as described in any of the above embodiments.

[0134] In one specific embodiment, the multi-unmanned vehicle cooperative control method based on reinforcement learning proposed in this application includes the following steps:

[0135] Step 1) Establish a two-dimensional space for multi-unmanned vehicle collaborative capture; and parameterize the obstacle information in the environment and the position and angle information of each unmanned vehicle to give the target unmanned vehicle escape strategy;

[0136] Step 2) Model the interaction information of the unmanned vehicles into a Markov decision model, and design the action space, observation space and reward function for the multi-unmanned vehicle cooperative encirclement task;

[0137] Step 3) After processing the state observation information of the captured unmanned vehicle, it is spliced ​​with the state observation information of the target unmanned vehicle and input into the algorithm module;

[0138] Step 4) Design a dual-network structure based on LSTM and attention mechanisms, using the Actor-Critic framework. The autonomous vehicle's actions, states, and rewards are stored in batches in an experience replay buffer. During training, the centralized Critic evaluation module optimizes network weights by sampling collaborative experience data from the multi-agent pool. Simultaneously, the policy generation module (Actor) dynamically adjusts policy parameters based on the state-action value (Q-value) evaluation results provided by the Critic to achieve gradient optimization.

[0139] Step 5) Decentralize decision-making and adopt distributed execution. During the execution phase, the strategy is obtained using only local observations.

[0140] This method makes the following key improvements based on MADDPG: First, to enhance the temporal consistency and decision robustness of strategies among multiple vehicles, a Long Short-Term Memory (LSTM) network is introduced into the policy network, endowing the agent with the ability to make behavioral judgments based on historical observation sequences, thereby enhancing its adaptability to dynamic environmental changes. Second, to overcome the problems of increased state space dimensionality and reduced training efficiency caused by the introduction of LSTM, an attention mechanism is further combined to perform weight calculation and feature selection on the input observation states, effectively compressing the state space dimensionality and focusing on key information highly relevant to the capture task, thereby improving the network's learning efficiency and decision accuracy. This method improves the collaborative ability, control stability, and capture success rate among multiple unmanned vehicles in complex dynamic environments, and has good engineering deployability and application prospects.

[0141] Furthermore, to improve the temporal consistency and decision robustness of strategies among multiple vehicles, this method introduces a Long Short-Term Memory (LSTM) network into the policy network, endowing the agent with the ability to make behavioral judgments based on historical observation sequences, thereby enhancing its adaptability to dynamic environmental changes. To overcome the problems of state space dimensional expansion and reduced training efficiency caused by the introduction of LSTM, an attention mechanism is further combined to perform weight calculation and feature selection on the input observation states, effectively compressing the state space dimension and focusing on key information highly relevant to the encirclement task, thereby improving the network's learning efficiency and decision accuracy.

[0142] Furthermore, in step 1), the specific process is as follows:

[0143] Step 1-1) Establish the Cartesian coordinate system, set the boundary of the two-dimensional space and the position coordinate information of each unmanned vehicle and obstacle.

[0144] Step 1-2) Define the conditions for successful encirclement based on angle and distance:

[0145]

[0146] in:

[0147] d min --Minimum collision distance;

[0148] --Various unmanned vehicles were used for the manhunt. i Distance to the target driverless vehicle T;

[0149] d success --The unmanned vehicle can precisely control and capture the target at the maximum distance;

[0150] θ iTj -- Hunting down the unmanned vehicle P i The driverless car P in the clockwise direction j And the angle formed by the target unmanned vehicle T.

[0151] Steps 1-3) Define the escape strategy for the target autonomous vehicle:

[0152] By calculating the angle between adjacent unmanned vehicles used for encirclement and suppression, the direction with the largest included angle is selected for movement.

[0153] Assume P i P j P n These represent the unmanned vehicles numbered i, j, and n, respectively, used for the pursuit and capture. The direction of movement is set from the location of the target vehicle to the pursuit vehicle P. i With P j The midpoint of the line connecting x r ,y rThe escape angle is defined as θ. E =arctan((y r -y T ) / x r -x T The escape strategy is shown in equation (2):

[0154]

[0155] Where ν0 and ω0 represent the velocity and angular velocity at the current moment; d Pi Indicates the encirclement and capture of driverless car P i The distance between the target unmanned vehicles; min(d) Pi ) represents the shortest distance between the three pursuit drones and the target drone; d success This indicates the maximum successful capture radius. If there is no need to escape, the target vehicle will continue moving at its original speed and angular velocity. When the target unmanned vehicle detects a distance of less than or equal to 0.5m from the capturing unmanned vehicle, the target vehicle will escape based on information such as its distance to the destination and its distance to the capturing vehicle.

[0156] Furthermore, in step 2), the specific process is as follows:

[0157] Step 2-1) Establish the trajectory tracking error model for the moving stage. The process includes:

[0158] In the multi-vehicle cooperative encirclement task, the task is modeled as a Markov game, formally defined as a tuple (S,O,A,T,R,γ), where:

[0159] S -- The global state space of the system;

[0160] O = {O1, O2, ..., O} n -- Joint observation space for each capture unit;

[0161] A = A1 × A2 × ... × A n --A set of combined actions from various capture units;

[0162] T -- State transition function, used to describe the probability of transitioning to the next state after taking a joint action A in state S;

[0163] R -- The set of reward functions for each capture unit;

[0164] γ -- Discount factor, used to control the degree of time decay of rewards.

[0165] Step 2-2) Establish the second-order kinematic equations of the unmanned vehicle and define the continuous action space:

[0166]

[0167] In this model, equations (3) and (4) represent the continuous-time kinematics model of the differential-driven autonomous vehicle, where (x, y) is the position of the autonomous vehicle in the global coordinate system, θ is the heading angle of the autonomous vehicle, v is the instantaneous linear velocity of the autonomous vehicle, and ω is the instantaneous angular velocity of the autonomous vehicle. The state variables are denoted as X = [x, y, θ]. Τ The velocity vector is u = [v, ω]. Formula (5) is its corresponding discrete-time motion model, where Δt is the control time step, (x t ,y t ,θ t (x) represents the state at the current time t. t+△t ,y t+△t ,θ t+△t ) represents the position and orientation after a time interval Δt.

[0168] Steps 2-3) For the third unmanned vehicle used for the encirclement and capture, the observation space is defined as follows:

[0169]

[0170] in Indicates the distance between the pursuit vehicle and the target vehicle; d 12 d 13 d 23 θ represents the distance between the three unmanned vehicles used for the maneuver. iTj (i = 1, 2, 3; j = 1, 2, 3) represents the angle between adjacent trapping machines and the target robot; v T1 v T2 v T3 The values ​​θ1, θ2, θ3, and θ represent the speed difference between the pursuing drone and the target drone. T This indicates the heading angle between the pursuit drone and the target drone.

[0171] Steps 2-4) design the following reward function for the multi-unmanned vehicle cooperative encirclement task:

[0172] In the design of the distance reward function, when the real-time distance between the capture vehicle and the target vehicle is less than the preset capture success threshold (defined as 0.3 meters in the simulation), the system will trigger a positive reward mechanism. The reward function is shown in formula (7):

[0173]

[0174] Since the number of unmanned vehicles to be surrounded is 3, the expected range of the surrounding angle is [110°, 120°]. When the actual surrounding angle θ... iTj When the angle falls within this range, the system provides a positive reward; if the angle exceeds 130°, a penalty is imposed, and when the angle is greater than 110°, a shape function is introduced. This function quantifies the dispersion among the three encirclement angles (the smaller the value, the more balanced the angle distribution), guiding the unmanned vehicles to collaboratively form a closed encirclement circle. Once the angles reach the desired range, the weight coefficient of this function dynamically increases to reinforce the angle consistency optimization objective. The complete angle reward function is shown in formula (8):

[0175]

[0176] When the unmanned vehicles collaboratively construct a closed encirclement, the system will provide additional rewards. This aims to enhance the collaborative efficiency of multiple vehicles, improve the target capture success rate, and optimize the training process through dynamic feedback. The reward function for forming the encirclement is shown in formula (9):

[0177]

[0178] The target unmanned vehicle T is located within a triangle formed by three surrounding unmanned vehicles P1, P2, and P3.

[0179] The collision detection threshold is set at 0.232 meters, and the distance between the unmanned vehicles in the encirclement is d. ij (i≠j), if the hunting drone collides with the target drone, a penalty mechanism will be triggered and the current training process will be forcibly terminated. The collision penalty function is shown in formula (10):

[0180]

[0181] This paper sets a penalty and stops training when the distance between the pursuit vehicle and the target vehicle is greater than 2m. The distance penalty function is shown in formula (11):

[0182]

[0183] The total reward function is shown in formula (12):

[0184]

[0185] To optimize the autonomous learning efficiency of autonomous vehicles and avoid the risks of infinite loops or task stagnation, a negative incentive factor (-0.1) is introduced into the reward mechanism. This design aims to enhance the algorithm's adaptability to complex scenarios by dynamically balancing exploration and convergence behaviors, while constraining the continuous execution of inefficient strategies.

[0186] Furthermore, in step 3), the specific process is as follows:

[0187] Step 3-1), obtain the observation information for each vehicle. i n :

[0188] Among them O i Includes the distance between the pursuit vehicle and the target vehicle. The distance between the three unmanned vehicles involved in the maneuver is d ij (i,j∈{1,2,3}); the speed difference v between the pursuit vehicle and the target vehicle. Ti The heading angle θ between the pursuit vehicle and the target vehicle i and θ T The angle θ between the adjacent trapping machine and the target robot iTj (i,j∈{1,2,3}).

[0189] Step 3-2) involves standardizing the features of various attributes (such as angle and distance) in the observation information to eliminate dimensional differences and map observations with different physical meanings to a unified numerical range, generating N sets of standardized 8-dimensional feature vectors. The observations of each autonomous vehicle are aggregated into a global state S of the entire environment. Each global state S consists of the observations of three autonomous vehicles, thus there are N sets of global states S.

[0190] Furthermore, in step 4), the specific process is as follows:

[0191] Step 4-1), LSTM and Attention Mechanism Structure Design

[0192] Actor network structure: The policy network input for each agent is the local state s. i After normalization and extraction of state features through a fully connected layer (128 dimensions), the data enters an LSTM module (128 units) to model temporal dependencies. Then, the attention mechanism module calculates the weights of key time steps, followed by ReLU activation and a fully connected layer (64 dimensions) to output the action.

[0193] The Critic network structure uses joint state and joint action as inputs. Features are extracted and concatenated through fully connected layers (128-dimensional state channel and 300-dimensional action channel) and then fed into an LSTM (128 units) to extract temporal dependency features. A multi-head attention mechanism is then used to extract interaction weights between agents, and finally, the joint Q-value is output through a fully connected layer.

[0194] Step 4-2), Initialize network parameters

[0195] Initialize a value network of N agents and policy network Parameter collection:

[0196]

[0197] And copy to obtain the target network parameter set:

[0198] θ Q′=θ Q ,θ μ′ =θ μ (14)

[0199] Step 4-3) Construct the experience replay pool

[0200] Construct an experience replay pool D to store the tuples sampled at each time step t:

[0201] D={(s t ,a t ,r t ,s' t (15)

[0202] in

[0203] --The state of all agents at time t;

[0204] --All agents execute actions at time t;

[0205] --The reward each agent receives at time t;

[0206] s' t --The joint state at the next moment.

[0207] Step 4-4) Calculate the target Q value

[0208] A batch of samples is sampled from the experience pool, and the target Q value is calculated:

[0209]

[0210] in This refers to the action output by the target policy network in the next time step.

[0211] Steps 4-5) Update Critic network parameters

[0212] Minimize the following loss function to update the Critic network for each agent:

[0213]

[0214] Where S is the total number of sequence sets drawn from the experience pool; Indicates from state s t Select action a t Expected reward that can be obtained later

[0215] Steps 4-6) Update Actor network parameters

[0216] The policy network parameters are updated using the policy gradient, which is calculated as follows:

[0217]

[0218] in:

[0219] --Policy gradient, used to update the parameters of the i-th Actor;

[0220] --Partial derivatives of the policy network with respect to the parameters;

[0221] --The partial derivative of the current Critic network with respect to actions.

[0222] Steps 4-7) Target network soft update

[0223] Update the target network parameters using a soft update method:

[0224]

[0225] Where τ∈[0,1] is the soft update factor.

[0226] Furthermore, in step 5), the specific process is as follows:

[0227] Decision-making is decentralized and executed in a distributed manner. During the execution phase, the strategy is obtained using only local observations.

[0228] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0229] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0230] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

[0231] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A multi-unmanned vehicle cooperative control method based on reinforcement learning, characterized in that, Includes the following steps: Establish a first space and generate an escape strategy within the first space; Acquire interaction information and generate a second space and reinforcement learning framework based on the interaction information; Collect state information, establish a dual-network architecture based on the first space and the second space, train the network based on the reinforcement learning framework, and perform gradient optimization on the escape strategy; The escape strategy is executed in a distributed manner.

2. The multi-unmanned vehicle cooperative control method according to claim 1, characterized in that, The training based on the reinforcement learning framework includes: The autonomous vehicle's actions, status, and reward data are stored in batches into the experience playback buffer pool. Optimize network weights by sampling collaborative experience data from multiple agents in a buffer pool.

3. The multi-unmanned vehicle cooperative control method according to claim 2, characterized in that, The gradient optimization of the escape strategy includes: Based on the state-action value (Q value) evaluation results provided by the dual-network architecture, the strategy parameters are dynamically adjusted.

4. The multi-unmanned vehicle cooperative control method according to claim 1, characterized in that, The establishment of the first space and the generation of the escape strategy within the first space include: Establish a two-dimensional space for collaborative capture by multiple unmanned vehicles; Acquire information on obstacles in a two-dimensional space and the position and angle information of each unmanned vehicle in the multi-unmanned vehicle collaborative capture operation; The obstacle information and the position and angle information of each unmanned vehicle are parameterized; Based on the parameterized information, an escape strategy for the target unmanned vehicle is generated.

5. The multi-unmanned vehicle cooperative control method according to claim 1, characterized in that, The step of acquiring interaction information and generating a second space and reinforcement learning framework based on the interaction information includes: Acquire the interaction information of the autonomous vehicle and model the interaction information of the autonomous vehicle into a Markov decision model; Based on the Markov decision model, the action space, observation space, and reward function for the multi-unmanned vehicle cooperative encirclement task are generated.

6. The multi-unmanned vehicle cooperative control method according to claim 5, characterized in that, The step of acquiring the interaction information of the autonomous vehicle and modeling the interaction information of the autonomous vehicle into a Markov decision model includes: The task is formalized as a tuple, which includes the global state space, joint observation space, joint action set, state transition function, reward function set, and discount factor; Define the continuous action space of the agent and describe the agent's motion state based on the second-order kinematic equations.

7. The multi-unmanned vehicle cooperative control method according to claim 1, characterized in that, The collected status information includes: Acquire status observation information of the unmanned vehicle used for the encirclement and capture; Acquire observation information about the target unmanned vehicle's own status; The state information is obtained by splicing the state observation information of the capture unmanned vehicle and the state observation information of the target unmanned vehicle itself.

8. The multi-unmanned vehicle cooperative control method according to claim 1, characterized in that, The multi-unmanned vehicle cooperative control method also includes: The input local state is processed by extracting features through a fully connected layer, and then the temporal dependency is modeled through an LSTM module. The attention mechanism is used to filter key features, and finally the agent's action is output. Input the joint state and joint action, extract features through a fully connected layer and LSTM module, capture the interaction relationship between agents using a multi-head attention mechanism, and output the joint Q value.

9. An electronic device, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor; when the processor runs the computer program, it implements the multi-unmanned vehicle cooperative control method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, It includes a computer program that, when run on a processor, implements the multi-unmanned vehicle cooperative control method as described in any one of claims 1 to 8.

Images