Land unmanned vehicle cooperative search and rescue simulation method

Abstract

The invention relates to the technical field of agent path planning, in particular to a land unmanned vehicle cooperative search and rescue simulation method, which comprises the following steps of: establishing a search and rescue simulation environment containing a plurality of unmanned vehicles based on a search and rescue scene; each unmanned vehicle obtains the states of all unmanned vehicles in real time; each unmanned vehicle transmits the obtained state to a neural network model and other unmanned vehicles, the neural network model generates the action probability distribution of each unmanned vehicle according to the input state information, and each unmanned vehicle selects the optimal action based on the current strategy and the corresponding action probability distribution; after each unmanned vehicle executes the optimal action and cooperates with other unmanned vehicles, the search and rescue simulation environment generates an award and a new state; training and updating the neural network model; according to the method, a near-end strategy optimization algorithm based on the reward center is utilized, the search strategy is optimized, and the task completion efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent agent path planning, and particularly to a collaborative search and rescue simulation method for land-based unmanned vehicles. Background Art

[0002] In land search and rescue missions, multi-agent collaboration is often relied upon to carry out search and rescue work. The basis for an intelligent agent to execute a task is path planning. Traditional path planning methods usually rely on static environment models and are difficult to effectively cope with complex and changeable land environments and real-time changing task requirements. During the actual operation process, multiple intelligent agents need to collaborate to complete tasks, which undoubtedly further increases the difficulty of path planning. How to ensure the collaboration between multiple intelligent agents and the overall task completion efficiency while guaranteeing the path planning efficiency has become a key problem that urgently needs to be solved.

[0003] The multi-agent proximal policy optimization algorithm provides an effective solution to the multi-agent path planning problem. By means of reinforcement learning, this algorithm enables intelligent agents to continuously learn and optimize their own action strategies during the interaction with the dynamic environment, and finally achieve efficient path planning. However, in actual application scenarios, the multi-agent proximal policy optimization algorithm often encounters the problems of slow convergence speed and unstable training process. Especially in complex multi-agent scenarios, the uneven distribution of reward signals further exacerbates these challenges. Summary of the Invention

[0004] Aiming at the deficiencies of the prior art, the present invention proposes a collaborative search and rescue simulation method for land-based unmanned vehicles, which uses a proximal policy optimization algorithm based on a reward center to optimize the search strategy and improve the task completion efficiency.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] The present invention proposes a collaborative search and rescue simulation method for land-based unmanned vehicles, including the following steps:

[0007] S1. Establish a search and rescue simulation environment containing multiple unmanned vehicles based on the search and rescue scenario;

[0008] S2. Each unmanned vehicle real-time obtains the state S of all unmanned vehicles t ;

[0009] S3. Each unmanned vehicle transmits the obtained state S t to the neural network model and other unmanned vehicles. The neural network model generates the action probability distribution of each unmanned vehicle according to the input state information, and each unmanned vehicle selects the optimal action a based on the current policy Π and the corresponding action probability distribution t ;

[0010] S4. Each of the unmanned vehicles executes the optimal action a t After collaborating with other unmanned vehicles, the search and rescue simulation environment generates a reward r t and a new state S t+1 ;

[0011] S5. Train and update the neural network model;

[0012] S6. Repeat S2 - S5 until the policy converges or the set number of training rounds is reached.

[0013] Specifically, in S1, the search and rescue simulation environment further includes the terrain, obstacles, and the positions of the wounded in the target area.

[0014] Specifically, in S2, the state S t includes the position, speed, remaining battery power, obstacle information, the position of the wounded, surrounding distress signals, etc.;

[0015] At each time step t, the state s t is crucial for the decision - making of the unmanned vehicle. The following are the specific state - constituting elements:

[0016] Current position: The position of each unmanned vehicle is represented by its coordinates in the simulation environment. The position state is used to determine the orientation of the unmanned vehicle and whether it is approaching the task target area.

[0017] Speed: The speed of each unmanned vehicle determines its movement ability in the next time step. The speed can be measured in real - time by sensors and affects the action selection of the unmanned vehicle in the simulation environment.

[0018] Remaining battery power: The battery power of the unmanned vehicle directly affects the duration of its task execution. If the battery power is too low, the unmanned vehicle may need to return for charging or adopt an energy - saving strategy. Therefore, the remaining battery power is an important dimension in the state information.

[0019] Obstacle information: Obstacles in the environment, such as buildings, rocks, or other obstacles, will affect the path planning and navigation of the unmanned vehicle. Each unmanned vehicle obtains information about the surrounding obstacles through sensors and incorporates it into the decision - making process.

[0020] Distress signal: If there are wounded in the target area, their distress signals will be transmitted to the unmanned vehicle. This information directly determines the search direction and action plan of the unmanned vehicle. Obtaining and responding to the distress signal in real - time is the key to the success of the search and rescue mission.

[0021] Specifically, in S3, the neural network model is constructed based on the Markov decision process model, and the Markov decision process model includes a state space, an action space, a transition probability, a reward function, and a discount factor.

[0022] State space: It contains all possible states in the simulation environment. Each state represents the environmental characteristics and the vehicle's own conditions at a certain moment of the driverless vehicle. For example, the state information of each driverless vehicle includes the current position, speed, remaining battery power, obstacle information, the position of the wounded, the surrounding distress signals, etc. These information constitute the state description s of each driverless vehicle. t And it changes over time.

[0023] Action space: Each driverless vehicle selects an action at each time step. The action space defines all possible behaviors that a driverless vehicle can choose in a certain state. For example, actions can include changing direction, accelerating / decelerating, conducting target search, or avoiding obstacles, etc. The action space can be discretized or continuous according to the task requirements, specifically depending on the strategies and actions implemented by the driverless vehicle in the task.

[0024] Transition probability: The transition probability defines the probability that the environment transfers from the current state s t to the next state s t after selecting the action a t+1 . In a multi-agent environment, the state of each driverless vehicle is determined not only by its own actions but also by the actions of other driverless vehicles. Therefore, the transition probability is the result of the combined action decisions of multiple driverless vehicles.

[0025] Reward function: The reward function defines the reward obtained after taking the action a t in the state s t . The reward function is a key part in reinforcement learning to guide the agent to learn optimal behaviors. In this method, the design of the reward includes task rewards, time step rewards, obstacle avoidance rewards, and multi-agent cooperation rewards, etc. These rewards are specifically described in step four, prompting the driverless vehicle to take actions that can accelerate the completion of the task.

[0026] Discount factor γ: γ is used to measure the importance of future rewards. In this method, the discount factor is used to calculate the cumulative rewards of future states, enabling each driverless vehicle to not only care about the current rewards but also consider future rewards, thus optimizing its long-term behavior strategy.

[0027] Specifically, in S3, the action probability distribution is the probability distribution of all currently available actions. All currently available actions include changing direction, accelerating / decelerating, conducting search, and avoiding obstacles.

[0028] Specifically, in S4, after each of the driverless vehicles executes the optimal action a t and cooperates with other driverless vehicles, the trajectory of each driverless vehicle under the current policy Π is obtained as {(s0, a0, r0, s1), (s1, a1, r1, s2),..., (s T-1 , aT-1 , r T-1 , s T )}, where: s t represents the state at time t, a t represents the action selected at time t, r t represents the reward obtained at time t.

[0029] Specifically, in S4, r t is calculated as follows:

[0030] Task progress reward: When the unmanned vehicle completes the task goal on time, a positive reward is given; if the task progress is slow or the task is not completed, a negative reward is given;

[0031] Time step reward: If the unmanned vehicle does not achieve the task goal in each time step, a penalty of -1 is given;

[0032] Cooperation reward: When multiple unmanned vehicles work together, all participating unmanned vehicles are rewarded;

[0033] Obstacle avoidance reward: When the unmanned vehicle successfully avoids an obstacle, a positive reward is given; if a collision occurs, a negative reward is given.

[0034] Specifically, in S5, the multi-agent proximal policy optimization algorithm based on the reward center is used to train and update the model.

[0035] Specifically, the reward center is introduced into the loss function of the Critic network in the multi-agent proximal policy optimization algorithm. The multi-agent proximal policy optimization algorithm is as follows:

[0036]

[0037] where N represents the number of samples sampled in one training, V θ (s i ) represents the value of state s i , θ represents the parameters of the neural network model, represents the reward center for the t-th update of the critic network, The update formula of is as follows:

[0038]

[0039] where β represents the learning rate parameter for the update, represents the reward center for the (t - 1)-th update of the critic network, and

[0040] Specifically, in S6, it is judged whether the policy converges according to the reward fluctuation, the stability of action selection or the set threshold. During the training process, the convergence of the policy is ensured through the following mechanism:

[0041] Iterative training: After each update, the policy is continuously optimized in multiple rounds until it converges to a stable policy.

[0042] Gradient descent and ascent: By updating the parameters of the Critic network through gradient descent of the loss function and updating the parameters of the Actor network through gradient ascent, the loss of the model is minimized.

[0043] Reward center mechanism: Introduce a reward center to accelerate the convergence of multi-agent learning, improve the overall cooperation efficiency by globally sharing reward information, and shorten the learning time.

[0044] Maximum step size and training termination condition: Set the maximum number of training rounds and set the target convergence.

[0045] Specifically, the method for judging whether the policy converges according to the reward fluctuation is as follows: In each round of training, evaluate the cumulative reward of each unmanned vehicle. If the change range of the cumulative reward within 5 consecutive training rounds is less than the set threshold, it is determined that the policy converges;

[0046] The method for judging whether the policy converges according to the stability of action selection is as follows: Check whether each unmanned vehicle selects the same action in the same environmental state. If the selected actions of the unmanned vehicle are the same in 5 consecutive training iterations, it is determined that the policy converges;

[0047] The method for judging whether the policy converges according to the set threshold is as follows: By calculating the change range of the policy, if the policy change in two consecutive rounds is less than the set threshold, it is determined that the policy converges.

[0048] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0049] (1) By combining the multi-agent proximal policy optimization algorithm and the reward center technology, the present invention enables the agents to perform path planning more efficiently in complex and dynamic environments. The multi-agent proximal policy optimization algorithm enables multiple agents to cooperate in the same environment, not only being able to quickly find the optimal path, but also being able to work together in the case of multiple tasks and multiple goals, thus significantly improving the efficiency of overall task completion; and the introduction of the reward center technology effectively accelerates the training process. By normalizing the reward signal, the variance of the reward is reduced, enabling the agents to learn effective path planning strategies in a shorter time, thus greatly shortening the training cycle. This acceleration effect is of great significance for the deployment of agents and real-time task execution in practical application scenarios.

[0050] (2) The present invention has significant advantages in terms of the stability of the algorithm. By centralizing the reward signal, the present invention reduces the instability phenomenon during the training process, making the algorithm more stable and reliable. Especially in a multi-agent system, training stability is a key factor to ensure the long-term efficient operation of the system. Therefore, the introduction of this technology effectively avoids the performance fluctuations caused by unstable training and improves the robustness of the system.

[0051] (3) The multi-agent proximal policy optimization algorithm of the present invention optimizes the collaboration ability between multiple agents by jointly optimizing the action value function. Multiple agents can better coordinate their actions to ensure that their respective goals can be effectively achieved during the collaboration process. In complex tasks and multi-objective environments, this collaborative ability is crucial for the efficiency and accuracy of task execution.

[0052] (4) By combining the multi-agent proximal policy optimization algorithm and the reward center technology, the present invention not only optimizes the path planning of a single agent but also significantly improves the ability of multiple agents to collaboratively execute complex tasks. Especially in a highly dynamic environment, agents can effectively cooperate to ensure the smooth completion of tasks, thus providing a more efficient and stable solution in practical applications. These technical advantages provide strong technical support for the application of multi-agent systems in military, rescue and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flowchart of the simulation method of the present invention;

[0054] Figure 2 is a schematic diagram of the search and rescue simulation environment of the present invention;

[0055] Figure 3 is a comparison chart of the effects of Example 1 and Comparative Example 1. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0057] Embodiment

[0058] Refer to Figure 1 , this embodiment proposes a collaborative search and rescue simulation method for unmanned ground vehicles, specifically the following steps:

[0059] S1. Refer to Figure 2,According to the complex multi-agent collaborative task, a reinforcement learning simulation environment containing multiple search and rescue unmanned vehicles is constructed.,The target area is an urban scene, including street networks (main roads, branch roads, lanes),,buildings (high-rise buildings, residential areas, collapsed buildings, ruins areas), firepower coverage,,fortifications, and the location of the wounded;

[0060] S2. During the search and rescue mission, each unmanned vehicle obtains the status of all unmanned vehicles s from the equipped perception system (such as lidar, infrared sensor, visual sensor, etc.) t , state s t Including unmanned vehicle information (current location, speed, remaining power), environmental information (obstacles, surrounding distress signals, location of the wounded, fire coverage area), the status s that the unmanned vehicle will obtain t After fusion, it is shared with other unmanned vehicles to perform tasks collaboratively. At the same time, the unmanned vehicle will obtain the state s t Input to the neural network model;

[0061] S3, the neural network according to the input environment state s t The action probability distribution of each unmanned vehicle is calculated, and the output probability distribution includes the possibility of different actions, including changing direction (selecting a new direction of travel to avoid obstacles or firepower), acceleration (speeding up and quickly approaching the location of the wounded), deceleration (decelerating and maintaining stability when passing through complex terrain), search (starting the search mode to locate potential wounded), and obstacle avoidance (by adjusting the direction or path planning to bypass the obstacles in front); wherein, the neural network model is constructed based on the Markov decision process model, and the Markov decision process model includes a state space, an action space, a transition probability, a reward function, and a discount factor. The state space includes all real-time states of the search and rescue simulation model; the action space defines all actions that the unmanned vehicle can choose at a certain moment; the transition probability defines the state s t Next, perform action a t After that, the environment moves from the current state to the next state s t+1 The reward function defines the probability of being in state s t Next, take action a t The discount factor is used to calculate the cumulative reward of future states.

[0062] S4. Each unmanned vehicle selects an optimal action from the action probability distribution generated by the neural network according to the current strategy and executes it, and obtains the trajectory of each unmanned vehicle {(s0, a0, r0, s0), (s1, a1, r1, s2), ..., (s T-1 , a T-1 , r T-1 , s T)}, where: s t represents the state at time t, a t represents the action selected at time t, r t represents the reward obtained at time t. 0 is the start time of data collection, and T is the end time of data collection. r t is generated based on the performance of the unmanned vehicle in the mission and serves as the feedback signal for each vehicle to guide the next decision-making. The calculation method is as follows:

[0063] Time-step reward: If the unmanned vehicle fails to achieve the mission goal at each time step, a penalty of -1 is given to promote the rapid completion of the mission;

[0064] Successfully search for the wounded: When the unmanned vehicle successfully finds the wounded and approaches its position, a positive reward (such as +10) is given. The reward value can increase according to the degree of proximity to the wounded;

[0065] Successful collaboration: If the unmanned vehicle successfully collaborates with other unmanned vehicles to optimize the mission execution efficiency (such as collaborative search for the wounded or avoiding enemy fire), a positive reward is given;

[0066] Mission failure: If the unmanned vehicle collides, enters the enemy fire range, or malfunctions, etc., a large negative reward (such as -10) is given.

[0067] In this way, each unmanned vehicle can adjust its behavior according to real-time data in the changing battlefield environment, making the overall mission execution more efficient and flexible. During the mission, each unmanned vehicle not only makes decisions based on its own perception and strategy but also shares real-time information with other unmanned vehicles, such as the position of the wounded and the fire coverage area. Through the coordination of the multi-agent system, it is ensured that there are no conflicts among the unmanned vehicles during mission execution, and the search and rescue efficiency can be maximized to complete complex missions;

[0068] S5. Construct the loss function of the Critic network based on the reward center as follows:

[0069]

[0070] where N represents the number of samples sampled in one training, V θ (s i ) represents the value of state s i , θ represents the parameters of the neural network model, represents the reward center for the t-th update of the critic network, The update formula of is as follows:

[0071]

[0072] where β represents the learning rate parameter for the update, represents the reward center for the (t-1)-th update of the critic network, and

[0073] the neural network model is trained and updated based on the critic network loss function of the reward center;

[0074] S6. Repeat S2 - S5 until the policy converges or reaches the set number of training rounds. To determine whether the policy converges, it can be based on reward fluctuations, action selection stability, or a set threshold;

[0075] The method for determining whether the policy converges based on reward fluctuations is: in each round of training, evaluate the accumulation of each unmanned vehicle. If the accumulated rewards in 5 consecutive training rounds tend to be stable and the change range is less than the set threshold (such as ±0.01), then it is determined that the policy converges;

[0076] The method for determining whether the policy converges based on action selection stability is: check whether each unmanned vehicle selects the same action under the same environmental state. If the selected actions of the unmanned vehicle are the same in 5 consecutive training iterations, then it is determined that the policy converges;

[0077] The method for determining whether the policy converges based on the set threshold is: by calculating the change range of the policy, if the policy change in two consecutive rounds is less than the set threshold (such as 0.001), then it is determined that the policy converges.

[0078] To avoid wasting computing resources due to too long training time and ensure an effective policy is obtained through training, a maximum number of training iterations is set. Preferably, the maximum number of training iterations is 1500 times.

[0079] In the multi-agent proximal policy optimization algorithm, the critic network and the actor network are interdependent. The critic network is mainly responsible for evaluating the value of the current policy and helping the actor network better adjust the policy. By introducing the concept of reward centralization into the loss function of the critic network, the rewards can be standardized during the training process, thereby reducing the fluctuations of the reward signals. In this way, the learning process of the critic network is more stable and the convergence speed is accelerated. At the same time, the optimization effect of the critic network can in turn accelerate the policy optimization of the actor network. Through this mutually promoting mechanism, the overall convergence speed of the algorithm is significantly improved.

[0080] Comparative Example

[0081] The difference between this comparative example and the embodiment is that in S5 of this comparative example, the neural network model is trained and updated based on multi-agent proximal policy optimization.

[0082] Reference Figure 3, it can be seen that the algorithm of the present invention is more stable and reliable, and greatly shortens the training period.

[0083] The specific embodiments of the present invention enable those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention.

[0084] It should be understood that the present invention is not limited to the above-described content and can be modified and changed without departing from its scope. The scope of the present invention is only limited by the appended claims.

Claims

1. A simulation method for collaborative search and rescue of land-based unmanned vehicles, characterized in that, It includes the following steps: S1. Establish a search and rescue simulation environment containing multiple unmanned vehicles based on the search and rescue scenario; S2. Each driverless vehicle obtains the status S of all driverless vehicles in real time t ; S3. Each driverless vehicle transmits the obtained state S t to the neural network model and other driverless vehicles. The neural network model generates the action probability distribution of each driverless vehicle according to the input state information. Each driverless vehicle selects the optimal action a based on the current policy Π and the corresponding action probability distribution t ; S4. Each of the driverless vehicles executes the optimal action a t After collaborating with other driverless vehicles, the search and rescue simulation environment generates a reward r t and a new state S t+1 ; S5. Train and update the neural network model; S6. Repeat S2 - S5 until the policy converges or reaches the set number of training rounds.

2. The onshore unmanned vehicle collaborative search and rescue simulation method according to claim 1, wherein In S1, the search and rescue simulation environment further includes the terrain, obstacles, and the positions of the wounded in the target area.

3. The onshore unmanned vehicle collaborative search and rescue simulation method according to claim 1, wherein In S2, status S t includes position, speed, remaining power, obstacle information, casualty position, surrounding distress signals, etc.

4. The onshore unmanned vehicle collaborative search and rescue simulation method according to claim 1, wherein In S3, the neural network model is constructed based on the Markov decision process model, which includes a state space, an action space, a transition probability, a reward function, and a discount factor. The state space contains all real-time states of the search and rescue simulation environment; the action space defines all actions that the unmanned vehicle can choose at a certain moment; the transition probability defines the state s t Next, perform action a t After that, the environment moves from the current state to the next state s t+1 The reward function defines the probability of being in state s t Next, take action a t The discount factor is used to calculate the cumulative reward of future states.

5. The onshore unmanned vehicle collaborative search and rescue simulation method according to claim 1, wherein, In S3, the action probability distribution is the probability distribution of all currently selectable actions, and all currently selectable actions include changing direction, accelerating / decelerating, conducting searches, and avoiding obstacles.

6. The onshore unmanned vehicle collaborative search and rescue simulation method according to claim 1, characterized in that In S4, the calculation method of r t is as follows: Task progress reward: When the unmanned vehicle completes the task objective on time, a positive reward is given; if the task progress is slow or the task is not completed, a negative reward is given; Time step reward: If the unmanned vehicle does not achieve the task objective at each time step, a penalty of -1 is given; Cooperation reward: When multiple unmanned vehicles work together, all participating unmanned vehicles are rewarded; Obstacle avoidance reward: When the unmanned vehicle successfully avoids an obstacle, a positive reward is given, and if a collision occurs, a negative reward is given.

7. The onshore unmanned vehicle collaborative search and rescue simulation method according to claim 1, wherein, In S5, the model is trained and updated using the multi - agent proximal policy optimization algorithm based on the reward center.

8. The onshore unmanned vehicle collaborative search and rescue simulation method according to claim 8, characterized in that, In the loss function of the Critic network in the multi - agent proximal policy optimization algorithm, the reward center is introduced. The multi - agent proximal policy optimization algorithm is as follows: where N represents the number of samples sampled in one training, V θ (s i ) represents the value of state s i , θ represents the parameters of the neural network model, represents the reward center for the t-th update of the critic network, and its update formula is as follows: where β represents the updated learning rate parameter, represents the reward center for the (t - 1)-th update of the critic network, and 9. The onshore unmanned vehicle collaborative search and rescue simulation method according to claim 1, wherein, In S6, it is judged whether the policy converges according to the reward fluctuation, the stability of action selection, or the set threshold.

10. The onshore unmanned vehicle collaborative search and rescue simulation method according to claim 9, wherein The method for judging whether the policy converges according to the reward fluctuation is: In each round of training, evaluate the cumulative reward of each unmanned vehicle. If the change range of the cumulative reward within 5 consecutive training rounds is less than the set threshold, it is determined that the policy converges; The method for judging whether the policy converges according to the stability of action selection is: Check whether each unmanned vehicle selects the same action under the same environmental state. If the selected actions of the unmanned vehicle are the same in 5 consecutive training iterations, it is determined that the policy converges; The method for judging whether the policy converges according to the set threshold is: By calculating the change range of the policy, if the policy change in two consecutive rounds is less than the set threshold, it is determined that the policy converges.

Images