Multi-unmanned aerial vehicle cooperative search method and device

Abstract

The invention relates to the technical field of collaborative search task planning, in particular to a multi-unmanned aerial vehicle collaborative search method and device, designs a state information representation method and a reward function of a multi-unmanned aerial vehicle collaborative search task, and designs a network structure and a training method of a strategy model by taking an unmanned aerial vehicle search position decision as an emphasis. The method has better search efficiency and convergence, is fast in decision making, and can meet various scenes such as centralized decision making and decentralized decision making; the multi-unmanned aerial vehicle cooperative search task planning method can realize multi-unmanned aerial vehicle cooperative search task planning considering the search position and the flight path at the same time, has a good practical application value, and can enable the unmanned aerial vehicle group to learn a better search strategy through continuous game and strategy model updating, thereby completing the cooperative search task more efficiently in practical application.

Description

Technical Field

[0001] The present invention relates to the technical field of collaborative search mission planning, and in particular to a multi-UAV collaborative search method and device. Background Art

[0002] Drone collaborative search refers to multiple drones performing maneuvers and detection actions in an orderly manner within a certain area to search for hidden targets in the area. Compared with single drone search, multi-drone collaborative search can make full use of the maneuverability and detection capabilities of multiple drones to improve the search efficiency of the search operation.

[0003] At present, relevant research uses reinforcement learning algorithms such as DQN (full name: Deep Q Network) and DDPG (full name: Deep Deterministic Policy Gradient) to plan UAV collaborative search tasks. However, most of them consider collision-free path planning when the search point location is known, while there is a lack of research on the planning of the search target location itself, which makes it difficult to support the complete action process of actual collaborative search tasks.

[0004] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in the field of this technology. Summary of the invention

[0005] The technical problem to be solved by the present invention is how to realize multi-UAV collaborative search.

[0006] The present invention adopts the following technical solution:

[0007] In a first aspect, a multi-UAV collaborative search method is provided, comprising:

[0008] Build a policy model and a state-value function model;

[0009] The expectation and standard deviation of the next search position of the UAV are obtained based on the observable state information, and the result of the next search position of the UAV is obtained based on the expectation and standard deviation of the next search position of the UAV;

[0010] Generate a flight path based on the next search position result of the drone, update the status of the drone and the search target based on the flight path, and calculate the reward function value when the drone reaches the search position;

[0011] The states of the drone and the search target, the next search position of the drone, and the corresponding reward function value are stored in the sample library as sample data;

[0012] The strategy model and the state-value function model are trained according to the sample data in the sample library, and then samples are collected and the sample library is updated according to the updated strategy model. The model is trained using the updated sample library, and this process is repeated until a preset number of games is reached to obtain a trained strategy model, which is then used to make collaborative search decisions among multiple UAVs.

[0013] Preferably, the construction of the strategy model and the state value function model includes:

[0014] The two-dimensional coordinates, heading angle, navigation speed and detection radius of the current position of each UAV are combined to obtain the first initial state vector of each UAV, and the two-dimensional coordinates, initial heading angle, initial navigation speed and detection radius of the initial position of the search target are combined to obtain the second initial state vector;

[0015] According to the decision-making method adopted by the UAV, the first initial state vector and the second initial state vector, the input and output of the strategy model and the state value function model are respectively determined to obtain the strategy model and the state value function model.

[0016] Preferably, the step of determining the input and output of the strategy model and the state value function model respectively according to the decision-making method adopted by the drone, the first initial state vector, and the second initial state vector comprises:

[0017] When the drone swarm adopts a centralized decision-making method, the first initial state vector and the second initial state vector of each drone are concatenated to obtain a global state vector, and the global state vector is used as an input of the strategy model and the state value function model;

[0018] When the drone swarm adopts a decentralized decision-making method, the global state vector is used as the input of the state value function model; the first initial state vector corresponding to each drone and the second initial state vector corresponding to the search target are respectively concatenated to form a local state vector of each drone, and the strategy model corresponding to each drone uses the local state vector of each drone as input;

[0019] In both decision-making modes, the output of the strategy model is the expectation and standard deviation of the next search position of the UAV, and the output of the state value function model is the state value function.

[0020] Preferably, when the drone swarm adopts a centralized decision-making method, the strategy model includes a decoding module and multiple drone action output heads; the expectation and standard deviation of the next search position of the drone obtained according to the observable state information include:

[0021] Obtaining the global state vector according to observable state information;

[0022] The decoding module decodes the global state vector to obtain a feature vector;

[0023] Each UAV action output head outputs the expectation and standard deviation of the next search position of the corresponding UAV according to the feature vector.

[0024] Preferably, when the drone swarm adopts a decentralized decision-making method, the strategy model includes a plurality of independent drone decision modules; the expectation and standard deviation of the next search position of the drone obtained according to the observable state information include:

[0025] According to the observable state information, the local state vector of each UAV is obtained;

[0026] Each UAV decision module outputs the expectation and standard deviation of the next search position of the corresponding UAV according to the local state vector of each UAV.

[0027] Preferably, obtaining the next search position result of the drone according to the expectation and standard deviation of the next search position of the drone includes:

[0028] According to the expectation and standard deviation of the next search position of the drone, the next search position distribution is established and sampled to obtain the coordinates of the next search position of the drone and the probability of action selection. The calculation method of action selection probability is:

[0029]

[0030] Among them, (x i ,y i ) is the next search position coordinate of drone i, For the action selection probability, drone i searches for the position coordinates (x i ,y i ) The expectation and standard deviation of the horizontal axis xi are Y-axis i The mean and standard deviation of x i and i are independent of each other and all obey the normal distribution, that is,

[0031]

[0032] Among them, each drone's next search position coordinate corresponds to an action selection probability, and the drone's next search position coordinate and the corresponding action selection probability are used as the drone's next search position result.

[0033] Preferably, the step of calculating the reward function value after the drone reaches the search location includes:

[0034] When the drone reaches the search position, the reward function value is calculated according to the state parameters of the drone and the search target, wherein the state parameters of the drone and the search target include: the detection radius of the drone, the current position coordinates of the drone, and the current position coordinates of the search target;

[0035] The calculation formula of the reward function value is:

[0036]

[0037] Among them, d i is the detection radius of the drone, k is the reward function coefficient and k∈(0,1), d is the Euclidean distance between drone i and the search target, (x i ,y i ) is the current position coordinate of UAV i, (x o ,y o ) is the current position coordinate of the search target.

[0038] Preferably, the training method of the strategy model includes: calculating the loss value L(θ) of each sample data according to the sample data in the sample library and the network parameter θ of the current strategy model, and the calculation method of L(θ) is:

[0039] L(θ)=-min(Rt(θ)At,clip(Rt(θ),1-ε,1+ε)At)

[0040] Among them, min() is the minimum function, ε is the hyperparameter, R t (θ) is the action selection probability ratio of the new and old strategies, clip() is the truncation function, A t is the advantage function value, R t (θ) and A t The calculation methods are:

[0041]

[0042] A t =r t +γV(s t+1 )-V(s t )

[0043] Among them, s t is the environmental state of the t-th step in the multi-UAV collaborative search game, a t is the drone action at step t, s t+1 is the environmental state of step t+1, which includes the state of the UAV and the search target, For the old strategy in s t Select action a in state t The action selection probability, πθ (a t |s t ) is the latest strategy model in s t Select action a in state t The action selection probability, r t For state s t Next, perform action a t The reward function value obtained, γ is the reward discount coefficient, and V() is the state value function;

[0044] According to the loss value of each sample data in the sample library, the average loss value of each sample data is calculated, and the network parameters of the strategy model are updated with the goal of minimizing the average loss value of the sample data to realize the training of the strategy model.

[0045] Preferably, the training method of the state value function model includes: calculating the state value function model loss value L of each sample data according to the sample data in the sample library and the network parameter φ of the current state value function model V (φ), L V (φ) is calculated as:

[0046] L V (φ)=[r t +γV(s t+1 )-V(s t )] 2

[0047] Among them, s t is the environmental state of the t-th step in the multi-UAV collaborative search game, s t+1 is the environmental state at step t+1, r t For state s t Next, perform action a t The reward function value obtained, γ is the reward discount coefficient, and V() is the state value function;

[0048] According to the state value function model loss value of each sample data in the sample library, the average loss value of each sample data is calculated, and the network parameters of the state value function model are updated with the goal of minimizing the average loss value of the sample data to achieve the training of the state value function model. Preferably, the generation of the flight path according to the next search position result of the drone includes:

[0049] A preset path planning algorithm is used to generate a flight path for the UAV according to the next search position result of the UAV, wherein the preset path planning algorithm includes an A* algorithm or an RRT algorithm.

[0050] In a second aspect, a multi-UAV collaborative search device is provided, the multi-UAV collaborative search device comprising: a processor and a memory for storing instructions executable by the processor;

[0051] Wherein, the processor is configured to execute the multi-UAV collaborative search method described in the first aspect.

[0052] In a third aspect, a non-volatile computer storage medium is provided, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the multi-UAV collaborative search method described in the first aspect.

[0053] In a fourth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in the memory from a memory to execute the multi-UAV collaborative search method as described in the first aspect.

[0054] In a fifth aspect, a computer program product comprising instructions is provided. When the instructions are executed on a computer or a processor, the computer or the processor executes the multi-UAV collaborative search method as described in the first to fourth aspects and any one of them.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] The present invention provides a multi-UAV collaborative search method, designs a state information representation method and a reward function for a multi-UAV collaborative search task, focuses on UAV search position decision-making, designs a network structure and a training method for a strategy model, has good search efficiency and convergence, makes decisions quickly, and can meet a variety of scenarios such as centralized decision-making and decentralized decision-making; can realize multi-UAV collaborative search task planning that simultaneously considers search positions and flight paths, has good practical application value, and through continuous game play and updating of strategy models, a swarm of UAVs can learn a better search strategy, thereby completing the collaborative search task more efficiently in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 It is a flowchart of a multi-UAV collaborative search method provided by an embodiment of the present invention;

[0059] Figure 2It is a schematic diagram of a construction process of a construction strategy model and a state value function model provided by an embodiment of the present invention;

[0060] Figure 3 It is a schematic diagram of a process for obtaining the expectation and standard deviation of the next search position of the drone provided by an embodiment of the present invention;

[0061] Figure 4 It is a structural schematic diagram of a multi-UAV collaborative search device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0063] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0064] In the description of the present invention, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more. In addition, for example, the same type of nouns may be described as two independent individuals by adding "A" and "B" at the end. In this case, the corresponding features defined as "A" and "B" are only used to distinguish the same type of individuals for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features.

[0065] When describing some embodiments, the expressions "coupling", "coupling" and "connection" and their derivatives may be used. For example, when describing some embodiments, the term "connection" may be used to indicate that two or more components are in direct physical or electrical contact with each other. For another example, when describing some embodiments, the term "coupling" may be used to indicate that two or more components are in direct physical or electrical contact. However, the terms "connection" or "coupling" may also refer to two or more components that are not in direct contact with each other, but still cooperate or interact with each other, such as "optical path coupling", "wireless connection", etc. The embodiments disclosed herein are not necessarily limited to the contents of the present invention.

[0066] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0067] Embodiment 1:

[0068] Drone collaborative search is the orderly maneuvering and detection actions of multiple drones in a specific area to search for hidden targets in the area. This means that drones need to cooperate and coordinate with each other to complete the search mission together. Compared with single drone search, multi-drone collaborative search has obvious advantages. Multiple drones can make full use of their own maneuvering and detection capabilities to expand the search range and improve the search speed and accuracy. By working together, drones can cover a larger area in a shorter time and improve the efficiency of search operations.

[0069] Most existing studies consider collision-free path planning when the search point location is known. When the drone knows where the target may appear, it can plan a collision-free path to search these locations. However, in actual situations, the location of the search target is often unknown, and the drone needs to determine the location of the search target through detection and search. There is a lack of research on the planning of the search target location itself. This makes it difficult for existing methods to support the complete action process of actual collaborative search tasks. In actual tasks, drones are not only required to be able to plan collision-free paths to known locations, but also to be able to effectively explore unknown areas, determine where the search target may appear, and search.

[0070] To address the deficiencies of the prior art, this embodiment provides a multi-UAV collaborative search method. In one embodiment, Figure 1 As shown, the method includes:

[0071] Step 101: Construct a policy model and a state-value function model.

[0072] Among them, the policy model is used to determine what actions the drone takes in different states, that is, to determine the next search location of the drone. According to the observable state information, the expectation and standard deviation of the next search location of the drone are output. The state value function model is used to evaluate the value under a specific state. In multi-drone collaborative search, it can help determine the long-term benefits of taking different actions in the current state and provide a reference for the optimization and update of the policy model.

[0073] In one embodiment, Figure 2 As shown, the step 101 specifically includes:

[0074] Step 1011: Combine the two-dimensional coordinates, heading angle, navigation speed and detection radius of each drone's current position to obtain the first initial state vector of each drone, and combine the two-dimensional coordinates, initial heading angle, initial navigation speed and detection radius of the search target's initial position to obtain the second initial state vector.

[0075] Among them, the two-dimensional coordinates of the current position of the drone are combined to clarify the specific position of the drone in space. The two-dimensional coordinates are usually composed of horizontal and vertical coordinates, which can accurately locate the location of the drone. The heading angle indicates the direction of the drone's flight and determines the possible direction of the drone's next movement. The navigation speed reflects the speed of the drone's movement and is used to predict the change of the drone's position within a certain period of time and the expansion of the search range. The detection radius limits the range within which the drone can detect the target, which helps to determine the effective search area of ​​the drone during the search process.

[0076] The two-dimensional coordinates of the initial position of the search target provide the position information of the target at the beginning of the search and are the reference point of the target for the drone to search. The initial heading angle and initial navigation speed of the search target may be used to infer the movement trend of the target. When the search target is in a dynamic moving state, these parameters can help the drone predict the possible location of the target.

[0077] Step 1012: Determine the input and output of the strategy model and the state value function model respectively according to the decision-making method adopted by the UAV, the first initial state vector and the second initial state vector to obtain the strategy model and the state value function model.

[0078] Among them, in the collaborative search of multiple drones, the centralized decision-making method and the distributed decision-making method have different characteristics and application scenarios. When the drone group adopts a centralized decision-making method, the first initial state vector and the second initial state vector of each drone are spliced ​​to obtain a global state vector, and the global state vector is used as the input of the strategy model and the state value function model. In centralized decision-making, the state information of all drones (such as position, heading angle, navigation speed, detection radius, etc.) and the state information of the search target are collected to a central controller or decision center. The central controller calculates and analyzes based on the global state vector (which is spliced ​​by the state vectors of all drones and targets) to determine the next search position of each drone. The global state vector is used as the input of the strategy model and the state value function model. The strategy model outputs the expectation and standard deviation of the next search position of the drone based on this global state vector to determine the next action direction of each drone in the entire drone group. The state value function model outputs a state value function for evaluating the current state value of the entire system.

[0079] When the drone swarm adopts a decentralized decision-making method, the global state vector is used as the input of the state value function model; the first initial state vector corresponding to each drone and the second initial state vector corresponding to the search target are respectively spliced ​​to form the local state vector of each drone, and the strategy model corresponding to each drone uses the local state vector of each drone as input. In both decision-making modes, the output of the strategy model is the expectation and standard deviation of the next search position of the drone, and the output of the state value function model is the state value function. In distributed decision-making, each drone has its own strategy model and makes decisions based on the local state vector (which is spliced ​​by its own state information and the search target state information). Drones can share information and coordinate through limited communication, but each drone mainly makes decisions based on its own local information and local goals.

[0080] Through different decision-making methods and using different state vectors as input, the strategy model and state-value function model can provide effective decision support for multi-UAV collaborative search to adapt to different application scenarios and needs.

[0081] In one embodiment, the network structure of the policy model and the state value function model can be determined according to the decision-making mode of the drone group and the dimension of the state vector (including the global state vector or the local state vector) of the model input. In centralized decision-making, since the state vectors of all drones and search targets are spliced ​​to obtain a global state vector as input, the dimension of this state vector is relatively high, covering all state information of the entire drone group and the target. In order to process such high-dimensional input, the policy model and the state value function model may need to adopt a more complex network structure. For example, a deep neural network can be used with multiple hidden layers and a large number of neurons to fully extract the feature information in the global state vector, so as to accurately output the expectation and standard deviation of the next search position of the drone and the state value function. In distributed decision-making, the policy model of each drone takes the local state vector as input, and the dimension of this vector is relatively low, which only contains the state information of the specific drone itself and the search target. The policy model of each drone can adopt a relatively simple network structure. For example, a shallower neural network or other model structure suitable for processing low-dimensional input can be used to quickly make decisions based on local state information.

[0082] It is worth noting that in both decision-making methods, the network of the state-value function model is the same, and the input is the global state vector, and the output is the state function scalar value. The network structure of the state-value function model can adopt a fully connected neural network.

[0083] In one embodiment, after the strategy model and the state value function model are obtained, the network parameters of the strategy model and the state value function model are initialized, and then the UAV collaborative search simulation environment and the initial states of the UAV and the search target are initialized.

[0084] When the drone swarm adopts a centralized decision-making method, the strategy model adopts the structure of "single decoding module + multiple drone action output heads". When the drone swarm adopts a decentralized decision-making method, compared with the strategy model structure of the centralized decision-making method, the main difference in its network structure is that the network models corresponding to each drone do not share a decoding module, and the network model of each drone inputs the local state vector corresponding to each drone. The network model of each drone can adopt a fully connected neural network, and the network structure remains the same. The network model of each drone can adopt 2 hidden layers, the number of hidden layer neurons is set to 128, and the Tanh activation function is adopted.

[0085] Step 102: Obtain the expectation and standard deviation of the next search position of the drone based on the observable state information, and obtain the next search position result of the drone based on the expectation and standard deviation of the next search position of the drone.

[0086] The observable state information includes the current position of the drone, the possible position range of the search target, and environmental information. The next search position result of each drone includes the next search position coordinates and action selection probability of each drone. In one embodiment, the global state vector is obtained based on the observable state information, such as Figure 3 As shown, step 102 includes:

[0087] Step 1021: The decoding module decodes the global state vector to obtain a feature vector.

[0088] In a multi-UAV system, the decoding module of the policy model is responsible for converting the global state vector into a feature vector that can be used for decision making. The decoding module receives the global state vector as input, processes the global state vector through a series of neural network layers (such as convolutional layers, recurrent layers, or Transformer layers), and extracts important features to obtain a feature vector.

[0089] Step 1022: Each UAV action output head outputs the expectation and standard deviation of the next search position of the corresponding UAV according to the feature vector.

[0090] The action output head of each drone contains four independent networks, which output the expectation and standard deviation of the next search position coordinates of the corresponding drone. Each network is responsible for outputting the expected value or standard deviation of the next search position coordinates of a specific drone. The above design allows each drone to make decisions independently based on the current state while maintaining the independence between their action choices.

[0091] In one embodiment, step 102 further includes: establishing a next search position distribution and performing sampling according to the expectation and standard deviation of the next search position of the drone, obtaining the next search position coordinates of the drone and the action selection probability, and the action selection probability is calculated as follows:

[0092]

[0093] Among them, (x i ,y i ) is the next search position coordinate of drone i, For the action selection probability, drone i searches for the position coordinates (x i ,y i ) The expectation and standard deviation of the horizontal axis xi are Y-axis i The mean and standard deviation of x i and i are independent of each other and all obey the normal distribution, that is,

[0094]

[0095] Among them, each drone's next search position coordinate corresponds to an action selection probability, and the drone's next search position coordinate and the corresponding action selection probability are used as the drone's next search position result.

[0096] For example, if a normal distribution is used to represent the next search position of each drone, then each action output head may output a normal distribution parameter, namely the mean (expected value) and variance (standard deviation). In this way, each drone can select a random position based on the parameters generated by its own action output head, and this position follows a specific normal distribution. In addition, in one embodiment, it is assumed that the decision results of each drone are independent of each other, and the decision results of the drone group are (x1, y1), (x2, y2), ..., (x n ,y n ), the overall action selection probability is equal to the product of the action selection probabilities of each drone, as follows:

[0097]

[0098] Among them, P is the overall action selection probability, Select probabilities for actions of a single drone.

[0099] In one embodiment, when the drone group adopts a decentralized decision-making method, the strategy model includes multiple independent drone decision modules; the expectation and standard deviation of the next search position of the drone according to the observable state information include: obtaining the local state vector according to the observable state information; each drone decision module outputs the expectation and standard deviation of the next search position of the corresponding drone according to the local state vector of each drone. Step 103: Generate a flight path according to the next search position result of the drone, update the state of the drone and the search target according to the flight path, and calculate the reward function value when the drone reaches the search position.

[0100] Among them, a preset path planning algorithm is used to generate the flight path of the drone according to the next search position result of the drone, wherein the preset path planning algorithm includes an A* algorithm or an RRT algorithm. According to the determined next search position result of the drone, the flight path of the drone is planned by a preset path planning algorithm. The flight path can take into account factors such as the flight performance of the drone, environmental restrictions (such as obstacles, wind direction, etc.). When the drone flies according to the flight path and reaches the search position, the status of the drone and the search target is updated. For example, if the position of the drone changes, the position of the search target may also be updated accordingly due to some factors (such as movement, being discovered by the drone, etc.).

[0101] The reward function value is used to measure the effect of a drone taking a specific action in a specific state. In multi-drone collaborative search, the reward function value can be designed based on the search results (including whether the target is found, search efficiency, collaborative effect, etc.). When the drone arrives at the search location, the reward function value is calculated based on the current state and action results. If the drone finds the target or makes some progress in the search process, the reward function value may be high; conversely, if the search does not progress or an error occurs, the reward function value may be low.

[0102] The flight path generated by the preset path planning algorithm specifies a series of navigation points for each drone, and the navigation points constitute the flight trajectory of the drone. In the simulation environment, as the drone flies, the status of the drone and the search target will be continuously updated. The reward function value is used to evaluate the behavior of the drone. The reward value is usually calculated based on factors such as whether the drone is close to the search target and whether it follows the established path. The design of the reward function can encourage the drone to take behaviors that are conducive to completing the task. In one embodiment, when the drone arrives at the search position, the reward function value is calculated based on the drone and search target state parameters, wherein the drone and search target state parameters include: the detection radius of the drone, the current position coordinates of the drone, and the current position coordinates of the search target; the calculation formula of the reward function value is:

[0103]

[0104] Among them, d i is the detection radius of the drone, k is the reward function coefficient and k∈(0,1), d is the Euclidean distance between drone i and the search target, (x i ,y i ) is the current position coordinate of UAV i, (x o ,y o ) is the current position coordinate of the search target.

[0105] Step 104: The states of the drone and the search target, the next search position result of the drone, and the corresponding reward function value are stored in the sample library as sample data.

[0106] Among them, the state of the drone includes the two-dimensional coordinates of the drone's position, heading angle, navigation speed, detection radius and other information. These state information describes the current situation of the drone during the search process. The state of the search target includes the two-dimensional coordinates of the search target's position, heading angle (if the target is movable), speed, detection radius (if the target has detection capability), etc.; the next search position result of the drone is the search position and action selection probability that the drone will go to next, calculated by the strategy model based on the states of the aforementioned drone and the search target, reflecting the decision result of the drone in the aforementioned state, and is the specific action instruction output by the strategy model. The corresponding reward function value is calculated based on the situation after the drone arrives at the search position, which is used to measure the effect of the drone taking action at this position. The states of the drone and the search target, the next search position result of the drone and the corresponding reward function value are stored in the sample library as sample data. The sample data records the actions and results of the drone in different states. The sample data in the sample library is equivalent to the experience playback in reinforcement learning, which helps the strategy model learn from past successes and failures to better adapt to different search scenarios and task requirements. For the specific training process, please refer to the following steps.

[0107] Step 105: Train the strategy model and the state-value function model according to the sample data in the sample library, collect samples according to the updated strategy model and update the sample library, and use the updated sample library to train the model again, and iterate until the preset number of games is reached to obtain a trained strategy model, and use the trained strategy model to make multi-UAV collaborative search decisions.

[0108] In the process of reinforcement learning, the sample library is an important data source for the training algorithm. When the number of samples in the sample library reaches a preset threshold, the training process can be started. In one embodiment, the training method of the strategy model includes: calculating the loss value L(θ) of each sample data according to the sample data in the sample library and the network parameter θ of the current strategy model. The calculation method of L(θ) is:

[0109] L(θ)=-min(Rt(θ)At,clip(Rt(θ),1-ε,1+ε)At)

[0110] Among them, min() is the minimum function, ε is the hyperparameter, R t (θ) is the action selection probability ratio of the new and old strategies, clip() is the truncation function, A t is the advantage function value, R t (θ) and A t The calculation methods are:

[0111]

[0112] At =r t +γV(s t+1 )-V(s t )

[0113] Among them, s t is the environmental state of the t-th step in the multi-UAV collaborative search game, a t is the drone action at step t, s t+1 is the environmental state of step t+1, which includes the state of the UAV and the search target, For the old strategy in s t Select action a in state t The action selection probability, π θ (a t |s t ) is the latest strategy model in s t Select action a in state t The action selection probability, r t For state s t Next, perform action a t The reward function value obtained, γ is the reward discount coefficient, and V() is the state value function.

[0114] According to the loss value of each sample data in the sample library, the average loss value of each sample data is calculated, and the network parameters of the strategy model are updated with the goal of minimizing the average loss value of the sample data to implement the training of the strategy model. In one embodiment, as shown in the following formula, the strategy model training is implemented, and the Adam algorithm can be used to update the network parameters θ of the strategy model.

[0115]

[0116] In the formula, α is the learning rate.

[0117] Regardless of whether it is a centralized decision-making method or a decentralized decision-making method, the above training method can be used to calculate the overall action selection probability ratio and advantage function value of the drone swarm, and then calculate the loss value to realize strategy model training.

[0118] In one embodiment, the training method of the state value function model is: according to the sample data in the sample library and the network parameter φ of the current state value function model, the state value function model loss value L of each sample data is calculated. V (φ), L V (φ) is calculated as:

[0119] L V (φ)=[r t +γV(s t+1 )-V(s t )]2

[0120] According to the state value function model loss value of each sample data in the sample library, the average loss value of each sample data is calculated, and the network parameters of the state value function model are updated with the goal of minimizing the average loss value of the sample data to realize the training of the state value function model. In one embodiment, as shown in the following formula, the state value function model training is realized, and the network parameter φ of the state value function model can be updated by using the Adam algorithm.

[0121] φ←φ-βΔL V (φ)

[0122] Where β is the learning rate. Based on the same batch of sample data, the policy model and the state value function model can be updated multiple times to make full use of the sample data. The number of update rounds can be 10.

[0123] In one embodiment, the training process will continue until a preset number of games (i.e., the number of simulated drone search tasks) is reached, and a trained policy model is obtained. The trained policy model will contain information on how drones should act in various environmental conditions, helping drones to work together in actual applications to most effectively complete search tasks.

[0124] In summary, the present invention provides a multi-UAV collaborative search method, designs a state information representation method and a reward function for a multi-UAV collaborative search task, focuses on UAV search position decision, designs a network structure and a training method for a strategy model, has good search efficiency and convergence, makes decisions quickly, and can meet a variety of scenarios such as centralized decision-making and decentralized decision-making; can realize multi-UAV collaborative search task planning that simultaneously considers the search position and flight path, has good practical application value, and through continuous game play and updating of the strategy model, the UAV swarm can learn a better search strategy, thereby more efficiently completing the collaborative search task in practical applications.

[0125] Embodiment 2:

[0126] In Example 1, a multi-UAV collaborative search method is provided. In this embodiment, a multi-UAV collaborative search device is proposed. The multi-UAV collaborative search device includes: a processor and a memory for storing processor executable instructions; wherein the processor is configured to execute the multi-UAV collaborative search method described in Example 1.

[0127] like Figure 4 As shown, the multi-UAV collaborative search device includes a processor 21 and a memory 22, wherein the processor 21 and the memory 22 can be connected via a bus or other means.

[0128] The processor 21 may be a central processing unit (CPU). The processor 21 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0129] The memory 22 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the multi-UAV cooperative search method in Embodiment 1 of the present invention. The processor executes various functional applications and training processes of the processor by running the non-transitory software programs, instructions and modules stored in the memory.

[0130] The memory 22 may include a program storage area and a training storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the training storage area may store training created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 22 may optionally include a memory remotely located relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The one or more modules are stored in the memory 22, and when executed by the processor 21, execution such as Figure 1 The multi-UAV collaborative search method in the embodiment 1 is shown. The specific details of the multi-UAV collaborative search method can be found in Figure 1 , Figure 2 and Figure 3 The corresponding descriptions and effects in the embodiments shown in the figure can be understood and will not be repeated here.

[0131] This embodiment also provides a computer storage medium, which stores a computer program. The computer program can be executed by a processor to complete the multi-UAV collaborative search method described in Example 1.

[0132] The computer storage medium stores computer executable instructions, and the computer executable instructions can execute the multi-UAV collaborative search method in any of the above method embodiments. The storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk (HDD), or a solid-state drive (SSD); the storage medium can also include a combination of the above types of memory.

[0133] The specific steps of the multi-UAV collaborative search method are shown in Example 1 and will not be repeated in this example.

[0134] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-UAV collaborative search method, characterized in that: include: Build a policy model and a state-value function model; The expectation and standard deviation of the next search position of the UAV are obtained based on the observable state information, and the result of the next search position of the UAV is obtained based on the expectation and standard deviation of the next search position of the UAV; Generate a flight path based on the next search position result of the drone, update the status of the drone and the search target based on the flight path, and calculate the reward function value when the drone reaches the search position; The states of the drone and the search target, the next search position of the drone, and the corresponding reward function value are stored in the sample library as sample data; The strategy model and the state-value function model are trained according to the sample data in the sample library, and then samples are collected and the sample library is updated according to the updated strategy model. The model is trained using the updated sample library, and this process is repeated until a preset number of games is reached to obtain a trained strategy model, which is then used to make collaborative search decisions among multiple UAVs.

2. The multi-UAV collaborative search method according to claim 1, characterized in that: The construction strategy model and state value function model include: The two-dimensional coordinates, heading angle, navigation speed and detection radius of the current position of each UAV are combined to obtain the first initial state vector of each UAV, and the two-dimensional coordinates, initial heading angle, initial navigation speed and detection radius of the initial position of the search target are combined to obtain the second initial state vector; According to the decision-making method adopted by the UAV, the first initial state vector and the second initial state vector, the input and output of the strategy model and the state value function model are respectively determined to obtain the strategy model and the state value function model.

3. The multi-UAV collaborative search method according to claim 2, characterized in that: The step of respectively determining the input and output of the strategy model and the state value function model according to the decision-making method adopted by the drone, the first initial state vector, and the second initial state vector includes: When the drone swarm adopts a centralized decision-making method, the first initial state vector and the second initial state vector of each drone are concatenated to obtain a global state vector, and the global state vector is used as an input of the strategy model and the state value function model; When the drone swarm adopts a decentralized decision-making method, the global state vector is used as the input of the state value function model; the first initial state vector corresponding to each drone and the second initial state vector corresponding to the search target are respectively concatenated to form a local state vector of each drone, and the strategy model corresponding to each drone uses the local state vector of each drone as input; In both decision-making modes, the output of the strategy model is the expectation and standard deviation of the next search position of the UAV, and the output of the state value function model is the state value function.

4. The multi-UAV collaborative search method according to claim 3, characterized in that: When the drone swarm adopts a centralized decision-making method, the strategy model includes a decoding module and multiple drone action output heads; the expectation and standard deviation of the next search position of the drone obtained based on the observable state information include: Obtaining the global state vector according to observable state information; The decoding module decodes the global state vector to obtain a feature vector; Each UAV action output head outputs the expectation and standard deviation of the next search position of the corresponding UAV according to the feature vector.

5. The multi-UAV collaborative search method according to claim 3, characterized in that: When the drone swarm adopts a decentralized decision-making method, the strategy model includes multiple independent drone decision modules; the expectation and standard deviation of the next search position of the drone obtained based on the observable state information include: Obtaining the local state vector according to observable state information; Each UAV decision module outputs the expectation and standard deviation of the next search position of the corresponding UAV according to the local state vector of each UAV.

6. The multi-UAV collaborative search method according to claim 1, characterized in that: The method of obtaining the next search position result of the drone according to the expectation and standard deviation of the next search position of the drone includes: According to the expectation and standard deviation of the next search position of the drone, the next search position distribution is established and sampled to obtain the coordinates of the next search position of the drone and the probability of action selection. The calculation method of action selection probability is: Among them, (x i ,y i ) is the next search position coordinate of drone i, For the action selection probability, drone i searches for the position coordinates (x i ,y i ) The expectation and standard deviation of the horizontal axis xi are Y-axis i The mean and standard deviation of x i and i are independent of each other and all obey the normal distribution, that is, Among them, each drone's next search position coordinate corresponds to an action selection probability, and the drone's next search position coordinate and the corresponding action selection probability are used as the drone's next search position result.

7. The multi-UAV collaborative search method according to claim 1, characterized in that: The calculation of the reward function value when the drone reaches the search position includes: When the drone reaches the search position, the reward function value is calculated according to the state parameters of the drone and the search target, wherein the state parameters of the drone and the search target include: the detection radius of the drone, the current position coordinates of the drone, and the current position coordinates of the search target; The calculation formula of the reward function value is: Among them, d i is the detection radius of the drone, k is the reward function coefficient and k∈(0,1), d is the Euclidean distance between drone i and the search target, (x i ,y i ) is the current position coordinate of UAV i, (x o ,y o ) is the current position coordinate of the search target.

8. The multi-UAV collaborative search method according to claim 1, characterized in that: The training method of the strategy model includes: calculating the loss value L(θ) of each sample data according to the sample data in the sample library and the network parameter θ of the current strategy model. The calculation method of L(θ) is: L(θ)=-min(Rt(θ)At,clip(Rt(θ),1-ε,1+ε)At) Among them, min() is the minimum function, ε is the hyperparameter, R t (θ) is the action selection probability ratio of the new and old strategies, clip is the truncation function, A t is the advantage function value, R t (θ) and A t The calculation methods are: Among them, s t is the environmental state of the t-th step in the multi-UAV collaborative search game, a t is the drone action at step t, s t+1 is the environmental state of step t+1, which includes the state of the UAV and the search target, For the old strategy in s t Select action a in state t The action selection probability, π θ (a t |s t ) is the current latest strategy model in s t Select action a in state t The action selection probability, r t For state s t Next, perform action a t The reward function value obtained, γ is the reward discount coefficient, and V() is the state value function; According to the loss value of each sample data in the sample library, the average loss value of each sample data is calculated, and the network parameters of the strategy model are updated with the goal of minimizing the average loss value of the sample data to realize the training of the strategy model.

9. The multi-UAV collaborative search method according to claim 1, characterized in that: The training method of the state value function model includes: calculating the state value function model loss value L of each sample data according to the sample data in the sample library and the network parameter φ of the current state value function model V (φ), L V (φ) is calculated as: 50 V (φ)=[r t +γV(s t+1 )-V(s t )] 2 Among them, s t is the environmental state of the t-th step in the multi-UAV collaborative search game, s t+1 is the environmental state at step t+1, r t For state s t Next, perform action a t The reward function value obtained, γ is the reward discount coefficient, and V() is the state value function; According to the loss value of the state value function model of each sample data in the sample library, the average loss value of each sample data is calculated, and the network parameters of the state value function model are updated with the goal of minimizing the average loss value of the sample data to realize the training of the state value function model.

10. A multi-UAV collaborative search device, characterized in that: The multi-UAV collaborative search device comprises: a processor and a memory for storing processor executable instructions; wherein the processor is configured to execute the multi-UAV collaborative search method according to any one of claims 1-8.

Images