Strategy optimization method of distributed unmanned aerial vehicle cluster control system

Abstract

The invention discloses a strategy optimization method of a distributed unmanned aerial vehicle cluster control system. The method comprises the following steps: simulating a depth camera in a simulation system based on a ray distance method to obtain a depth image; and inputting the depth image and the unmanned aerial vehicle state information into a strategy network to obtain a three-axis control vector. And each unmanned aerial vehicle samples trajectory data of a plurality of parallel environments with preset time window lengths from the tail end of the last round of trajectory. Total strategy loss is calculated based on the trajectory data through the evaluation network, and the strategy network is updated; the total policy loss includes a state value of the environment at the end of the time window. And calculating a time sequence difference target of all data pairs in the trajectory data, and updating the evaluation network through the target network. According to the method, a solution normal form of centralized training and decentralized execution is adopted, rapid training of a visual end-to-end obstacle avoidance algorithm of an unmanned aerial vehicle cluster is realized through micro-simulation, the algorithm is deployed on the unmanned aerial vehicles to realize vision-based autonomous obstacle avoidance, and the formation capability can be realized without communication between the unmanned aerial vehicles.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle control, and particularly relates to a strategy optimization method of a distributed unmanned aerial vehicle cluster control system. BACKGROUND

[0002] The unmanned aerial vehicle cluster control method can realize efficient coverage, rapid reconnaissance and precise target tracking of a specific area and other complex tasks by precisely coordinating the flight actions and task allocation of multiple unmanned aerial vehicles, greatly improving the operation efficiency and task execution capability.

[0003] There are two kinds of current unmanned aerial vehicle cluster control schemes: the first kind is the traditional unmanned aerial vehicle cluster control method, which is mostly centralized control, in which a central processor receives the observation information of all unmanned aerial vehicles, uniformly calculates and processes all information to obtain the corresponding control amount of each individual. This method highly depends on comprehensive and accurate state information input and a large amount of computing resources, and at the same time, it is necessary to ensure the communication between unmanned aerial vehicle individuals, which limits the flexibility and adaptability of the unmanned aerial vehicle cluster control system, and also increases the difficulty and cost of system design and implementation.

[0004] The second kind of technical scheme is distributed control, in which each unmanned aerial vehicle individual calculates the corresponding control amount through its own observation, which has lower requirements for unmanned aerial vehicle communication and computing power compared with centralized control, ensures the flexibility and adaptability of the cluster control system, and reduces the system complexity. However, the distributed control still mostly follows the technical path of mapping, precise positioning and trajectory planning, and the cluster control using this scheme usually has a slow flight speed, which cannot fully exert the advantages of unmanned aerial vehicles in search and rescue, hunting and other tasks.

[0005] In recent years, deep learning and reinforcement learning technologies have been integrated into unmanned aerial vehicle cluster control, using neural networks to replace mapping, positioning and trajectory planning modules, and a series of learning-based end-to-end cluster control methods have been developed, which have the advantages of short control process and fast flight speed, and have higher obstacle avoidance success rate in some diverse scenarios. However, the learning-based unmanned aerial vehicle cluster control method consumes a large amount of time for continuous interaction with the environment when dealing with tasks involving high-dimensional inputs such as images, and has low sample utilization rate, which makes it difficult to learn the extraction of image features, and task optimization is very difficult.

[0006] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0007] The technical problem solved by the present application is to provide a strategy optimization method of a distributed unmanned aerial vehicle cluster control system aiming at the above-mentioned defects of the prior art, so as to solve the problem that the learning-based unmanned aerial vehicle cluster control method in the prior art needs to consume a large amount of time for continuous interaction with the environment when involving high-dimensional input such as image, and the sample utilization rate is low, the extraction and learning of image features are insufficient, and the task optimization is very difficult.

[0008] The technical solution adopted by the present application to solve the problem is as follows: In a first aspect, the embodiment of the present application provides a strategy optimization method of a distributed unmanned aerial vehicle cluster control system, which comprises: A simulation system and a particle dynamics model of an unmanned aerial vehicle are constructed in advance; wherein in the simulation system, a depth camera is simulated based on a ray distance method to obtain a depth image; In the centralized training stage, a three-axis control vector of the unmanned aerial vehicle is output by a strategy network based on the depth image and the state information of the unmanned aerial vehicle generated by the particle dynamics model; For each unmanned aerial vehicle, a plurality of trajectory data of a preset time window length of a parallel environment are sampled from the end of the last round of trajectory; A total strategy loss is calculated by an evaluation network based on the trajectory data, and the strategy network is updated according to the total strategy loss; wherein the total strategy loss includes the state value of the environment at the end of the time window; The time difference target of all data pairs in the trajectory data is calculated, the loss value of the evaluation network is calculated according to the time difference target, and the evaluation network is updated by a target network; wherein the network structure of the target network is the same as that of the evaluation network, and the target network is updated in a soft update mode; In the distributed execution stage, the trained strategy network is deployed on the unmanned aerial vehicle individuals in the system, so that each unmanned aerial vehicle individual realizes the formation obstacle avoidance capability based on its own local observation.

[0009] In one embodiment, the depth camera is simulated based on the ray distance method to obtain the depth image, which comprises: Light rays are emitted from the position of the depth camera to the scene at a certain angle; The depth value of the corresponding pixel point is determined according to the distance from the intersection of the light ray and the environmental obstacle to the depth camera; wherein the types of the environmental obstacles include: cubic obstacles and spherical obstacles.

[0010] In one embodiment, the strategy network comprises: A convolutional neural network block is used to extract the depth image at the current time into an image feature vector; an observation model configured to generate an observation vector based on the image feature vector and current state information of the UAVs; a gated recurrent unit configured to generate a hidden state based on the observation vector; and a fully connected layer configured to generate the three-axis control vector based on the hidden state.

[0011] In an embodiment, the particle dynamics model is configured to: generate next state information of the UAVs based on the three-axis control vector and the current state information of the UAVs.

[0012] In an embodiment, the total policy loss is calculated by the evaluation network based on the trajectory data, including: the policy loss is calculated by the evaluation network based on the trajectory data, including: calculate a state value of each environment at the end of the time window based on the global state, wherein the global state includes state information of all the UAVs and environment information; calculate the total policy loss based on the policy loss and the state value.

[0013] In an embodiment, the policy loss includes a speed constraint, a control quantity constraint, a target position constraint, a UAV spacing constraint, an obstacle avoidance constraint, a formation constraint, and a collision penalty term.

[0014] In an embodiment, for each environment, if any collision occurs, the environment at the current time is a termination state. if the running time exceeds a maximum upper limit, the environment at the current time is a truncated state. if the termination state or the truncated state occurs, the environment at the current time is a reset state, and the state value of the subsequent trajectory is irrelevant to the current trajectory.

[0015] In an embodiment, for each environment, if the environment becomes a reset state due to the termination state or the truncated state, the temporal difference based on the generalized advantage function is only accumulated to the reset time, and the generalized advantage function after the reset time is re-accumulated to the end of the time window.

[0016] In a second aspect, an embodiment of the present application further provides a distributed UAV cluster control system, wherein the trained policy network is deployed on UAV individuals in the system; and the policy network is trained by using the policy optimization method of the distributed UAV cluster control system according to any one of the above embodiments.

[0017] In a third aspect, the embodiments of the present application also provide a terminal, which comprises a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing the policy optimization method of the distributed unmanned aerial vehicle cluster control system according to any one of the above; and the processors are used to execute the programs.

[0018] The embodiments of the present application have the following beneficial effects: in the simulation system, the embodiments of the present application simulate a depth camera based on a ray distance method to obtain a depth image; the depth image and unmanned aerial vehicle state information are input into a policy network to obtain a three-axis control vector. Each unmanned aerial vehicle samples a plurality of parallel environment trajectory data with a preset time window length from the end of a last round of trajectory. A total policy loss is calculated based on the trajectory data by an evaluation network, and the policy network is updated; the total policy loss comprises a state value of the environment at the end of the time window. A time difference target of all data pairs in the trajectory data is calculated, and the evaluation network is updated by a target network. The embodiments of the present application adopt a solving paradigm of centralized training and decentralized execution, realize fast training of a visual end-to-end obstacle avoidance algorithm of an unmanned aerial vehicle cluster through a differentiable simulation, and deploy the algorithm to the unmanned aerial vehicle to realize autonomous obstacle avoidance based on vision, and the unmanned aerial vehicles can realize formation capability without communication. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0020] Figure 1 is a flowchart of the policy optimization method of the distributed unmanned aerial vehicle cluster control system provided by the embodiments of the present application.

[0021] Figure 2 is a first-person rendering image of an unmanned aerial vehicle and a corresponding depth image provided by the embodiments of the present application.

[0022] Figure 3 is a schematic diagram of the training and execution idea of the cluster control algorithm provided by the embodiments of the present application.

[0023] Figure 4 is a schematic diagram of the policy network structure provided by the embodiments of the present application.

[0024] Figure 5 is a comparison diagram of the success rate and corresponding convergence time of the method of the embodiments of the present application and the proximal policy optimization algorithm (PPO) of reinforcement learning under different obstacle numbers.

[0025] Figure 6FIG. 1 is a comparison chart of success rate and convergence time of the method of the embodiment of the present application and the proximal policy optimization algorithm (PPO) of reinforcement learning under different numbers of unmanned aerial vehicles.

[0026] Figure 7 FIG. 1 is a principle block diagram of the terminal provided by the embodiment of the present application. DETAILED DESCRIPTION

[0027] The present application discloses a strategy optimization method of a distributed unmanned aerial vehicle cluster control system, in order to make the purpose, technical scheme and effect of the present application more clear and definite, the present application will be further described in detail below with reference to the drawings and by taking embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0028] Those skilled in the art can understand that, unless specifically stated otherwise, the singular forms "a", "an" and "the" used herein also include the plural forms. It should be further understood that the phrase "comprising" used in the specification of the present application means that the features, integers, steps, operations, elements and / or components exist, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be an intermediate element. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any single unit and all combinations of the associated listed items.

[0029] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as that generally understood by those skilled in the art to which the present application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have meanings consistent with those in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as such.

[0030] In view of the above defects of the prior art, the present application provides a strategy optimization method of a distributed unmanned aerial vehicle cluster control system, as shown in Figure 1 The method comprises the following steps: Step S100, a simulation system and a particle dynamics model of an unmanned aerial vehicle are constructed in advance; wherein, in the simulation system, a depth camera is simulated based on a ray distance method to obtain a depth image.

[0031] Specifically, first, a simulation system for unmanned aerial vehicle cluster visual navigation obstacle avoidance task is constructed, this step can be executed by using a preset simulation environment module, and the simulation scene is as shown inFigure 2 To reduce the complexity of the system, a simplified model is chosen as much as possible under the premise of ensuring that the model can reflect the important motion characteristics of the UAV and guarantee the accuracy. Therefore, the point mass model of the UAV is adopted in this embodiment. The modeling method is to regard the UAV as a point mass, without considering the influence of air resistance and gyroscopic moment on the UAV, only retaining the position, velocity and acceleration information, and obtaining the point mass dynamics model: ; wherein, is the acceleration vector of the UAV, is the gravity acceleration vector, is the velocity vector, is the position vector, is the control thrust vector generated by the combined thrust of the four motors, is the delay response coefficient.

[0032] In the simulation system, the perception of the UAV to the external environment is provided by simulating a depth camera. The depth image calculation of the depth camera is based on the ray distance method, and the depth image calculation process can be realized through a preset depth perception model. Further, the state iteration of the UAV based on the point mass dynamics model and the corresponding depth perception model are both realized through parallel computing on GPU by pytorch-CUDA, which can support parallel simulation of thousands of UAV clusters for autonomous navigation, and the number of UAVs in each cluster is 3 to 10.

[0033] In an implementation manner, the depth camera is simulated based on the ray distance method to obtain a depth image, including: emitting a light ray at a certain angle from the position of the depth camera to the scene; determining the depth value of the corresponding pixel point according to the distance from the intersection point of the light ray and the environmental obstacle to the depth camera; wherein the type of the environmental obstacle includes a cubic obstacle and a spherical obstacle.

[0034] Specifically, the simulation algorithm of the depth camera is realized based on the ray distance method: a ray is emitted from the position of the depth camera to the scene, for example, a light ray is emitted at a certain angle, and the light ray intersects with the environmental obstacle. Then the depth value is calculated through the intersection distance, that is, the distance from the intersection point to the camera is the depth value of the pixel point, and the depth map is a single-channel tensor.

[0035] For example, it is assumed that the starting point of a ray is , the direction of the ray is , and the distance of the ray is , then the position coordinates of the front end of the light ray are: ; Two kinds of cube obstacles and spherical obstacles are selected in the environmental obstacle setting, and for the spherical obstacle, the following conditions are met to inversely solve the ray advancing distance : ; wherein, is the spherical center coordinate, is the spherical radius. For the cube obstacle, only the following conditions are met on the six planes of the cube, the ray advancing distance corresponding to each plane can be solved , and the final ray advancing distance is: , indicating that the maximum perception distance of the depth camera is 10 meters.

[0036] Step S200, in the centralized training phase, the policy network outputs a three-axis control vector of the unmanned aerial vehicle based on the depth image and the unmanned aerial vehicle state information generated by the particle dynamics model.

[0037] Specifically, in the embodiment, the policy network (actor network) outputs a three-axis control vector (or simply referred to as a control quantity) of the unmanned aerial vehicle, and the unmanned aerial vehicle continuously interacts with the environment according to the three-axis control vector output by the policy network. The input of the policy network includes the depth image at the current time and the unmanned aerial vehicle state information. The unmanned aerial vehicle state information is generated by the particle dynamics model, and the generation manner is to input the three-axis control vector and the unmanned aerial vehicle state information at the current time into the particle dynamics model to generate the unmanned aerial vehicle state information at the next time.

[0038] In one implementation manner, the policy network comprises: a convolutional neural network block for extracting the depth image at the current time into an image feature vector; an observation model for generating an observation vector according to the image feature vector and the unmanned aerial vehicle state information at the current time; a gated recurrent unit for generating a hidden state according to the observation vector; and a fully connected layer for generating the three-axis control vector according to the hidden state.

[0039] Specifically, the structure of the policy network is as shown in Figure 4 The unmanned aerial vehicle perceives the obstacle information in the environment through the depth camera, and the depth image at each time is included in the input of the policy network. The depth image is extracted into an image feature vector by a convolutional neural network (CNN) block, which is spliced with the unmanned aerial vehicle state information to obtain an observation vector by an observation model. Then, the observation vector is input into a gated recurrent unit (GRU) layer to obtain a hidden state of the GRUThen the input into the full connection layer with 256 and 128 hidden layers, whose activation function is ELU function, to obtain the three-axis control vector , where n represents the nth UAV in the swarm, and the UAV state information is obtained through the dynamic equation: .

[0040] For example, the depth image channel number of the simulation environment is 1, and the resolution is , i.e. 16 horizontal pixels and 9 vertical pixels, which can be represented as a tensor , where is a separator, and the left side is the channel number and the right side is the shape of the corresponding tensor of the depth image. The CNN block has a total of 4 convolutional layers: (1) the input is , the convolution kernel size is , the step is 1, the padding is 2, and the output is ; (2) the convolution kernel size is , the step is 2, the padding is 1, and the output is ; (3) the convolution kernel size is , the step is 1, the padding is 0, and the output is ; (4) the convolution kernel size is , the step is 2, the padding is 1, and the output is . The activation function between the convolutional layers is the ELU function, and the output of the last convolution layer is flattened into a 96-dimensional image feature vector , which is concatenated with the UAV state information and input into the recurrent gate unit.

[0041] The observation vector of the UAV is set as follows: ; wherein , , represent the target speed, attitude quaternion and flight speed of the nth UAV, respectively.

[0042] Step S300, for each UAV, a plurality of trajectory data of a preset time window length of the parallel environment are sampled from the end of the trajectory of the last round.

[0043] Step S400, the total policy loss is calculated based on the trajectory data by the evaluation network, and the policy network is updated according to the total policy loss; wherein the total policy loss includes the state value of the environment at the end of the time window.

[0044] ​Specifically, the embodiment outputs a three-axis control vector of the UAV through a policy network, and the UAV continues to interact with the environment and collects trajectory data according to the three-axis control vector output by the policy network. Then, the policy network is trained according to the collected trajectory data to realize the update and optimization of parameters, which can be performed through a preset differentiable simulation training module. The pipeline process of training the policy network according to the collected trajectory data is performed on a GPU, without data exchange between the GPU and the CPU, so as to accelerate the interaction and policy training speed of the simulation system. For the multi-agent solution problem of UAV cluster control, in order to accurately represent the trajectory loss, the embodiment adopts a centralized training and decentralized execution mode, and a UAV cluster control algorithm based on a short-sighted actor-critic algorithm (SHAC) introduces an evaluation network (Critic network ) for evaluating the state value of the environment at the end of the time window (also referred to as the end value), thereby reducing the dependence on the length of the time window.

[0045] In an implementation manner, the total policy loss is calculated based on the trajectory data through the evaluation network, including: The loss of all UAVs in each environment at each time is calculated based on the trajectory data through the evaluation network, to obtain a policy loss; The state value of each environment at the end of the time window is calculated according to a global state; the global state includes: all UAV state information and environment information; The total policy loss is calculated according to the policy loss and the state value.

[0046] Specifically, as shown in Figure 3 , in the execution phase, each UAV is controlled according to the same policy network ; in the training phase, the evaluation network evaluates the value of the policy network according to the global state containing all UAV state information and environment information, to help the training of the policy network.

[0047] The global state is set as follows: ; Among them, , , , , , represent the position vector, attitude quaternion vector, velocity vector, target position vector, field size, and obstacle state information vector of all UAVs respectively. The obstacle state information vector contains the state of the spherical obstacle and cube obstacle state , and the two are spliced to obtain .

[0048] Further, the objective function of the policy network is set as follows, used to calculate / represent The total policy loss of the H-length trajectory starting from the initial state of the parallel environment : ; Wherein, is the number of parallel environments, H is the length of the time window, is the discount factor, is the loss function of the environment at time step , which contains the loss of all UAVs in the environment at time step . is the state value function of the environment at time step . Therefore, each sampling needs to collect data at step again to calculate the state value, multiplied by the discount , to calculate the complete total policy loss . Unlike the time sequence back propagation algorithm, the of the embodiment can pass the gradient to .

[0049] Further, the gradient of the objective function with respect to the policy parameter contains two parts: and , which can be represented as: ; .

[0050] In one implementation, the policy loss includes: speed constraint, control amount constraint, target position constraint, UAV spacing constraint, obstacle avoidance constraint, formation constraint, and collision penalty term.

[0051] Specifically, to guide the agent to learn the swarm navigation and obstacle avoidance strategy, the loss function of the policy network is designed to include the following parts: In order to make the speed v of each UAV as close as possible to the target speed, a speed penalty term is designed for each UAV: ; Wherein, is the exponential average speed of the UAV, which is obtained by updating . ​

[0052] Further, in order to prevent the unmanned aerial vehicle from being subjected to excessive control amount (jerk) and causing large shock, the control amount is limited: ; Further, the position of the unmanned aerial vehicle is constrained to reduce the distance to the target position : ; Further, the distance between the unmanned aerial vehicles is penalized: ; wherein, is the closest distance between the unmanned aerial vehicle and other unmanned aerial vehicles.

[0053] Further, the evaluation index for the unmanned aerial vehicle to avoid obstacles during flight is the approach speed and the escape speed , is the shortest distance from the unmanned aerial vehicle to the obstacle. For each obstacle, the approach penalty of the unmanned aerial vehicle can be calculated, and the obstacle K with the largest approach penalty is taken to calculate the obstacle avoidance loss : ; .

[0054] Further, the formation of the unmanned aerial vehicle formation is constrained, the unmanned aerial vehicle is regarded as a node, and the distance between the unmanned aerial vehicles is regarded as the weight between the nodes to construct a Laplacian matrix , is the degree matrix, which describes how many nodes are connected around each node, and A is the adjacency matrix, which describes the distance between nodes. The L matrix of the current time unmanned aerial vehicle formation (formation) is subjected to degree normalization: ; ; wherein, is the inverse square root of the degree matrix diagonal matrix, is the degree of the th node.

[0055] The normalized is calculated with the preset formation F norm: .

[0056] Further, the collision loss is defined as follows: ; The penalty term is triggered when the UAV collides with an obstacle, the ground or another UAV, and the penalty value is 100. The loss is greater than the sum of other loss terms, thereby effectively increasing the loss of the agent in the state-action, guiding the agent to avoid collision.

[0057] In summary, the loss function of the policy network can be represented as: .

[0058] Step S500, calculate the time difference target of all data pairs in the trajectory data, calculate the loss value of the evaluation network according to the time difference target, and update the evaluation network through the target network; wherein the network structure of the target network and the evaluation network is the same, and the target network is updated in a soft update manner.

[0059] Specifically, the training process of the embodiment requires three networks, namely the policy network , the evaluation network and the target network , and have the same network structure, and are all fully connected layers with hidden layers of 256 and 128. The target network is used to help the evaluation network train, that is, update the parameters, and the update method needs to calculate the time difference target of all data pairs in the trajectory data. The parameters are updated in a soft update manner. After updating the parameters of the evaluation network using N parallel environment trajectory data of length H+1 each time, ; wherein, is the target update coefficient, and are the parameters of the target network and the evaluation network respectively.

[0060] In one implementation, for each environment, if any collision occurs, the environment at the current time is a termination state; if the running time exceeds the maximum upper limit, the environment at the current time is a truncated state; if the termination state or the truncated state appears, the environment at the current time is a reset state, and the state value of the subsequent trajectory is irrelevant to the current trajectory.

[0061] Specifically, this embodiment defines three states for each environment: terminated state, truncated state, and reset state. If any collision occurs (including collisions between drones, collisions between drones and obstacles or the floor), the environment is currently in the terminated state. If the running time exceeds the maximum limit, the environment is currently in the truncated state. When either the terminated state or the truncated state occurs, the environment is in the reset state, meaning that the value calculation of subsequent trajectories is irrelevant to the current trajectory.

[0062] For example, in calculating the total policy loss based on collected trajectory data... At that time, the state value adopts the target network. Calculation. If there are moments of truncation in the collected trajectory. ,Right now The strategy loss for that trajectory segment is: ; in, for The global state at the moment before the reset. If If the time is reset due to the termination state, then A value of 0 indicates that the state of the trajectory after a collision is not relevant.

[0063] Furthermore, evaluate the network The objective function is: ; in, It is a temporal difference target (TD-Target), which acts as a constant in the objective function and does not participate in gradient propagation. It is generated by the target network. and Critic Network The following results were obtained through generalized advantage estimation (GAE): .

[0064] Generalized dominance function It is obtained by the exponentially weighted average of the multi-step time series differences: ; ; ; in, It is an exponential weighting factor. It is a timing differential error (TD-Error). It is a k-step advantage estimation.

[0065] In an implementation, for each environment, if the environment becomes a reset state due to a terminal state or a truncated state, the temporal difference of the generalized advantage function only accumulates to the reset time, and the generalized advantage function after the reset time needs to be re-accumulated to the end of the time window.

[0066] Specifically, in the calculation of the temporal difference target of all data pairs in trajectory sampling, if the time is reset due to a terminal state, , the in is 0. If reset is caused by a terminal state or a truncated state, the temporal difference of the generalized advantage function only accumulates to the reset time, and after the reset time needs to be re-accumulated to the end of the time window H.

[0067] The strategy optimization method of the distributed unmanned aerial vehicle cluster control system of the embodiment is based on the MASHAC algorithm (a multi-unmanned aerial vehicle version designed based on the short-sighted actor-critic algorithm (SHAC)) and a differentiable simulation design. For ease of understanding, the key steps in the actual application scenario are as follows: 1. Initialize the policy network , the evaluation network , and the target network ; 2. For update iteration (Iteration) do; 3. According to the policy network , sample trajectory data of length H+1 from the end of the trajectory of the last round for N parallel environments ; 4. Calculate the policy loss from the trajectory sampling according to the target network ; 5. Calculate the policy gradient , and update the policy network using the Adam optimizer; 6. Calculate the temporal difference target of all data pairs in trajectory sampling ; 7. Calculate the loss of the evaluation network , and update the evaluation network using the Adam optimizer; 8. Softly update the target network: ; 9. End for.

[0068] Step S600, in the distributed execution stage, the trained policy network is deployed to the UAV individuals in the system, so that each UAV individual realizes the formation obstacle avoidance capability based on its own local observation.

[0069] Specifically, finally, the trained policy network can be deployed to the UAV individuals by the control execution module, so that the UAV individuals realize the autonomous obstacle avoidance and formation capability based on only the local observation information of the individual UAV. In the implementation of the embodiment, distributed control is adopted, the policy network is deployed to the individual UAV, and the formation obstacle avoidance capability can be realized according to only the local observation of the individual UAV, so that the sample utilization rate is improved, the local optimum is avoided, and the training time is significantly shortened.

[0070] The final model effect is as shown in Figure 5 and Figure 6 As shown in the figures, compared with the classic reinforcement learning algorithm PPO, the method used in the embodiment can quickly converge to a higher success rate under different obstacle quantities in the visual end-to-end distributed UAV cluster obstacle avoidance navigation control task scene, and has a faster convergence speed and better obstacle avoidance performance: when the number of obstacles is 60, the embodiment only needs 18 minutes and 55 seconds to converge to a higher level, which is one third of the time length of the PPO algorithm. The embodiment can quickly converge to a higher success rate under different numbers of UAVs in the cluster, and when the number of UAVs is 5, the embodiment can converge to a higher level compared with the PPO algorithm using 32% of the time. The embodiment realizes the visual autonomous obstacle avoidance and formation capability of the UAV individuals under the condition of no communication, and is suitable for scenes such as security monitoring, emergency rescue and cooperative operation.

[0071] The advantages of the embodiment are as follows: 1. UAV cluster obstacle avoidance simulation task scene supporting GPU parallel acceleration: the embodiment is completely realized based on the deep learning framework of Pytorch-CUDA, a UAV particle dynamics model is built, and the image depth perception of a depth camera is calculated based on the ray distance method. In the random generation of a dense obstacle avoidance scene, thousands of UAV clusters can be simulated in parallel at a high speed, and the number of UAVs in each cluster is 3 to 10. In the UAV cluster, a single individual only maps high-dimensional information such as depth vision and radar images to the distribution of the corresponding control quantity according to its own observation.

[0072] 2. Strategy network structure based on a recurrent convolutional neural network: the embodiment adopts a convolutional neural network to extract the visual image features of the UAV, splices and fuses the features with the state information of the UAV, inputs the fused features and the hidden features extracted by the recurrent unit at the previous moment to the recurrent unit together, obtains the hidden feature output at the current moment, and then maps the control quantity through a multilayer perceptron to realize the autonomous navigation obstacle avoidance and formation capability of a single machine based on local observation.

[0073] 3. A differentiable simulation algorithm with terminal value estimation: The present application designs its multi-UAV version MASHAC based on the existing short-sighted actor-critic algorithm, adopts the actor-critic framework, and introduces terminal value estimation to accurately express the value of a trajectory compared with the existing differentiable simulation algorithm, time backpropagation algorithm, so that the optimization algorithm can include the value of the future trajectory into the optimization target, prevent the algorithm from falling into local optimum, and significantly reduce the optimization time window length requirement and speed up the calculation. The present application can realize the rapid training of the visual end-to-end obstacle avoidance algorithm of the UAV cluster in the simulator through differentiable simulation, deploy the trained strategy to each individual UAV to realize autonomous obstacle avoidance based on vision, and realize certain formation capability without communication between individual UAVs. The differentiable simulation runs completely in the GPU, can be accelerated through CUDA to achieve high-speed operation and quickly obtain a large number of samples, and its first-order gradient optimization characteristics improve the sample utilization rate, so that the image input end-to-end controller can be efficiently learned compared with the traditional reinforcement learning method.

[0074] 4. Cluster solving paradigm: The solving paradigm of centralized training and decentralized execution (CTDE) is adopted, combined with the MASHAC algorithm, and the overall value of the global state of the environment is evaluated by an evaluation network during training to help the strategy network train. The global state includes the state information of all UAVs, obstacle information, and scene information, and the strategy network only accepts local observation information of individual UAVs. During execution, distributed control is adopted, the strategy network is deployed to individual UAVs, and the formation obstacle avoidance capability can be realized only according to the local observation of itself, the sample utilization rate is improved, the local optimum is avoided, and the training time is significantly shortened.

[0075] Based on the above embodiments, the present application also provides a distributed UAV cluster control system, and the trained strategy network is deployed to individual UAVs in the system; and the strategy network is trained by using the strategy optimization method of the distributed UAV cluster control system according to any one of the above embodiments.

[0076] Based on the above embodiments, the present application also provides a terminal, and the principle block diagram thereof can be as shown in Figure 7 The terminal includes a processor, a memory, a network interface, and a display screen connected through a system bus. The processor of the terminal is used to provide calculation and control capability. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement the strategy optimization method of the distributed UAV cluster control system. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0077] Those skilled in the art can understand that, Figure 7 The principle block diagram shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the terminal to which the scheme of the present application is applied. The specific terminal can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0078] In an implementation manner, the memory of the terminal stores more than one program, and is configured to execute the more than one program by more than one processor, and the more than one program contains instructions for performing the method for policy optimization of the distributed unmanned aerial vehicle cluster control system.

[0079] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiments can be completed by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in each embodiment of the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0080] In summary, the application discloses a strategy optimization method of a distributed unmanned aerial vehicle cluster control system. The method comprises the following steps: constructing a simulation system and a particle dynamics model of an unmanned aerial vehicle in advance; in the simulation system, a depth camera is simulated based on a ray distance method to obtain a depth image; in a centralized training stage, a three-axis control vector of the unmanned aerial vehicle is output by a strategy network based on the depth image and state information of the unmanned aerial vehicle generated by the particle dynamics model; for each unmanned aerial vehicle, a preset time window length of trajectory data of a parallel environment is sampled from the end of a trajectory of the last round; a total strategy loss is calculated based on the trajectory data by an evaluation network, and the strategy network is updated according to the total strategy loss; the total strategy loss comprises a state value of the environment at the end of the time window; a time difference target of all data pairs in the trajectory data is calculated, a loss value of the evaluation network is calculated according to the time difference target, and the evaluation network is updated by a target network; the network structure of the target network is the same as that of the evaluation network, and the target network is updated in a soft update mode. The application can realize the rapid training of a visual end-to-end obstacle avoidance algorithm of an unmanned aerial vehicle cluster in a simulator through a differentiable simulation, and the autonomous obstacle avoidance based on vision can be realized by deploying the trained strategy to each individual unmanned aerial vehicle, and the unmanned aerial vehicle individuals do not need to communicate to realize a certain formation capability.

[0081] It should be understood that the application of the application is not limited to the above examples, and can be improved or changed according to the above description for those skilled in the art, and all these improvements and changes should belong to the protection scope of the appended claims of the application.

Claims

1. A strategy optimization method for a distributed unmanned aerial vehicle (UAV) swarm control system, characterized in that, The method includes: A simulation system and a particle dynamics model of the UAV are pre-constructed; wherein, in the simulation system, a depth camera is simulated based on the ray distance method to obtain a depth image; During the centralized training phase, the policy network outputs the three-axis control vector of the UAV based on the depth image and the UAV state information generated by the particle dynamics model. For each drone, trajectory data of a preset time window length is sampled starting from the end of the previous trajectory. The evaluation network calculates the total policy loss based on the trajectory data, and updates the policy network according to the total policy loss; wherein, the total policy loss includes the state value of the environment at the end of the time window; Calculate the temporal difference target for all data pairs in the trajectory data, calculate the loss value of the evaluation network based on the temporal difference target, and update the evaluation network through the target network; wherein, the target network and the evaluation network have the same network structure, and the target network is updated using a soft update method; During the distributed execution phase, the trained policy network is deployed to the individual drones in the system, enabling each drone to achieve formation obstacle avoidance capabilities based on its own local observations.

2. The strategy optimization method for a distributed unmanned aerial vehicle (UAV) swarm control system according to claim 1, characterized in that, A depth image is obtained by simulating a depth camera using the ray-distance method, including: Light rays are emitted from the position of the depth camera into the scene at a certain angle; The depth value of the corresponding pixel is determined based on the distance from the intersection of the light ray and the environmental obstacle to the depth camera; wherein, the types of environmental obstacles include: cubic obstacles and spherical obstacles.

3. The strategy optimization method for a distributed unmanned aerial vehicle (UAV) swarm control system according to claim 1, characterized in that, The policy network includes: A convolutional neural network block is used to extract the depth image at the current time step into an image feature vector; An observation model is used to generate observation vectors based on the image feature vectors and the current UAV state information; A gated loop unit is used to generate hidden states based on the observation vector; A fully connected layer is used to generate the three-axis control vector based on the hidden state.

4. The strategy optimization method for a distributed unmanned aerial vehicle (UAV) swarm control system according to claim 1, characterized in that, The particle dynamics model is used for: Based on the three-axis control vector and the UAV state information at the current moment, the UAV state information at the next moment is generated.

5. The strategy optimization method for a distributed unmanned aerial vehicle (UAV) swarm control system according to claim 1, characterized in that, The total policy loss is calculated by evaluating the network based on the trajectory data, including: The evaluation network calculates the loss of all UAVs at each moment and in each environment based on the trajectory data, and obtains the strategy loss. The state value of each environment at the end of the time window is calculated based on the global state; the global state includes: all UAV state information and environment information; The total policy loss is calculated based on the policy loss and the state value.

6. The strategy optimization method for a distributed unmanned aerial vehicle (UAV) swarm control system according to claim 5, characterized in that, The strategy losses include: speed constraints, control quantity constraints, target position constraints, UAV spacing constraints, obstacle avoidance constraints, formation constraints, and collision penalty items.

7. The strategy optimization method for a distributed unmanned aerial vehicle (UAV) swarm control system according to claim 5, characterized in that, For each environment, if any collision occurs, the environment at the current moment is in a terminated state; If the runtime exceeds the maximum limit, the environment at the current moment is in a truncated state; If a termination or truncation state occurs, the environment at the current moment is in a reset state, and the state value of subsequent trajectories is irrelevant to the current trajectory.

8. The strategy optimization method for a distributed unmanned aerial vehicle (UAV) swarm control system according to claim 7, characterized in that, For each environment, if the environment becomes a reset state due to a termination state or a truncation state, the time difference based on the generalized dominance function is only accumulated up to the reset time, and the generalized dominance function after the reset time is re-accumulated to the end of the time window.

9. A distributed unmanned aerial vehicle (UAV) swarm control system, characterized in that, The system comprises several individual drones, each of which is equipped with a trained policy network. The policy network is trained using the policy optimization method of the distributed drone swarm control system as described in any one of claims 1 to 8.

10. A terminal, characterized in that, The terminal includes a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing the strategy optimization method of the distributed unmanned aerial vehicle swarm control system as described in any one of claims 1 to 8; the processors are used to execute the programs.

Images