Method, equipment and medium for cooperative control of multiple unmanned aerial vehicles based on neural network

Abstract

The present application discloses a method, device and medium for cooperative control of multiple drones based on neural networks, which relates to the field of deep learning. The method includes: determining the corresponding topological structure for each formation mode; generating the corresponding topological matrix based on the direction of information flow between nodes in the topological structure; generating a graph convolutional neural network; extracting the multi-dimensional flight features corresponding to the drone formation through the skeleton network; outputting the corresponding feature vector according to the multi-dimensional flight features and the adjacency matrix through the graph convolution module; outputting the corresponding attention features according to the feature vector through the attention module, and obtaining the feature matrix corresponding to the drone formation according to the attention features; outputting the cooperative control instructions corresponding to the drone formation based on the feature matrix. The graph convolutional neural network can generate more accurate and efficient cooperative control instructions, thereby improving the perception range and improving the cooperative efficiency of the formation.

Description

Technical Field

[0001] The present application relates to the field of deep learning, and specifically to a method, device and medium for collaborative control of multiple drone formations. Background Art

[0002] With the development of technology, drone multi-machine formation technology has shown wide application potential in many fields. Through the collaborative work of multiple drones, the efficiency and success rate of mission execution can be significantly improved, reconnaissance and surveillance capabilities can be enhanced, endurance can be extended, and anti-interference capabilities can be enhanced. For example, in some fields, it can improve the efficiency and benefits of industries such as logistics distribution, agricultural plant protection, and environmental monitoring. In addition, drone formations can also bring wonderful visual effects to the performance field and play an important role in resource exploration, forest fire prevention and disaster relief, and other fields.

[0003] However, in practical applications, multi-drone formation technology not only requires each drone to maintain a close coordination relationship, but also needs to cope with various complex environmental conditions, such as wind changes, obstacle avoidance, and possible communication interference. Traditional methods, such as rule-based control algorithms or model-based optimization methods, often find it difficult to simultaneously meet the requirements of positioning accuracy, communication efficiency, perception capabilities, and dynamic response.

[0004] Specifically, the limitations of traditional methods include:

[0005] 1. Limited perception range: The perception range of drones is usually limited by the type and performance of sensors. In complex environments, such as dense buildings and complex terrain, the perception range of drones may be difficult to cover all potential obstacles.

[0006] 2. Limited dynamic obstacle avoidance capability. Although the UAV has a certain dynamic obstacle avoidance capability, the computational complexity and real-time requirements of the obstacle avoidance algorithm may exceed the processing capability of the UAV in high-speed flight or complex environments, resulting in obstacle avoidance failure.

[0007] 3. High algorithm complexity: The formation control algorithm needs to handle the coordinated flight of multiple drones, and the algorithm complexity is relatively high. In scenarios with high real-time requirements, the algorithm's calculation speed and optimization level may be difficult to meet the requirements.

[0008] 4. Insufficient robustness. In complex environments, such as severe weather conditions such as strong winds and rain, the robustness of the formation control algorithm may be challenged, resulting in unstable or out-of-control UAV formation flight. Summary of the invention

[0009] In order to solve the above problems, this application proposes a multi-machine cooperative control method of UAVs based on neural network, including:

[0010] Based on the current drone application scenarios, determine the various formation modes required for drone formation, and determine the corresponding topological structure for each formation mode;

[0011] Based on the information flow direction between the nodes in the topological structure, generating a corresponding topological matrix;

[0012] Generate a graph convolutional neural network; the graph convolutional neural network includes at least a skeleton network, a graph convolution module, and an attention module;

[0013] Extracting multi-dimensional flight features corresponding to the UAV formation through the skeleton network;

[0014] Using the topological matrix as an adjacency matrix, and outputting corresponding feature vectors according to the multi-dimensional flight features and the adjacency matrix through the graph convolution module;

[0015] By means of the attention module, a corresponding attention feature is output according to the feature vector, and a feature matrix corresponding to the UAV formation is obtained according to the attention feature;

[0016] Based on the feature matrix, the coordinated control instructions corresponding to the UAV formation are output.

[0017] On the other hand, the present application also proposes a multi-machine cooperative control device for unmanned aerial vehicles based on a neural network, comprising:

[0018] at least one processor; and,

[0019] a memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the neural network-based drone multi-machine collaborative control method as described in the above example.

[0021] On the other hand, the present application also proposes a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as: the neural network-based UAV multi-machine collaborative control method described in the above example.

[0022] The neural network-based multi-machine cooperative control method proposed in this application can bring the following beneficial effects:

[0023] Through deep learning and analysis of the topological structure and dynamic characteristics of drone formations, graph convolutional neural networks can generate more accurate and efficient collaborative control instructions, thereby improving the perception range and improving the collaborative efficiency of the formation. Under complex environmental conditions, graph convolutional neural networks can capture and respond to various changes in real time to ensure that the formation can maintain a stable flight state. As a data-driven model, graph convolutional neural networks can continuously optimize and adapt to different formation sizes and mission requirements through training, and have strong scalability and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] Figure 1 This is a flowchart of a multi-machine cooperative control method of unmanned aerial vehicles based on a neural network in an embodiment of the present application;

[0026] Figure 2 A schematic diagram of a topological structure in one scenario in an embodiment of the present application;

[0027] Figure 3 This is a schematic diagram of the architecture of a graph convolutional neural network in one scenario in an embodiment of the present application;

[0028] Figure 4 Schematic diagram of a multi-machine collaborative control device for unmanned aerial vehicles based on a neural network in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0030] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0031] like Figure 1 As shown, the embodiment of the present application provides a method for cooperative control of multiple drones based on a neural network, comprising:

[0032] S101: Based on the current drone application scenario, various formation modes required for the drone formation are determined, and for each formation mode, a corresponding topological structure is determined.

[0033] Generally speaking, in different drone application scenarios, different formation modes are required to meet the needs. Each formation mode corresponds to a topological structure. For each topological structure, it is trained separately to obtain its own corresponding graph convolutional neural network, and finally multiple graph convolutional neural networks are obtained.

[0034] The application of UAV formation flying technology in military and civilian fields is becoming more and more extensive. For example, in the military field, it can be used for reconnaissance, attack, communication relay and other tasks. In the civilian field, it can be used for logistics distribution, agricultural plant protection, environmental monitoring, UAV performances and other tasks.

[0035] For a drone application scenario, it may be necessary to change the formation during the mission execution based on needs. At this time, it has multiple formation modes, and each formation mode corresponds to its own topological structure. When switching the formation mode, the topological structure and the corresponding graph convolutional neural network also need to be switched.

[0036] Each drone in a drone formation can be regarded as a node, and these nodes exchange information through wireless communication links. In the topology, nodes represent drones, and nodes are connected by directed edges. The direction of the edge is also the direction of information flow, which refers to the path and direction of information transmission from one node to another. In a drone formation, the direction of information flow is usually consistent with the communication direction, that is, information is transmitted from one drone node (sender) to another drone node (receiver).

[0037] For drone formations, mutual communication between nodes can determine their own positions. Drones receive location information from other drones and update their own position estimates, thereby positioning themselves more accurately. This mutual communication helps drones maintain accurate positioning in complex environments and avoid collisions and getting lost. It can also help maintain formations. By communicating with each other, drones can understand the location and speed of other drones in real time, thereby adjusting their own flight status to maintain formations and improve flight efficiency. It can also be used to complete tasks collaboratively. By communicating with each other, drones can share mission information, resource allocation, and collaborative strategies, etc., so as to complete tasks more efficiently. For example, when performing search and rescue missions, drones can collaboratively search target areas to improve search efficiency; when performing reconnaissance missions, drones can share reconnaissance information to enhance overall reconnaissance capabilities.

[0038] like Figure 2As shown, an exemplary topological structure is provided, in which the out-degree of node 1 is 0, which only receives information but does not send information, while the out-degree of node 5 is 4, which only sends information but does not receive information. Therefore, in this drone formation, the drone corresponding to node 5 is the most important, and the drone corresponding to node 1 has the least impact on the formation.

[0039] S102: Generate a corresponding topology matrix based on the information flow direction between nodes in the topology structure.

[0040] Number each node in advance to facilitate subsequent node management and control. Figure 2 For example, at this time, the node numbers include: node 1 to node 5.

[0041] Select each node in turn according to the order of the node numbers, and correspond each node to a single row parameter in the topology matrix. Figure 2 The topological structure in is selected from nodes 1 to 5 in sequence, where node 1 corresponds to the parameters of the first row in the topological matrix, and so on, node 5 corresponds to the parameters of the last row in the topological matrix.

[0042] In each row of parameters, select the nodes other than the current node in the order of the node numbers, and generate the corresponding parameters based on the direction of information flow between the current node and other nodes. Here, when the node receives information sent by other nodes, the parameter is 1, otherwise it is 0, so the parameters of the diagonal of the topology matrix are all 0. In the first row of parameters, corresponding to node 1, it receives information from the other four nodes (node ​​2 to node 5), so the remaining four parameters are all 1. At this point, the first row of the topology matrix is ​​[0,1,1,1,1]. Similarly, the parameters of each of the remaining rows can be obtained, and finally Figure 2 The topological matrix corresponding to the topological structure can be shown in Formula 1:

[0043] . Where A is the topological matrix.

[0044] S103: Generate a graph convolutional neural network; the graph convolutional neural network includes at least a skeleton network, a graph convolution module, and an attention module.

[0045] like Figure 3 As shown, the graph convolutional neural network (GCN) in this application mainly includes three parts: skeleton network, graph convolution module, and attention module. Each part plays its own corresponding function.

[0046] The parameters of the graph convolutional neural network are optimized and adjusted according to the key parameters of UAV flight. Parameters in multiple dimensions are selected as the input of the graph convolutional neural network, so as to output the corresponding feature matrix to characterize the parameters of the UAV formation, thereby facilitating the control of the UAV formation.

[0047] Specifically, the following flight information (also called multi-dimensional flight information) can be selected as the input of the graph convolutional neural network during the optimization process. The dimensions include: the speed, acceleration, altitude, heading angle, pitch angle, roll angle, wind speed, and obstacle position of the drone. Generally speaking, the multi-dimensional flight information of a single drone in the drone formation is calculated, so as to extract its corresponding multi-dimensional flight characteristics, which are combined with the multi-dimensional flight characteristics of other drones to achieve collaborative control of the drone formation.

[0048] Through model training, the multi-dimensional flight information is accurately matched and mapped with the parameters of the graph convolutional neural network, achieving a direct correspondence between the parameters of the graph convolutional neural network and the flight parameters of the drone, thereby ensuring that the model parameters can be updated in real time and accurately. This improves the efficiency and accuracy of the graph convolutional neural network when processing drone flight data. In the parameter matching process, the dynamic characteristics of drone flight and its high requirements for real-time performance are fully considered, ensuring that the graph convolutional neural network can quickly adapt to changes in the flight environment.

[0049] Among them, the training parameters can be obtained by collecting historical data of each drone formation and through preprocessing, labeling and other methods.

[0050] S104: Extracting multi-dimensional flight features corresponding to the UAV formation through the skeleton network.

[0051] The backbone network can take many forms, such as ResNet network, VGG network, etc. It is mainly used to match the multi-dimensional flight information of the drone with the input shape of the graph convolutional neural network.

[0052] Specifically, the multi-dimensional flight information is input into the skeleton network for feature extraction. The last layer of the skeleton network is a fully connected layer (Dense layer), which is referred to as the first fully connected layer here.

[0053] At this time, the multi-dimensional flight information corresponding to the input UAV formation is matched with the input shape of the graph convolution module through the first fully connected layer of the skeleton network.

[0054] The formula of the first fully connected layer is shown in Formula 2:

[0055] ;

[0056] x is the input multi-dimensional flight information, y is the output multi-dimensional flight characteristics, is the weight matrix, is the bias matrix.

[0057] For the first fully connected layer, the formula is relatively fixed. In order to ensure that the output of the multi-dimensional flight feature conforms to the corresponding shape (that is, the shape corresponding to the multi-dimensional flight feature), the weight matrix The matrix size is set to (n-1) × n, and the bias matrix The matrix size of is set to 1×n, and the matrix size of the input multi-dimensional flight information x is set to n×n; where n is the number of dimensions of the multi-dimensional flight information.

[0058] Taking the 8 dimensions in the above example as an example, at this time, n=8, the weight matrix The size of the bias matrix is ​​7×8. Size 8×1.

[0059] The weight matrix After multiplying with the multi-dimensional flight information x, and the flipped bias matrix The splicing is performed to obtain a splicing vector S; wherein the matrix size of the splicing vector S is n×n.

[0060] Still taking the 8 dimensions mentioned above as an example, the weight matrix The matrix obtained by multiplying the multi-dimensional flight information x is still 7×8, and the bias matrix after flipping is The size of is 1×8, and the size of the spliced ​​vector S obtained by splicing the two is 8×8, which matches the input multi-dimensional flight information and the input of the graph convolution module.

[0061] Divide the concatenated vector S into multiple sub-vectors , each sub-vector is used to correspond to one dimension of the multi-dimensional flight characteristics; among them, the sub-vector The matrix size is n×1.

[0062] Still taking the 8 dimensions mentioned above as an example, each sub-vector The matrix size is 8×1, and each sub-vector is used to correspond to one dimension of the drone's speed, acceleration, altitude, heading angle, pitch angle, roll angle, wind speed, and obstacle position information.

[0063] S105: Using the topological matrix as an adjacency matrix, and outputting corresponding feature vectors according to the multi-dimensional flight features and the adjacency matrix through the graph convolution module.

[0064] Specifically, each subvector Input to the graph convolution module, and output the corresponding sub-feature vectors , the sub-eigenvector Concatenate and get the corresponding feature vector T.

[0065] The formula of the graph convolution module is shown in Formula 3:

[0066] ;

[0067] in, Represented as the output of the graph convolution module, corresponding to the sub-feature vector , Represented as the input of the graph convolution module, corresponding to the sub-vector , is the adjacency matrix, and is the parameter of the graph convolution module, and ReLU is the activation function.

[0068] Among them, the parameters and parameters The formula is shown in Formula 4:

[0069] ;

[0070] in, For the The weight of the layer, For the The bias of the layer, is the adjacency matrix, is the activation function; Indicates When l = 0 or 1, it corresponds to or .

[0071] Taking the 8 dimensions mentioned above as an example, the sub-vector Input to the graph convolution module and output the sub-feature vector , concatenated to obtain a feature vector T of size 8×8, and then the feature vector T is input into the attention module for the next step of processing.

[0072] S106: Outputting corresponding attention features according to the feature vector through the attention module, and obtaining a feature matrix corresponding to the UAV formation according to the attention features.

[0073] Specifically, the formula of the attention module is shown in Formula 5:

[0074] ;

[0075] in, is the Sigmoid function, is a 5×5 convolution process, Indicates attention characteristics;

[0076] To perform channel splicing on two feature matrices of average pooling AvgPool and maximum pooling MaxPool;

[0077] The parameters of the two feature matrices are both H×W×C, where H is the feature matrix height, W is the feature matrix width, and C is the number of feature matrix channels; the two feature matrices are channel-joined, and the resulting joint feature parameters are H×W×2C.

[0078] S107: Based on the feature matrix, output the collaborative control instructions corresponding to the UAV formation.

[0079] After the attention module, the features are input to the second fully connected layer, and the output parameter of the second fully connected layer is set to 8. The output result is a feature matrix with a feature size of 8×1. Taking the multi-dimensional flight information in the above example, the output result can be directly mapped to the key flight parameters of the drone, including speed, acceleration, altitude, heading angle, pitch angle, roll angle, wind speed, and precise obstacle location information.

[0080] This information forms the basis for cooperative control decisions of drones. Relying on this information, the flight control system can generate a series of cooperative control instructions aimed at optimizing and regulating the flight path of each drone, flexibly adjusting its flight speed, and accurately controlling its flight attitude. These instructions are designed to ensure that the drone formation can maintain a high degree of coordination and consistency under any conditions, whether it is maintaining a tight formation or responding quickly to dynamic changes in the external environment. In this way, the drone formation can not only maintain efficient operation in a complex and changing environment, but also effectively avoid obstacles, ensure the continuity and safety of mission execution, and demonstrate a highly intelligent cooperative flight control capability.

[0081] Specifically, the collaborative control instructions adjust the drone's current speed, acceleration, altitude, heading angle, pitch angle, roll angle and other parameters according to the parameters output by the graph convolutional neural network to ensure flight stability.

[0082] When collaboratively controlling a formation of UAVs, different control methods can be selected based on actual needs. For example, it can be the leader-follower method or distributed collaborative control. This requires different modifications for different environments. The topology matrix is ​​mainly determined based on the number of UAVs in the formation and the direction of information flow between UAVs.

[0083] Among them, the pilot-follower method sets one or more drones as pilots and other drones as followers. The pilot is responsible for planning the flight path and speed, and the followers adjust the flight status according to the pilot's information. The distributed cooperative control method sets the behavior rules of drones to control their movements in a distributed manner to achieve the purpose of formation control.

[0084] Of course, it can also include virtual piloting, where there is no actual pilot drone, but a virtual pilot point is used to guide the formation flight. Based on the consistency control method, multiple drones are virtualized into a structure, and the drone formation is controlled by controlling the structure.

[0085] Through deep learning and analysis of the topological structure and dynamic characteristics of drone formations, graph convolutional neural networks can generate more accurate and efficient collaborative control instructions, thereby improving the perception range and improving the collaborative efficiency of the formation. Under complex environmental conditions, graph convolutional neural networks can capture and respond to various changes in real time to ensure that the formation can maintain a stable flight state. As a data-driven model, graph convolutional neural networks can continuously optimize and adapt to different formation sizes and mission requirements through training, and have strong scalability and adaptability.

[0086] In one embodiment, when determining the topology matrix, the number of drones corresponding to each formation mode can be determined, as well as the corresponding central nodes and edge nodes in the formation mode. A central node refers to a node that has other nodes in all directions around itself, while an edge node refers to a node that has other nodes only in a certain range of angle directions around itself.

[0087] If the number of drones is lower than the first preset threshold, it means that the number of drones in the current formation is small, and it is a small formation. At this time, selecting a few main drones to send information outward can simplify the communication structure, improve communication efficiency and reduce costs. Therefore, among the edge nodes, a preset number of nodes (usually 1 / 5~1 / 10 of the number of drone formations) are selected as pilot nodes, so that the out-degree value of the pilot node in the topological structure is the highest value.

[0088] As mentioned above, nodes with higher out-degree are relatively more important, and edge nodes are generally less subject to communication interference. Therefore, one or more nodes with the highest out-degree are selected from the edge nodes as pilot nodes to guide the drone formation.

[0089] If the number of drones is higher than the first preset threshold, it is considered that the number of the current drone formation is large, and the drones sending information to each other can provide better robustness and real-time performance. Therefore, the difference between the out-degree values ​​of all nodes in the formation mode is lower than the second preset threshold. At this time, the out-degree values ​​between the nodes are not much different, and they can communicate with each other to increase robustness.

[0090] In one embodiment, in the actual work of the application scenario, there may be a formation switching situation, at which time, the corresponding topological structure and graph convolutional neural network also need to be switched. During the switching process, due to the different parameters in different graph convolutional neural networks, the feature matrices output under the same state may be different, which may cause excessive adjustments between flight information and flight characteristics in certain dimensions at the moment of formation switching, which is not conducive to the stable control of the flight formation.

[0091] Based on this, it is determined that the formation mode will be switched within a preset time in the future (for example, 3 seconds), and the current first formation mode will be switched to the second formation mode. At this time, based on the first graph convolutional neural network corresponding to the first formation mode, and the second graph convolutional neural network corresponding to the second formation mode, the corresponding first feature matrix and second feature matrix are generated respectively.

[0092] If there are flight features of specified dimensions, such that the difference between the first feature matrix and the second feature matrix is ​​higher than the preset difference, for example, the speed dimension corresponds to an n×1 single-row parameter in the feature matrix, and the difference between the values ​​of this parameter in the first feature matrix and the second feature matrix is ​​large, it means that different graph convolutional neural networks have large differences in the analysis results of the speed features.

[0093] At this time, based on the preset weights corresponding to the flight characteristics of the specified dimension (generally speaking, different weights are set for different dimensions. For example, for state information such as speed, acceleration, altitude, heading angle, pitch angle, and roll angle, the current first feature matrix has a higher weight, while for external information such as wind speed and obstacle position, the weights of the two are the same, or the subsequent second feature matrix has a higher weight), the values ​​of the flight characteristics of the specified dimension in the first feature matrix and the second feature matrix are weighted and summed to obtain the final value of the specified dimension.

[0094] That is to say, for the specified dimensions with large differences, the possible errors are reduced by weighted summation.

[0095] As for the flight characteristics of the remaining dimensions, the difference between the first characteristic matrix and the second characteristic matrix is ​​small, and it can be considered that there is no error. Therefore, at this time, according to whether the current moment is before the formation mode is switched or during the formation mode switching process, the value in the first characteristic matrix or the second characteristic matrix can be selected as the final value. For example, if the current moment is before the formation mode is switched, the value of the currently used first characteristic matrix can be used. If the current moment is already in the formation mode switching process, the value in the corresponding second characteristic matrix can be used. At this time, based on the final value, the corresponding collaborative control instructions for the UAV formation are generated and output.

[0096] After the formation mode is switched, the first feature matrix is ​​no longer used. Only the second graph convolutional neural network is used to generate the second feature matrix to generate collaborative control instructions.

[0097] like Figure 4 As shown, the embodiment of the present application also proposes a multi-machine cooperative control device for unmanned aerial vehicles based on a neural network, including:

[0098] at least one processor; and,

[0099] a memory communicatively connected to the at least one processor; wherein,

[0100] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the neural network-based unmanned aerial vehicle multi-machine collaborative control method as described in any of the above embodiments.

[0101] The embodiment of the present application also proposes a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as: the neural network-based multi-machine collaborative control method for unmanned aerial vehicles described in any of the above embodiments.

[0102] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0103] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0104] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0105] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0106] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0107] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0108] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0109] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0110] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0111] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0112] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A multi-machine cooperative control method for unmanned aerial vehicles based on neural network, characterized in that: include: Based on the current drone application scenarios, determine the various formation modes required for drone formation, and determine the corresponding topological structure for each formation mode; Based on the information flow direction between the nodes in the topological structure, generating a corresponding topological matrix; Generate a graph convolutional neural network; the graph convolutional neural network includes at least a skeleton network, a graph convolution module, and an attention module; Extracting multi-dimensional flight features corresponding to the UAV formation through the skeleton network; Using the topological matrix as an adjacency matrix, and outputting corresponding feature vectors according to the multi-dimensional flight features and the adjacency matrix through the graph convolution module; By means of the attention module, a corresponding attention feature is output according to the feature vector, and a feature matrix corresponding to the UAV formation is obtained according to the attention feature; Based on the feature matrix, the coordinated control instructions corresponding to the UAV formation are output.

2. The method according to claim 1, characterized in that Based on the information flow direction between the nodes in the topological structure, a corresponding topological matrix is ​​generated, which specifically includes: Determine the node number corresponding to each node in the topological structure; Select each node in turn according to the order of numbers corresponding to the node numbers, and correspond each node to a single row parameter in the topology matrix; In each row of parameters, other nodes except the current node are selected in turn according to the numbering sequence corresponding to the node numbers, and corresponding parameters are generated based on the information flow direction between the current node and the other nodes.

3. The method according to claim 1, characterized in that: The multi-dimensional flight features corresponding to the UAV formation are extracted through the skeleton network, specifically including: Based on the first fully connected layer of the skeleton network, the multi-dimensional flight information corresponding to the input UAV formation is matched with the shape of the input of the graph convolution module; Among them, the formula of the first fully connected layer is: ; x is the input multi-dimensional flight information, y is the output multi-dimensional flight characteristics, is the weight matrix, is the bias matrix.

4. The method according to claim 3, characterized in that The multi-dimensional flight features corresponding to the UAV formation are extracted through the skeleton network, specifically including: The weight matrix The matrix size is set to (n-1) × n, and the bias matrix The matrix size of is set to 1×n, and the matrix size of the input multi-dimensional flight information x is set to n×n; wherein n is the number of dimensions of the multi-dimensional flight information; The weight matrix After multiplying with the multi-dimensional flight information x, and the flipped bias matrix Perform splicing to obtain a splicing vector S; wherein the matrix size of the splicing vector S is n×n; Divide the concatenated vector S into multiple sub-vectors , each sub-vector is used to correspond to one dimension of the multi-dimensional flight characteristics; among them, the sub-vector The matrix size is n×1.

5. The method according to claim 4, characterized in that Outputting corresponding feature vectors according to the multi-dimensional flight features and the adjacency matrix through the graph convolution module specifically includes: Each sub-vector Input to the graph convolution module, and output the corresponding sub-feature vectors , the sub-feature vector Splice to get the corresponding feature vector T; Among them, the formula of the graph convolution module is: ; in, Represented as the output of the graph convolution module, corresponding to the sub-feature vector , Represented as the input of the graph convolution module, corresponding to the sub-vector , is the adjacency matrix, ReLU is the activation function; Among them, the parameters The formula is: ; in, For the The weight of the layer, For the The bias of the layer, is the adjacency matrix, is the activation function; is the parameter of the graph convolution module, indicating the The node features of the layer, the initial value of l is 0, l = 0 or 1, corresponding to and .

6. The method according to claim 1, characterized in that By means of the attention module, the corresponding attention feature is output according to the feature vector, and the feature matrix corresponding to the UAV formation is obtained according to the attention feature, specifically including: The formula of the attention module is: ; in, is the Sigmoid function, is a 5×5 convolution process, Indicates attention characteristics; To perform channel splicing on two feature matrices of average pooling AvgPool and maximum pooling MaxPool; The parameters of the two feature matrices are both H×W×C, where H is the feature matrix height, W is the feature matrix width, and C is the number of feature matrix channels; the two feature matrices are channel-joined, and the resulting joint feature parameters are H×W×2C.

7. The method according to claim 1, characterized in that For each formation mode, determine its corresponding topological structure, including: For each formation mode, determine the number of drones corresponding to it, and determine the corresponding central nodes and edge nodes in the formation mode; If the number of the drones is lower than a first preset threshold, a preset number of nodes are selected from the edge nodes as pilot nodes, so that the out-degree value of the pilot node in the topological structure is the highest value; If the number of the drones is higher than a first preset threshold, the difference between the out-degree values ​​of all nodes in the formation is set to be lower than a second preset threshold.

8. The method according to claim 1, characterized in that After obtaining the feature matrix corresponding to the drone formation according to the attention feature, the method further includes: Determine to execute the switching of the formation mode within a preset time in the future, and switch the current first formation mode to the second formation mode; Based on a first graph convolutional neural network corresponding to the first formation mode and a second graph convolutional neural network corresponding to the second formation mode, respectively generating a first feature matrix and a second feature matrix corresponding to each other; If there is a flight feature of a specified dimension, such that the difference between the flight feature in the first feature matrix and the second feature matrix is ​​higher than a preset difference, based on a preset weight corresponding to the flight feature of the specified dimension, weighted sum is performed on the values ​​of the flight feature of the specified dimension in the first feature matrix and the second feature matrix to obtain a final value of the specified dimension; For the flight characteristics of the remaining dimensions, according to whether the current moment is before the formation mode is switched or during the formation mode is switched, the value in the first characteristic matrix or the second characteristic matrix is ​​selected as the final value; Based on the final value, a collaborative control instruction corresponding to the UAV formation is generated and output.

9. A multi-machine cooperative control device for unmanned aerial vehicles based on neural network, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the neural network-based unmanned aerial vehicle multi-machine collaborative control method as described in any one of claims 1 to 8.

10. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are set to: the neural network-based unmanned aerial vehicle multi-machine collaborative control method described in any one of claims 1 to 8.

Images