UAV swarm information fusion method

By cropping photoelectric reconnaissance images from the first node in the UAV swarm and extracting feature codes using a convolutional neural network, the images are transmitted to the second node for target clustering and information fusion. This solves the problem of low data transmission and information fusion efficiency in UAV swarms and achieves efficient target matching and information fusion.

CN119904719BActive Publication Date: 2026-11-17INFORMATION SCI RES INST OF CETC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411954149.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-12-27
Publication Date
2026-11-17
Estimated Expiration
2044-12-27

AI Technical Summary

Technical Problem

When there are many targets in a drone swarm, the method of transmitting images for target matching consumes a lot of network communication bandwidth, resulting in low data transmission efficiency and low efficiency of multi-drone information fusion.

Method used

The first UAV node in the UAV swarm acquires photoelectric reconnaissance images and performs cropping. A pre-trained convolutional neural network is used to extract feature codes. Only the feature codes are transmitted to the second UAV node for target clustering and information fusion. The Dempster synthesis rule is combined to fuse category and location information.

Benefits of technology

This significantly reduces the amount of information transmitted, decreases network communication bandwidth usage, improves the efficiency of data transmission and multi-machine information fusion, and enhances the accuracy of target matching and information fusion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119904719B_ABST
    Figure CN119904719B_ABST
Patent Text Reader

Abstract

This application discloses a method for information fusion in a drone swarm, comprising: a first drone node acquiring an electro-optical reconnaissance image of a target object, first category information, and first location information; cropping the electro-optical reconnaissance image based on the first location information to obtain a target image; inputting the target image into a pre-trained convolutional neural network to obtain a first feature code; sending the category information, location information, and feature code to a second drone node; the second drone node dividing a plurality of target objects into several target clusters based on the first location information and first feature code received from each of the first drone nodes, wherein the target objects in each target cluster are identical; when a target cluster includes at least two target objects, fusing the first category information and first location information corresponding to each of the at least two target objects to obtain target category information and target location information. This method can improve transmission efficiency and multi-drone information fusion efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application belongs to the field of unmanned aerial vehicle (UAV) technology, and in particular relates to a method for information fusion of UAV swarms. Background Technology

[0002] In recent years, with the rapid development of technology, all industries are undergoing digital and intelligent transformation. Continuous advancements in sensor technology, network communication technology, and data processing capabilities have significantly improved the performance of drones in information acquisition, transmission, and processing. Drone swarms, as an innovative application, have received widespread attention in fields such as agricultural monitoring, environmental monitoring, traffic management, and disaster relief. Through the collaborative work of multiple drones, they enable more efficient monitoring, data collection, and multi-faceted data analysis to meet the needs of modern industries.

[0003] Information fusion in drone swarms integrates data from different platforms and sensors, providing decision-makers with more valuable reference information through comprehensive and in-depth analysis. Currently, multi-drone information fusion is achieved by transmitting target images. However, when there are many targets, this method of target matching by transmitting images consumes a large amount of network communication bandwidth, resulting in low data transmission efficiency and low efficiency of multi-drone information fusion. Summary of the Invention

[0004] This application provides a method for information fusion in a drone swarm, which at least solves the problems of low data transmission efficiency and low efficiency of multi-drone information fusion in related technologies.

[0005] This application provides a method for information fusion in a drone swarm, wherein the drone swarm includes a plurality of first drone nodes and a second drone node;

[0006] The method includes:

[0007] The first UAV node acquires the photoelectric reconnaissance image of the target object, the first category information, and the first location information;

[0008] The first UAV node crops the photoelectric reconnaissance image based on the first location information to obtain the target image;

[0009] The first UAV node inputs the target image into a pre-trained convolutional neural network to obtain the first feature code corresponding to the target image;

[0010] The first drone node sends the first category information, the first location information, and the first feature code to the second drone node;

[0011] The second UAV node divides the corresponding target objects into several target clusters based on the first location information and first feature code sent by each of the first UAV nodes. The number of target clusters is less than or equal to the number of target objects, and the target objects in each target cluster are the same.

[0012] For each target cluster, if the target cluster includes one target object, the first category information and the first location information of the target object are determined as target category information and target location information, respectively; if the target cluster includes at least two target objects, the first category information and the first location information corresponding to each of the at least two target objects are fused to obtain target category information and target location information, respectively.

[0013] The UAV swarm information fusion method of this application embodiment uses neural networks to extract features from target images at each node UAV, and then only transmits the feature vector of the target to the fusion UAV node for target matching to achieve information fusion. This can greatly reduce the amount of information transmitted, reduce the bandwidth occupied by network communication transmission, improve transmission efficiency, and thus improve the efficiency of multi-aircraft information fusion. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart illustrating a method for information fusion in a drone swarm provided in an embodiment of this application;

[0016] Figure 2 This is an exemplary schematic diagram of cropping to a target image provided in the embodiments of this application;

[0017] Figure 3 This is a schematic diagram of training a convolutional neural network provided in an embodiment of this application;

[0018] Figure 4 This is an exemplary schematic diagram of obtaining the first feature encoding of a target image using a pre-trained convolutional neural network, provided by an embodiment of this application.

[0019] Figure 5 This is a flowchart illustrating another method for unmanned aerial vehicle (UAV) swarm information fusion provided in this application embodiment. Detailed Implementation

[0020] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0021] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0022] In recent years, with the rapid development of technology, all industries are undergoing digital and intelligent transformation. Continuous advancements in sensor technology, network communication technology, and data processing capabilities have significantly improved the performance of drones in information acquisition, transmission, and processing. Drone swarms, as an innovative application, have received widespread attention in fields such as agricultural monitoring, environmental monitoring, traffic management, and disaster relief. Through the collaborative work of multiple drones, they enable more efficient monitoring, data collection, and multi-faceted data analysis to meet the needs of modern industries.

[0023] Information fusion in drone swarms integrates data from different platforms and sensors, providing decision-makers with more valuable reference information through comprehensive and in-depth analysis. Currently, multi-drone information fusion is achieved by transmitting target images. However, when there are many targets, this method of target matching by transmitting images consumes a large amount of network communication bandwidth, resulting in low data transmission efficiency and low efficiency of multi-drone information fusion.

[0024] To address the problems in related technologies, this application provides a method for information fusion in unmanned aerial vehicle (UAV) swarms.

[0025] The UAV swarm information fusion method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0026] It should be noted that the drone cluster in this embodiment includes several first drone nodes and one second drone node. It can be understood that the second drone node is the fused drone node.

[0027] refer to Figure 1 This is a flowchart illustrating a method for information fusion in a drone swarm according to an embodiment of this application. Figure 1 As shown, the drone swarm information fusion method may specifically include the following steps:

[0028] S101, The first UAV node acquires the photoelectric reconnaissance image of the target object, the first category information and the first location information;

[0029] S102. The first UAV node crops the photoelectric reconnaissance image according to the first location information to obtain the target image;

[0030] S103. The first UAV node inputs the target image into a pre-trained convolutional neural network to obtain the first feature code corresponding to the target image;

[0031] S104. The first UAV node sends the first category information, the first location information and the first feature code to the second UAV node;

[0032] S105. The second UAV node divides the corresponding target objects into several target clusters according to the first location information and first feature code sent by each of the first UAV nodes. The number of target clusters is less than or equal to the number of target objects, and the target objects in each target cluster are the same.

[0033] S106. For each target cluster, if the target cluster includes one target object, the first category information and the first location information of the target object are determined as target category information and target location information, respectively; if the target cluster includes at least two target objects, the first category information and the first location information corresponding to each of the at least two target objects are fused to obtain target category information and target location information, respectively.

[0034] Therefore, by using neural networks to extract features from target images at each node UAV, and then transmitting only the target's feature vector to the fusion UAV node for target matching to achieve information fusion, the amount of information transmitted can be greatly reduced, the bandwidth occupied by network communication transmission can be reduced, and the transmission efficiency can be improved, thereby improving the efficiency of multi-aircraft information fusion.

[0035] The specific implementation methods for each of the above steps are described below.

[0036] In some embodiments, in S101, each first UAV node in the UAV cluster obtains the category and location information of the target object being detected, namely the first category information and the first location information, based on the photoelectric payload reconnaissance data and through the target identification and target positioning algorithm on the airborne end.

[0037] In practice, the drone pose information is acquired; the drone pose information is used to calculate the position to obtain the first position information of the target object.

[0038] In some embodiments, in S102, based on the first location information, a recognition algorithm is used to identify the target object in the photoelectric reconnaissance image to obtain a first region including the target object; a preset multiple of the first side length of the first region is determined as the target side length, and the center of the first region is determined as the target center; a preset shape region corresponding to the target side length and the target center is determined as the target region; and the region in the photoelectric reconnaissance image other than the target region is cropped to obtain the target image.

[0039] Figure 2 An exemplary diagram illustrating cropping to a target image is shown. Figure 2 As shown, the red box is the recognition bounding box of the recognition algorithm, corresponding to the first region; the green box is the cropping box, corresponding to the target region. If the length of the longer side of the red box (i.e., the first side length of the first region) is defined as L, then the target side length can be set to 3 / 2L. Optionally, the preset shape can be a square. Figure 2 The green box corresponding to the target area is a square coinciding with the center of the red box, with a side length of 3 / 2L. Thus, each first UAV node crops the electro-optical reconnaissance image to obtain the target image based on the detected target location and size information.

[0040] As an optional embodiment, the Convolutional Neural Network (CNN) is obtained by training a preset model, specifically, the preset model is a Triplet network, and the CNN is obtained by training the Triplet network based on Triplet Loss.

[0041] refer to Figure 3 This is a schematic diagram for training a convolutional neural network. (Example) Figure 3As shown, the Triplet network comprises three identical CNNs that share weights. The specific implementation of training the convolutional neural network may include: acquiring an image sample set, which includes several triples, each triple comprising a baseline sample, a positive sample, and a negative sample; inputting the triples into a preset model to obtain the sample feature codes corresponding to the baseline sample, positive sample, and negative sample, wherein the preset model is a Triplet network, which comprises three identical CNNs that share weights; calculating the loss value of the triples based on the sample feature codes corresponding to the baseline sample, positive sample, and negative sample; updating the network weights based on the loss values ​​and training the updated preset model to obtain the trained convolutional neural network.

[0042] Optionally, the loss value of the triplet is calculated using the following formula (1) based on the sample feature encodings corresponding to the benchmark sample, positive sample, and negative sample:

[0043] Loss=max(d(a,p)-d(a,n)+margin,0); (1)

[0044] Where Loss represents the loss value of the triple; d(a,p) represents the cosine distance between the feature codes of the benchmark sample and the positive sample; d(a,n) represents the cosine distance between the feature codes of the benchmark sample and the negative sample; margin represents a pre-set interval value used to control the distance between the positive and negative samples, which can be set through cross-validation results. It should be noted that in this embodiment, cosine distance is used to calculate the distance between the two feature codes.

[0045] In this way, by using Triplet Loss, the model can learn feature representations that make similar samples closer together and dissimilar samples farther apart, thereby improving the performance of the target feature representation.

[0046] In other words, during training, a triplet (comprising a baseline sample, a positive sample, and a negative sample) is randomly selected from the training sample set and input into the network. This yields the feature encodings for the three samples, and the network weights are updated by calculating the Triplet Loss of the triplet. By training a Triplet network for feature extraction from target images, the expressiveness of the features is enhanced, resulting in smaller distances between features of similar target samples and larger distances between features of different target samples. This significantly improves the accuracy of target matching in subsequent steps.

[0047] Thus, the trained convolutional neural network is obtained. Specifically, this convolutional neural network includes a base network ResNet18 and two fully connected layers connected in sequence. Therefore, after the convolutional neural network training is completed, step S103 is executed.

[0048] In practice, the target image is sampled to a preset size and then input into the base network ResNet18 of the convolutional neural network. The output of the average pooling layer is extracted to obtain the feature vector of the first preset dimension. Then, the first feature encoding of the second preset dimension is obtained through the output of two fully connected layers.

[0049] Figure 4 This diagram illustrates an exemplary method for obtaining the first feature encoding of a target image using a pre-trained convolutional neural network. For example... Figure 4 As shown, the target image is first uniformly sampled to a size of 64×64, and then input into the basic network ResNet18 model. The output of the average pooling layer is extracted to obtain a 512-dimensional feature vector. Then, through two fully connected layers, a 128-dimensional feature vector is obtained, which is used to represent the first feature encoding of the input target image.

[0050] Furthermore, in some optional embodiments, S105 can be implemented in the following ways:

[0051] A number of target objects are sorted in descending order according to their recognition confidence to obtain a target list; a first target cluster is created, the first target object in the target list is determined as the first member of the first target cluster, and the mean of the first position information and the mean of the first feature code of all members in the target cluster are respectively determined as the second position information and the second feature code of the target cluster;

[0052] Iteratively execute steps A to B until all target objects in the target list have been traversed, resulting in several target clusters:

[0053] Step A: For the next target object of the current target object in the target list, calculate the appearance distance between the target object and all existing target clusters based on the first feature code of the target object and the second feature code of the target cluster; calculate the spatial distance between the target object and all existing target clusters based on the first position information of the target object and the second position information of the target cluster.

[0054] Step B: If the appearance distance and spatial distance meet the preset conditions, calculate the similarity between the target object and the target cluster that meets the preset conditions; divide the target object into the target cluster with the highest similarity, and determine the mean of the first position information and the mean of the first feature code of all members in the updated target cluster as the second position information and the second feature code of the updated target cluster, respectively.

[0055] If the appearance distance and spatial distance do not meet the preset conditions, a new target cluster is created; the target object is assigned to the new target cluster; the first position information and the first feature code of the target object are respectively determined as the second position information and the second feature code of the new target cluster.

[0056] In practice, the apparent distance between the target object and the target cluster can be calculated according to the following formula (2):

[0057]

[0058] Among them, D 外观 f1 represents the appearance distance; f2 represents the first feature code of the target object; f3 represents the second feature code of the target cluster.

[0059] In specific implementation, the first location information includes first longitude information and first latitude information, and the second location information includes second longitude information and second latitude information. Then, the spatial distance between the target object and the target cluster can be calculated according to the following formula (3):

[0060]

[0061] Among them, D 空间 The values ​​represent spatial distance; x1 represents the first longitude information of the target object; x2 represents the second longitude information of the target cluster; y1 represents the first latitude information of the target object; and y2 represents the second latitude information of the target cluster.

[0062] In specific implementation, the preset conditions are that the visual distance is less than a first preset threshold and the spatial distance is less than a second preset threshold. Then, the similarity between the target object and the target cluster that meets the preset conditions can be calculated according to the following formula (4):

[0063]

[0064] Where S represents similarity; T 外观 T represents the first preset threshold; 空间 This indicates the second preset threshold.

[0065] In other words, target matching is performed at the fusion drone node. Assuming the fusion drone node collects a total of n targets, the target matching algorithm outputs m target clusters, where targets within each cluster are the same target. The detailed matching steps are as follows:

[0066] ① Sort all targets from highest to lowest according to their target recognition confidence, resulting in a target list, denoted as Obj = [Obj1, Obj2, Obj3, ..., Obj2]. n ], where n is the total number of targets;

[0067] ② Denote the first target cluster after target matching as Cluster1, take the first target Obj1 in the target list Obj as the first member of Cluster1, so that Cluster1=[Obj1], and use the mean of feature encodings and the mean of positions of all members in the cluster as the feature encoding and position information of this cluster;

[0068] ③ Traverse the next target in Obj, and calculate the appearance distance D between this target and all existing target clusters 外观 and spatial distance D 空间 ;

[0069] ④ If there exists a target cluster that satisfies D 外观 <T 外观 and D 空间 <T 空间 (where T 外观 is the set threshold of the appearance distance, T 空间 is the threshold of the spatial distance), then calculate the similarity S between the current target and the current target cluster; after the calculation is completed, sort all the target clusters that meet the conditions according to the S value, and the current target belongs to the target cluster corresponding to the maximum S value. Add the current target to this target cluster, and recalculate the mean of feature encodings and the mean of positions of all members in the target cluster to update the feature encoding and position information of the target cluster;

[0070] ⑤ If there is no cluster that satisfies D 外观 <T_appearance and D 空间 <T 空间 , create a new target cluster, add the current target to the newly created target cluster, and the feature encoding and position information of the newly created target cluster are the feature encoding and position information of the newly added target;

[0071] ⑥ Repeat steps ③ to ⑤ until all targets in Obj are traversed, and finally output [Cluster1,Cluster2,Cluster3...,Cluster m , where m is the total number of target clusters.

[0072] In this way, the target matching method based on the combination of spatial and appearance information realizes the matching of the same target object among several target objects, can improve the accuracy of target matching, and is of great help to improve the accuracy of information fusion in subsequent steps.

[0073] Further, in some embodiments, in S106, data fusion is performed on the first category information respectively corresponding to at least two target objects based on the Dempster's combination rule, and the fusion probability corresponding to each category after fusion is calculated: the category corresponding to the maximum value of the fusion probability is determined as the target category information.

[0074] Specifically, the target categories are fused using the DS evidence theory, and the detailed steps are as follows:

[0075] ① Determine the recognition framework Θ = {class1, class2, class3, ..., class...} n}, where n represents the types of targets to be identified;

[0076] ②The power of the recognition frame Θ

[0077] 2 Θ =

[0078] {φ,class1,class2,class3...,class n ,{class1,class2}...{class n-1 ,class n}...,Θ} is determined to be the set of all subsets of the recognition framework Θ, containing a total of 2 n One element;

[0079] ③ Determine the Basic Probability Assignment (BPA). The BPA is a 2-1 probability distribution. Θ → a function m in [0,1], satisfying m(φ)=0 and Assuming the target cluster contains *a* targets, it also contains *a* m functions, denoted as m1, m2, m3...m. i ...m a It is the probability distribution for target recognition, satisfying:

[0080] m i (class1)+m i (class2)+....+m i (class n =1, power 2 Θ The probability of any other element in the set is 0.

[0081] ④ Data fusion is performed based on Dempster's synthesis rules, and the fused probability for each category is calculated using the following formula:

[0082]

[0083] ⑤ After calculating the fusion probability of all categories, the target category is the category corresponding to the maximum fusion probability.

[0084] In another embodiment, the specific implementation of location information fusion may include:

[0085] Iteratively execute steps C to E until the number of target objects in the target cluster remains constant, and then determine the average of the first position information corresponding to each target object in the final target cluster as the target position information:

[0086] Step C: Calculate the mean value of the position information of the first position information corresponding to all target objects in the target cluster, i.e. Where 'a' represents the number of targets within the target cluster;

[0087] Step D: According to the formula Calculate the distance between each target object in the target cluster and the mean value (mean point) of the location information;

[0088] Step E: If the distance between the target object and the average location information is greater than a third preset threshold, the target object is determined to be an anomaly, and the corresponding target object is removed from the target cluster.

[0089] In other words, after removing the target object corresponding to the outlier from the target cluster each time, the mean value of the position information of the remaining targets within the target cluster is recalculated. Where b is the number of remaining targets in the target cluster; then, the distance between all remaining targets in the target cluster and the mean point is calculated. If the distance is greater than the set threshold (third preset threshold), the target is judged as an anomaly and removed from the target cluster; repeat the above steps until the number of targets in the target cluster remains unchanged, and then the mean of the position information of the remaining targets in the final target cluster is used as the fused mean position.

[0090] Therefore, by executing S106, after obtaining the target clusters, information fusion is performed sequentially within each target cluster to obtain the fused target information. It should be understood that if there is only one target within a target cluster, fusion is not required, and the information of that target is directly used as the fused information. Information fusion includes target category fusion and location information fusion.

[0091] also, Figure 5 A flowchart illustrating another method for unmanned aerial vehicle (UAV) swarm information fusion according to an embodiment of this application is shown. Figure 5As shown, another method for information fusion in a drone swarm may specifically include the following steps: all drone nodes in the swarm obtain the target's category and location information through electro-optical payload reconnaissance; each drone node crops the target image based on the target location information and inputs the cropped target image into a pre-trained convolutional neural network for feature extraction to obtain the target's feature representation; each drone node transmits the target's category, location, and feature representation to the fusion drone node through a communication network; the fusion drone node performs target association based on the target information (location and feature representation) of all drone nodes using a matching algorithm; the associated targets are fused using the DS evidence theory algorithm based on the target category information of all drone nodes to obtain fused target category information; the associated targets are fused using the mean algorithm after removing outliers based on the target location information of all drone nodes to obtain fused target location information. Thus, the final fused target information is obtained.

[0092] As can be seen, the low-data-transmission-volume UAV swarm information fusion method of this application, based on the feature-encoded target information transmission method, has a small target information transmission volume and high efficiency. In other words, the information fusion method that uses target feature encoding for target matching can improve data transmission efficiency, occupy less communication bandwidth, and improve the efficiency of multi-UAV information fusion. Furthermore, the target matching method combining appearance and spatial features has stronger target feature expressiveness and can improve target matching accuracy, ensuring higher accuracy in information fusion. This solves the problems of large data transmission volume and insufficient target matching accuracy in current UAV swarm fusion technologies. It realizes the use of multi-UAV information fusion technology to improve the efficiency and accuracy of UAV swarms in the information acquisition process, which is of great significance for improving the overall system performance, meeting the growing industry demands, and coping with complex and ever-changing application scenarios.

[0093] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0094] It should also be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0095] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0096] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for information fusion in unmanned aerial vehicle (UAV) swarms, characterized in that, The drone cluster includes several first drone nodes and one second drone node; The method includes: The first UAV node acquires the photoelectric reconnaissance image of the target object, the first category information, and the first location information; The first UAV node crops the photoelectric reconnaissance image based on the first location information to obtain the target image; The first UAV node inputs the target image into a pre-trained convolutional neural network to obtain the first feature code corresponding to the target image; The first drone node sends the first category information, the first location information, and the first feature code to the second drone node; The second UAV node divides the corresponding target objects into several target clusters based on the first location information and first feature code sent by each of the first UAV nodes. The number of target clusters is less than or equal to the number of target objects, and the target objects in each target cluster are the same. For each target cluster, if the target cluster includes one target object, the first category information and the first location information of the target object are determined as target category information and target location information, respectively; if the target cluster includes at least two target objects, the first category information and the first location information corresponding to each of the at least two target objects are fused to obtain target category information and target location information, respectively. The second UAV node, based on the first location information and first feature code received from each of the first UAV nodes, divides the corresponding target objects into several target clusters, including: A list of targets is obtained by sorting several target objects in descending order of recognition confidence. Create a first target cluster, determine the first target object in the target list as the first member of the first target cluster, and determine the mean of the first position information and the mean of the first feature code of all members in the target cluster as the second position information and the second feature code of the target cluster, respectively. Iteratively execute steps A to B until all target objects in the target list have been traversed, resulting in several target clusters: Step A: For the next target object of the current target object in the target list, calculate the appearance distance between the target object and all existing target clusters based on the first feature code of the target object and the second feature code of the target cluster; calculate the spatial distance between the target object and all existing target clusters based on the first position information of the target object and the second position information of the target cluster. Step B: If the appearance distance and spatial distance meet the preset conditions, calculate the similarity between the target object and the target cluster that meets the preset conditions; divide the target object into the target cluster with the highest similarity, and determine the mean of the first position information and the mean of the first feature code of all members in the updated target cluster as the second position information and the second feature code of the updated target cluster, respectively. If the appearance distance and spatial distance do not meet the preset conditions, a new target cluster is created; the target object is assigned to the new target cluster; the first position information and the first feature code of the target object are respectively determined as the second position information and the second feature code of the new target cluster.

2. The method according to claim 1, characterized in that, The pre-trained convolutional neural network comprises a base network ResNet18 and two fully connected layers connected in sequence; The first UAV node inputs the target image into a pre-trained convolutional neural network to obtain a first feature code corresponding to the target image, including: The target image is sampled to a preset size and then input into the basic network ResNet18. The output of the average pooling layer is extracted to obtain the feature vector of the first preset dimension. Then, the first feature encoding of the second preset dimension is obtained through the output of two fully connected layers.

3. The method according to claim 1, characterized in that, Before the first UAV node inputs the target image into a pre-trained convolutional neural network to obtain the first feature code corresponding to the target image, the method further includes: Obtain an image sample set, which includes several triples, each triple including a reference sample, a positive sample, and a negative sample; The triples are input into a preset model to obtain the sample feature codes corresponding to the baseline sample, positive sample and negative sample respectively. The preset model is a Triplet network, which includes three identical CNNs and the CNNs share weights. The loss value of the triple is calculated based on the sample feature codes corresponding to the benchmark sample, positive sample, and negative sample. The network weights are updated based on the loss value, and the updated preset model is trained to obtain the trained convolutional neural network.

4. The method according to claim 3, characterized in that, The step of calculating the loss value of the triplet based on the sample feature encodings corresponding to the benchmark sample, positive sample, and negative sample includes: Based on the sample feature codes corresponding to the benchmark sample, positive sample, and negative sample, the loss value of the triplet is calculated using the following formula: ; in, d(a,p) represents the loss value of the triple; d(a,p) represents the cosine distance between the sample feature encoding of the benchmark sample and the sample feature encoding of the positive sample; d(a,n) represents the cosine distance between the sample feature encoding of the benchmark sample and the sample feature encoding of the negative sample; margin represents the preset interval value used to control the distance between the positive and negative samples.

5. The method according to claim 1, characterized in that, The preset conditions are that the apparent distance is less than a first preset threshold and the spatial distance is less than a second preset threshold. The first location information includes first longitude information and first latitude information, and the second location information includes second longitude information and second latitude information. The step of calculating the appearance distance between the target object and all existing target clusters based on the first feature code of the target object and the second feature code of the target cluster includes: The apparent distance between the target object and the target cluster is calculated using the following formula: ; in, f1 represents the appearance distance; f2 represents the first feature encoding of the target object; f3 represents the second feature encoding of the target cluster. The step of calculating the spatial distance between the target object and all existing target clusters based on the first location information of the target object and the second location information of the target cluster includes: The spatial distance between the target object and the target cluster is calculated using the following formula: ; in, Indicates spatial distance; Indicates the first longitude information of the target object; This represents the second longitude information of the target cluster; This represents the first dimension information of the target object; This represents the second dimension information of the target cluster; The step of calculating the similarity between the target object and the target cluster that meets the preset conditions, when the appearance distance and spatial distance meet the preset conditions, includes: The similarity between the target object and the target cluster is calculated using the following formula: ; Where S represents similarity; This indicates the first preset threshold. This indicates the second preset threshold.

6. The method according to claim 1, characterized in that, When the target cluster includes at least two target objects, the first category information and the first location information corresponding to each of the at least two target objects are fused to obtain target category information and target location information, including: Based on the Dempster synthesis rule, data fusion is performed on the first category information corresponding to at least two target objects, and the fusion probability corresponding to each category after fusion is calculated: The category corresponding to the maximum fusion probability is determined as the target category information.

7. The method according to claim 1, characterized in that, When the target cluster includes at least two target objects, the first category information and the first location information corresponding to each of the at least two target objects are fused to obtain target category information and target location information, including: Iteratively execute steps C to E until the number of target objects in the target cluster remains constant, and then determine the average of the first position information corresponding to each target object in the final target cluster as the target position information: Step C: Calculate the mean value of the position information of the first position information of each target object in the target cluster; Step D: Calculate the distance between each target object in the target cluster and the average location information; Step E: If the distance between the target object and the average location information is greater than a third preset threshold, remove the corresponding target object from the target cluster.

8. The method according to claim 1, characterized in that, The first UAV node crops the photoelectric reconnaissance image based on the first location information to obtain a target image, including: Based on the first location information, a recognition algorithm is used to identify the target object in the photoelectric reconnaissance image, and a first region including the target object is obtained; The target side length is determined by a preset multiple of the first side length of the first region, and the center of the first region is determined as the target center. The preset shape region corresponding to the target side length and the target center is defined as the target region; The target image is obtained by cropping out the area other than the target region from the photoelectric reconnaissance image.

9. The method according to claim 1, characterized in that, The first drone node acquires the first location information of the target object, including: Obtain UAV pose information; The position information of the UAV is calculated to obtain the first position information of the target object.

Citation Information

Patent Citations

  • Target identification method and device based on image latitude and longitude feature fusion

    CN117935086A

  • Target recognition method and system for micro uav

    GB202410898D0