A lightweight multi-view collaborative image recognition method for unmanned swarm
By adopting a lightweight multi-view collaborative image recognition method in the drone cluster, the hierarchical structure of leaf nodes and head nodes and graph convolutional networks are used to solve the problem of insufficient computing resources for multi-view image recognition in the drone cluster. , improves identification accuracy and efficiency.
Patent Information
- Application Number
- CN202410414569.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-08
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-04-08
AI Technical Summary
The prior art is difficult to effectively realize multi-view image recognition in drone clusters, which is mainly due to the limitations of computing resources and complex algorithms, resulting in insufficient recognition accuracy and efficiency.
The lightweight multi-view co-image recognition method is adopted to construct a multi-viewing angle recognition unmanned cluster system. The leaf node drone collects single-viewing angle images and extracts features, transmits them to the head node drone for correlation aggregation, and uses the improved ResNet18 The model and graph convolution network perform feature fusion to reduce computing and communication resources.
It achieves the reduction of computing resource requirements while improving the accuracy and efficiency of target identification, significantly reducing the missed detection and false alarm rates, and is suitable for complex environments of drone clusters.
Smart Images

Figure CN118247639B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of unmanned cluster collaborative target detection, and specifically relates to a lightweight multi-view collaborative image recognition method for unmanned clusters. Background Art
[0002] With the development and popularization of drone technology, drone swarms have become a field of great concern. In this field, efficient and accurate target recognition using multi-view images acquired by multiple drones has always been an important challenge. For some application scenarios, such as military reconnaissance, border monitoring, and disaster relief, the use of multi-view image recognition methods can significantly improve accuracy and help reduce missed detections and false alarms.
[0003] Currently in the field of deep learning, scholars usually use voxel-based, point-set-based, and view-based methods for 3D object recognition. Voxel-based methods represent objects as 3D grids and use 3D networks for analysis. Point-set-based methods represent objects as unordered point sets and use point clouds for prediction. These two methods are also called model-based methods, which use 3D convolutional filters to convolve 3D shapes and directly generate 3D representations. However, due to computational cost constraints, their practical applications are somewhat limited. In contrast, view-based methods represent 3D objects as multiple 2D views and convolve these views using 2D convolutional filters. This method does not rely on complex 3D features, is easy to capture input views, and can use mature and advanced network frameworks. In the case of occlusion, views from different perspectives can complement the details of object features, resulting in excellent recognition performance.
[0004] At present, no scholars have applied these 3D recognition methods to the field of drone swarms, mainly due to the high computing resources and complex algorithms required for multi-view target recognition. This is obviously not suitable for our goal of applying multi-view image recognition methods to drone swarms. Summary of the invention
[0005] The technical problem to be solved by the present invention is to provide a lightweight multi-view collaborative image recognition method for an unmanned cluster, which can achieve efficient and accurate target recognition while reducing the demand for computing resources.
[0006] To solve the above technical problems, the embodiments of the present invention adopt the following technical solutions:
[0007] The embodiment of the present invention provides a lightweight multi-view collaborative image recognition method for an unmanned cluster, comprising the following steps:
[0008] Step 10, constructing a multi-view recognition unmanned cluster system, wherein the multi-view recognition unmanned cluster system includes a head node UAV and multiple leaf node UAVs, and the multiple leaf node UAVs all communicate with the head node UAV; the multiple leaf node UAVs are distributed in different directions of the recognition area;
[0009] Step 20, the leaf node drone collects a single-view image of the identified object, extracts single-view image features, and transmits the single-view image features to the head node drone;
[0010] In step 30, the head node drone associates and aggregates the single-view image features sent by all leaf node drones to obtain multi-view image features; and uses the multi-view image features to perform recognition to obtain recognition results.
[0011] As a further improvement of the embodiment of the present invention, in step 20, extracting single-view image features specifically includes:
[0012] The single-view image is input into the improved ResNet18 model, and the features before the last fully connected layer are extracted as the single-view image features.
[0013] As a further improvement of an embodiment of the present invention, the improved ResNet18 model adopts separable convolution.
[0014] As a further improvement of the embodiment of the present invention, in step 30, the head node drone associates and aggregates the single-view image features sent by all leaf node drones to obtain multi-view image features, specifically including:
[0015] Step 301, constructing an initial multi-view based on the single-view image features of all leaf node drones;
[0016] Step 302, based on the adjacency relationship between the leaf node drones in the initial multi-view, update the single-view image features of each leaf node drone to obtain the initial updated image features of the leaf node drone;
[0017] Step 303, based on the remote relationship between the leaf node drones in the initial multi-view, the relationship features between the initial image features of the leaf node drones are obtained, and then the initial image features of the leaf node drones are fused to obtain the fused image features;
[0018] Step 304, coarsening the graph constructed by fusion image features to obtain multi-view image features.
[0019] As a further improvement of the embodiment of the present invention, step 302 specifically includes:
[0020] The K nearest neighbor algorithm is used to find the k nearest leaf node drones of each leaf node drone, and a sparse adjacency matrix A∈M×M is obtained, where M represents the total number of leaf node drones.
[0021]
[0022] In the formula, A i,j represents the connection relationship between leaf node drone i and leaf node drone j, n i,1 ,…,n i,k represents the set of k nearest neighboring node IDs of leaf node drone i, n j,1 ,…,n j,k represents the set of k nearest neighboring node IDs of drone j;
[0023] The sparse adjacency matrix A is used to update the single-view image features of each leaf node drone using formula (2) to obtain the initial image features of the leaf node drone:
[0024]
[0025] In the formula, F′ i represents the initial image features of leaf node drone i, F j represents the single-view image features of the leaf node drone j, σ represents the activation function, and W represents the weight matrix.
[0026] As a further improvement of the embodiment of the present invention, step 303 specifically includes:
[0027] The initial image features of each leaf node drone are spliced together, and the relationship features of each leaf node drone are obtained through the output of the association module;
[0028] The relationship features of the leaf node drones are spliced with the initial image features of the leaf node drones, and the fused image features are obtained through the output of the fusion module.
[0029] As a further improvement of the embodiment of the present invention, in step 304, the farthest point sampling algorithm is used to coarsen the graph constructed by fusion of image features.
[0030] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:
[0031] The present invention provides a lightweight multi-perspective collaborative image recognition method for an unmanned cluster. A plurality of leaf-node UAVs are used to collect single-perspective images of an identification object at different orientations. Each leaf-node UAV uses a lightweight model to extract single-perspective image features to reduce computing resources. Each leaf-node UAV transmits the single-perspective image features to a node UAV instead of sending the single-perspective image to a head-node UAV, thereby reducing communication resources within the unmanned cluster and lowering communication costs. The head-node UAV correlates and fuses the single-perspective image features sent by the leaf-node UAVs to obtain multi-perspective image features, thereby using the multi-perspective image features for identification and improving the identification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0033] Figure 1 is a flow chart of a lightweight multi-view collaborative image recognition method for an unmanned cluster provided in this embodiment;
[0034] Figure 2 is a structural diagram of the multi-view recognition unmanned cluster system constructed in the method of this embodiment;
[0035] Figure 3 yes Figure 2 Processing flow chart of the mid-leaf node drone;
[0036] Figure 4 yes Figure 2 Processing flow chart of the mid-head node drone. DETAILED DESCRIPTION
[0037] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.
[0038] The embodiment of the present invention provides a lightweight multi-view collaborative image recognition method for an unmanned cluster, such as Figure 1 As shown, the following steps are included:
[0039] Step 10: Build a multi-view recognition unmanned cluster system, such as Figure 2 As shown, the multi-view recognition unmanned cluster system includes a head node drone and multiple leaf node drones, and the multiple leaf node drones all communicate with the head node drone. The multiple leaf node drones are distributed in different directions of the recognition area.
[0040] Step 20: The leaf node drone collects a single-view image of the identified object, extracts single-view image features, and transmits the single-view image features to the head node drone. Figure 3 shown.
[0041] In step 30, the head node drone associates and aggregates the single-view image features sent by all leaf node drones to obtain multi-view image features; and uses the multi-view image features to perform recognition to obtain recognition results.
[0042] As a preferred example, in step 20, extracting single-view image features specifically includes:
[0043] The single-view image is input into the improved ResNet18 model, and the features before the last fully connected layer are extracted as the single-view image features. Preferably, the improved ResNet18 model adopts separable convolution.
[0044] In the above embodiment, the ResNet18 model can achieve relatively better accuracy and smaller model size compared with other convolutional neural network (CNN) models. The convolution in the ResNet18 model is replaced with separable convolution to obtain an improved ResNet18 model. Among them, the separable convolution realizes lightweight convolution operation by decomposing the traditional convolution operation into deep convolution and point-by-point convolution. The deep convolution operates on each channel of the input tensor separately, and the point-by-point convolution uses a 1×1 kernel to convolve the feature map obtained from the deep convolution to adjust the number of channels in the feature map. The separable convolution operation significantly reduces the parameters and the amount of calculation, making the model lightweight. The two-dimensional images in the initial data set are trained by image classification, and the single-view image is input into the improved ResNet18 model, and the features before the last fully connected layer are extracted as the single-view image features.
[0045] As a preferred example, in step 30, the head node drone associates and aggregates the single-view image features sent by all leaf node drones to obtain multi-view image features, such as Figure 4 As shown, specifically including:
[0046] Step 301: construct an initial multi-view image G based on the single-view image features of all leaf node drones l The initial multi-view G l Contains M leaf nodes and a single-view feature F for each leaf node i .
[0047] Step 302: based on the adjacency relationship between the leaf node drones in the initial multi-view, update the single-view image features of each leaf node drone to obtain the initial updated image features of the leaf node drone.
[0048] Step 303, based on the remote relationship between the leaf node drones in the initial multi-view, the relationship features between the initial image features of the leaf node drones are obtained, and then the initial image features of the leaf node drones are fused to obtain the fused image features.
[0049] Step 304, coarsening the graph constructed by fusion image features to obtain multi-view image features.
[0050] Preferably, step 302 specifically includes:
[0051] According to the coordinates v of each leaf node drone i , the K nearest neighbor algorithm is used to find the k nearest leaf node drones of each leaf node drone, and a sparse adjacency matrix A∈M×M is obtained, where M represents the total number of leaf node drones;
[0052]
[0053] In the formula, A i,j represents the connection relationship between leaf node drone i and leaf node drone j, n i,1 ,…,n i,k represents the set of k nearest neighboring node IDs of leaf node drone i, n j,1 ,…,n j,k Represents the set of k nearest neighboring node IDs of drone j.
[0054] The sparse adjacency matrix A is used to update the single-view image features of each leaf node drone using formula (2) to obtain the initial image features of the leaf node drone:
[0055]
[0056] In the formula, F′ i represents the initial image features of leaf node drone i, F j represents the single-view image features of the leaf node drone j, σ represents the activation function, and W represents the weight matrix, which is used to map the node features to a new feature space.
[0057] The embodiment of the present invention updates the features of each leaf node based on the adjacency relationship. By aggregating the features of adjacent nodes, the node can obtain information from surrounding nodes, thereby enriching its own feature representation. This information transfer and aggregation process can help nodes better understand their position and role in the graph. When aggregating the features of adjacent nodes, by setting the weight matrix, the features of the closer leaf nodes have larger weights, and the closer the leaf nodes are, the more important the information is during aggregation. Using a sparse matrix to represent the adjacency relationship during updating can greatly reduce the memory usage and computational complexity of the adjacency matrix, thereby improving the efficiency of the local graph convolution layer.
[0058] Preferably, step 303 specifically includes:
[0059] Step 3031, stitching the initial image features of the leaf node drones in pairs.
[0060] Specifically, the initial image feature F′1 of the leaf node drone 1 is copied M times, the initial image feature F′1 of the leaf node drone 1 is spliced with the initial image feature F′1 of the leaf node drone 1, the initial image feature F′1 of the leaf node drone 1 is spliced with the initial image feature F′2 of the leaf node drone 2, ..., the initial image feature F′1 of the leaf node drone 1 is spliced with the initial image feature F′2 of the leaf node drone M. M Splicing; copy the initial image feature F′2 of leaf node drone 2 M times, splice the initial image feature F′2 of leaf node drone 2 with the initial image feature F′1 of leaf node drone 1, splice the initial image feature F′2 of leaf node drone 2 with the initial image feature F′2 of leaf node drone 2, ..., splice the initial image feature F′2 of leaf node drone 2 with the initial image feature F′ of leaf node drone M M Splicing; and so on.
[0061] For example, there are three leaf node drones. The initial image feature F′1 of leaf node drone 1 is copied three times, and the initial image feature F′1 of leaf node drone 1 is spliced with the initial image feature F′1 of leaf node drone 1, and the initial image feature F′1 of leaf node drone 1 is spliced with the initial image feature F′2 of leaf node drone 2, and the initial image feature F′1 of leaf node drone 1 is spliced with the initial image feature F′3 of leaf node drone 3; the initial image feature F′2 of leaf node drone 2 is copied three times, and the initial image feature F′2 of leaf node drone 2 is spliced with the initial image feature F′1 of leaf node drone 1, and the initial image feature F′2 of leaf node drone 2 is spliced with the initial image feature F′1 of leaf node drone 1. The initial image feature F′2 of leaf node drone 2 is spliced with the initial image feature F′2 of leaf node drone 2, and the initial image feature F′2 of leaf node drone 2 is spliced with the initial image feature F′3 of leaf node drone 3; the initial image feature F′3 of leaf node drone 3 is copied 3 times, the initial image feature F′3 of leaf node drone 3 is spliced with the initial image feature F′1 of leaf node drone 1, the initial image feature F′3 of leaf node drone 3 is spliced with the initial image feature F′2 of leaf node drone 2, and the initial image feature F′3 of leaf node drone 3 is spliced with the initial image feature F′3 of leaf node drone 3.
[0062] The concatenated features of the two initial image features are input into the association module to obtain the relationship features between the two leaf node drones. Among them, the relationship module is used to learn the relationship between the initial image features of the two leaf node drones and fuse their features. The association module consists of a linear layer and an activation function, receives an input of dimension 2*n, and outputs a tensor of dimension n after a series of linear transformations and activation functions. In addition, the convolution in the linear layer is replaced by a separable convolution to reduce the size of the module parameters.
[0063] For example, there are 3 leaf node drones, and the relationship features between the initial image features of node drone 1 and the initial image features of node drone 1, the relationship features between the initial image features of node drone 1 and the initial image features of node drone 2, the relationship features between the initial image features of node drone 1 and the initial image features of node drone 3, the relationship features between the initial image features of node drone 2 and the initial image features of node drone 1, the relationship features between the initial image features of node drone 2 and the initial image features of node drone 2, the relationship features between the initial image features of node drone 2 and the initial image features of node drone 3, the relationship features between the initial image features of node drone 3 and the initial image features of node drone 1, the relationship features between the initial image features of node drone 3 and the initial image features of node drone 2, and the relationship features between the initial image features of node drone 3 and the initial image features of node drone 3.
[0064] Step 3032, the relationship features of the leaf node drones are spliced with the initial image features of each leaf node drone, and the fused image features are obtained through the fusion module output. Among them, the fusion module is used to fuse the learned relationship features with the single-view image features of the leaf node drones, so that they can better interact and jointly affect the final output. The fusion module consists of a linear layer, a batch normalization layer and an activation function, receives an input with a dimension of 2*n, and outputs a vector with a dimension of n after processing by linear transformation, batch normalization and activation function. Moreover, the convolution in the linear layer is replaced by a separable convolution to reduce the size of the module parameters.
[0065] Preferably, in step 304, a farthest point sampling algorithm is used to coarsen the graph constructed by fusion of image features.
[0066] Specifically, the new graph after coarsening is obtained using formula (3):
[0067] F new = FPS(F b , K) Formula (3)
[0068] In the formula, F new represents the multi-view image features, Fb represents the fused image feature obtained in step 303, and K represents the number of points after the preset coarsening.
[0069] The basic idea of the FPS algorithm is to select a starting point from the graph, and then gradually select the point farthest from the selected point until the required number of sampling points is reached. Specifically: In the first step, select a starting point as the first sampling point; in the second step, calculate the distance between each point and the selected point, and select the point with the farthest distance as the next sampling point; in the third step, repeat the second step until a sufficient number of sampling points are selected.
[0070] The farthest point sampling (FPS) algorithm is used to coarsen the graph constructed by fusion image features, reduce the number of nodes and edges in the graph, reduce computing resources, and maintain the overall structure and relationship of the graph as much as possible. By selecting the farthest point as the sampling point, different areas can be evenly covered and key features and information can be retained.
[0071] In order to verify the performance of the lightweight multi-view collaborative image recognition method for unmanned clusters provided by an embodiment of the present invention, an evaluation was performed on the ModelNet40 dataset and compared with other multi-view methods. ModelNet40 is a large 3D object classification dataset consisting of 40 categories, each category has about 550 models, and a total of about 22,000 models. Each model is provided in .obj format and contains a training set and a test set. The three evaluation indicators of model parameters, category accuracy and overall accuracy are adopted: model parameters refer to the number of learnable parameters in the model. Generally, the more model parameters, the higher the model complexity, but too many model parameters will also increase the risk of overfitting. Therefore, under the same model structure and data set, the fewer model parameters, the better. Category accuracy refers to the classification accuracy of the model for each category in the classification task. A higher category accuracy indicates that the model has better classification ability in different categories. The overall accuracy refers to the overall classification accuracy of the model on the test data set. A higher overall accuracy indicates that the model has better classification ability. Cross entropy is selected as the loss function, which is widely used in classification problems. In the classification problem, we need to classify the input samples into different categories. The cross entropy measures the difference between the probability distribution predicted by the model and the probability distribution of the true label to evaluate the accuracy of the model. The evaluation results are shown in Table 1.
[0072] Table 1 Comparison results of multi-view recognition methods
[0073]
[0074] As can be seen from Table 1, compared with other multi-view methods, the method of the present invention has achieved significant success in performance. Compared with models such as MVCNN-New (multi-view convolutional neural network), PointNet (point cloud network), RetationNet (relational network), View-GCN (multi-view graph convolutional network) G-CNN (gated convolutional neural network), and DRCNN (dynamic routing convolutional neural network), the method of the present invention achieved an accuracy of 96.96%, which is higher than the model with smaller model parameters. The accuracy of the method of the present invention is only 0.64% lower than that of View-GCN, but the model scale of the method of the present invention is 73.83% smaller than that of View-GCN, and the required computing resources are significantly reduced. The accuracy of the method of the present invention is only 0.41% lower than that of RotationNet, but the model scale of the method of the present invention is 67.3% smaller than that of RotationNet, and the required computing resources are significantly reduced.
[0075] Compared with the traditional view-based multi-view recognition method, the method of the present invention has several advantages in drone clusters. First, the method of the present invention uses a lightweight feature extraction model and a graph convolutional network, which greatly reduces the demand for computing resources and can operate within the limited resources of drone clusters. Secondly, the method of the present invention adopts a hierarchical drone cluster structure, including leaf nodes and head nodes, which is suitable for more complex environments. Finally, by using multi-view images for target recognition, the recognition accuracy can be significantly improved, and missed detections and false alarms can be reduced.
[0076] The basic principles, main features and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above specific embodiments, and the descriptions in the above specific embodiments and the specification are only for further illustrating the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed for protection.
Claims
1. A lightweight multi-view collaborative image recognition method for unmanned clusters, characterized in that: The following steps are involved: Step 10, constructing a multi-view recognition unmanned cluster system, wherein the multi-view recognition unmanned cluster system includes a head node UAV and multiple leaf node UAVs, and the multiple leaf node UAVs all communicate with the head node UAV; the multiple leaf node UAVs are distributed in different directions of the recognition area; Step 20, the leaf node drone collects a single-view image of the identified object, extracts single-view image features, and transmits the single-view image features to the head node drone; Step 30, the head node drone associates and aggregates the single-view image features sent by all leaf node drones to obtain multi-view image features; and uses the multi-view image features to perform recognition to obtain recognition results; In step 20, extracting single-view image features specifically includes: Inputting the single-view image into an improved ResNet18 model, extracting features before the last fully connected layer as single-view image features; the improved ResNet18 model adopts separable convolution; In step 30, the head node drone associates and aggregates the single-view image features sent by all leaf node drones to obtain multi-view image features, which specifically includes: Step 301, constructing an initial multi-view based on the single-view image features of all leaf node drones; Step 302, based on the adjacency relationship between the leaf node drones in the initial multi-view, update the single-view image features of each leaf node drone to obtain the initial updated image features of the leaf node drone; Step 303, based on the remote relationship between the leaf node drones in the initial multi-view, the relationship features between the initial image features of the leaf node drones are obtained, and then the initial image features of the leaf node drones are fused to obtain the fused image features; Step 304, coarsening the graph constructed by fusion image features to obtain multi-view image features; The step 303 specifically includes: The initial image features of each leaf node drone are concatenated, and the relational features of each leaf node drone are obtained through the output of the association module; the association module is used to learn the relationship between the initial image features of two leaf node drones and fuse their features; the association module consists of a linear layer and an activation function, and the convolution in the linear layer is replaced by a separable convolution; The relationship features of the leaf node drone are spliced with the initial image features of the leaf node drone, and the fused image features are obtained through the output of the fusion module; the fusion module is used to fuse the learned relationship features with the single-view image features of the leaf node drone, so that they can better interact and jointly affect the final output; the fusion module consists of a linear layer, a batch normalization layer and an activation function, and the convolution in the linear layer is replaced by a separable convolution.
2. The lightweight multi-view collaborative image recognition method for unmanned clusters according to claim 1 is characterized in that: The step 302 specifically includes: The K nearest neighbor algorithm is used to find the k nearest leaf node drones of each leaf node drone, and a sparse adjacency matrix A∈M×M is obtained, where M represents the total number of leaf node drones. In the formula, A i,j represents the connection relationship between leaf node drone i and leaf node drone j, n i,1 , ..., n i,k represents the set of k nearest neighboring node IDs of leaf node drone i, n j,1 , ..., n j,k represents the set of k nearest neighboring node IDs of drone j; The sparse adjacency matrix A is used to update the single-view image features of each leaf node drone using formula (2) to obtain the initial image features of the leaf node drone: In the formula, F′ i represents the initial image features of leaf node drone i, F j represents the single-view image features of the leaf node drone j, σ represents the activation function, and W represents the weight matrix.
3. The lightweight multi-view collaborative image recognition method for unmanned clusters according to claim 1 is characterized in that: In step 304, the farthest point sampling algorithm is used to coarsen the graph constructed by fusion of image features.
