A Multi-View Target Identity Determination Method for Unmanned Aerial Vehicles Based on Twin Networks

Abstract

This invention discloses a method for identifying the identity of unmanned aerial vehicles (UAVs) and unmanned surface vessels (USVs) from multiple perspectives based on Siamese networks. The method includes: acquiring images of sea surface targets using UAVs and USVs; filtering the acquired images; and annotating the acquired and filtered images to obtain an image dataset. The method outputs bounding boxes and category information from the UAV and USV target detection networks, crops the bounding box images, and inputs them into the constructed Siamese network for training. The Siamese network uses a dual-branch feature extraction module and a relation-aware self-attention mechanism to refine and enhance features, and then performs cosine similarity measurement on the enhanced features to obtain a target identity score. Based on the identity score and a set threshold, the method determines whether the targets from two perspectives are the same target. This invention effectively addresses complex environmental interference, significantly improves robustness, and maintains stable accuracy and recall when identifying the identity of vessels in air and sea cross-perspective scenarios.

Description

Technical Field

[0001] This invention relates to the field of maritime multi-source target situational awareness fusion technology, and in particular to a method for identifying the identity of unmanned aerial vehicles (UAVs) and ships from multiple perspectives based on twin networks. Background Technology

[0002] With the rapid development of UAV and unmanned surface vessel (USV) technologies, their applications in the marine field are becoming increasingly widespread. UAVs can provide a broad field of view from the air, quickly acquiring information about large areas of the sea surface; USVs can observe and track targets at close range on the sea surface. However, in practical applications, due to differences in perspective, images of ships on the sea surface acquired by UAVs and USVs vary significantly in appearance, attitude, and lighting, making it difficult to accurately determine whether ships from different perspectives are the same target. Traditional target matching methods are mainly based on feature extraction and matching algorithms, such as Scale Invariant Feature Transform (SIFT) and Histogram of Oriented Gradients (HOG). These methods are effective in handling single-view target matching problems, but they have limitations when facing multi-view target matching problems. They are difficult to accurately extract representative features, and in practical applications, the sea surface environment is complex and variable, with interference from factors such as lighting changes, occlusion, and noise. Traditional matching methods are not robust to these factors. Traditional matching methods focus on the complementary use of information under the same viewpoint or similar conditions, such as improving the recall rate of target detection or enhancing detection stability in noisy environments. However, they remain insufficient for the problem of target identity discrimination across different viewpoints. Specifically, traditional methods often lack the ability to model the appearance changes of the same target under different viewpoints, and cannot effectively eliminate feature inconsistencies caused by viewpoint differences, scale variations, and complex sea surface environments. This makes it difficult to accurately determine whether targets in images from different sources belong to the same object during UAV-UV collaborative detection. Therefore, relying solely on traditional target fusion methods cannot meet the actual needs of target identity discrimination in multi-view collaborative detection by UAVs and UVs. In conclusion, there is a need for a stable and reliable multi-view target identity discrimination method for UAVs and UVs that can overcome interference from factors such as viewpoint differences and sea surface environments, and adapt to the current needs of UAV-UV collaborative detection.

[0003] This invention addresses the following technical problems: In collaborative operations between UAVs and unmanned surface vessels (USVs), existing technologies have significant shortcomings in determining the identity of maritime targets from multiple perspectives. First, the significant differences in perspective between UAVs (air-to-sea) and USVs (sea-to-sea) lead to marked inconsistencies in appearance, scale, and lighting for the same target, making traditional feature matching methods (such as the SIFT algorithm) ill-suited. Second, the complex marine environment, with its waves, fog, and varying lighting, interferes with the stability of target features, rendering existing methods insufficiently robust. Furthermore, existing methods largely rely on multi-view geometric projection, requiring precise calibration of the relative poses of the UAV and USV, which is difficult to achieve in real-world scenarios. Finally, traditional methods suffer from high computational complexity, making them unsuitable for real-time performance, especially in high-density target scenarios. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method for identifying the identity of unmanned aerial vehicles (UAVs) and ships from multiple perspectives based on twin networks, addressing the deficiencies in the existing technology.

[0005] The technical solution adopted by this invention to solve its technical problem is: This invention provides a method for identifying the identity of unmanned aerial vehicles (UAVs) from multiple perspectives based on twin networks. The method includes the following steps: Step 1: Using UAVs and unmanned surface vessels (USVs) equipped with visible light image sensors, deploy an air-to-sea target detection network on the UAVs and a sea-to-sea target detection network on the USVs; acquire images of typical sea surface targets at long distances, filter the acquired images, remove sample images from harsh environments and those with completely obscured targets, and divide the dataset according to the shooting platform and perspective; label the acquired and filtered images to obtain the image dataset. Step two: Using the image dataset processed in step one, the target bounding boxes and category information are output through the target detection network for UAVs and unmanned surface vessels. The cropped target bounding box images are then input into the constructed Siamese network. The Siamese network includes a feature extraction stage and a similarity measurement stage. In the feature extraction stage, a dual-branch feature extraction module is used, with a lightweight convolutional neural network as the backbone network, and a relationship-aware self-attention mechanism is introduced. In the similarity measurement stage, feature matching scores are calculated using cosine similarity and fully connected layers. The Siamese network is then trained. Step 3: Acquire the image to be identified, crop the image, and input it into the trained Siamese network. The Siamese network uses a two-branch feature extraction module and a relation-aware self-attention mechanism to refine and enhance the features, and then performs cosine similarity measurement on the enhanced features to obtain the target identity score. Based on the identity score and a set threshold, it is determined whether the target in the two views is the same target. If the score is greater than the threshold, it is considered to be the same target; otherwise, it is considered not to be the same target. The final detection result is output.

[0006] Furthermore, the target detection network of this invention employs the deep learning model YOLOv8.

[0007] Furthermore, the method for feature extraction in step two of this invention specifically includes: A dual-branch feature extraction module is adopted, using a lightweight convolutional neural network MobileNetV3 as the backbone network to extract depth features of targets from both air-to-sea and sea-to-sea perspectives by sharing weights. A relation-aware self-attention mechanism, namely the RASA module, is introduced to dynamically weight key feature regions from different perspectives. For each key feature region, the RASA module is used to stack various relationships to capture global structural information and local appearance information. In other words, the pairwise correlations of all feature regions are combined with the features themselves and a shallow neural network is used to learn the attention.

[0008] Furthermore, the specific implementation method of the RASA module of the present invention includes: The output feature map of the lightweight convolutional neural network is , Where C represents the dimension, i.e., a C-dimensional feature vector. H and W This indicates that the RASA module needs to learn a scale of size. Spatial relationship diagram; All spatial locations form A graph with 1 node The spatial locations are scanned level by level, and their identifiers are assigned as 1, ..., N. The relationship between each pair of nodes is calculated, and the result is obtained using the dot product method.

[0009] in, x i and x j yes X Spatial feature vectors in f s ( ) represents the dot product. and For two embedded functions, through It consists of convolutional layers, batch normalized (BN) layers, and ReLU activation functions; the same method is used to calculate... Using relationships To describe and The bidirectional relationships between nodes are calculated; similarly, all relationships between nodes are calculated, resulting in an affinities matrix. This relation is linked to the original input feature nodes, and the corresponding weight values ​​and weight graph are obtained through a global average pooling.

[0010] Furthermore, the method for the similarity measurement stage in step two of this invention specifically includes: The feature matching score is calculated using cosine similarity and a fully connected layer, outputting the probability that two targets are the same ship. The formula for calculating cosine similarity is as follows:

[0011] in, This refers to the feature vector output by the fully connected layer of the UAV-view branch network in a Siamese network. This refers to the feature vector output by the fully connected layer of the unmanned surface vessel (USV) perspective branch network in the Siamese network. Let L2 be the vector norm.

[0012] Furthermore, the training method in step three of the present invention specifically includes: Training is performed using triplet loss, where positive samples are images of the same ship target from different viewpoints, and negative samples are images of different ship targets; the triplet loss function is shown in the following equation:

[0013] in, Anchor sample, i.e., the perspective of the drone; A positive sample is the same vessel target from the perspective of the unmanned surface vessel. For negative samples, different ship targets; For interval parameters, The cosine similarity between two samples.

[0014] Furthermore, the interval parameters of the present invention Set it to 0.2.

[0015] Furthermore, the method for cropping the image in step three of the present invention specifically includes: The system utilizes both drones and unmanned surface vessels to detect targets on the sea surface. Based on the bounding boxes and category information of the detection results, it performs a cropping operation on the sea surface targets and scales the cropped images to a uniform pixel size.

[0016] This invention provides a multi-view target identity determination system for unmanned aerial vehicles and ships based on twin networks, comprising: Memory, used to store executable computer programs; The processor, when executing an executable computer program stored in memory, implements the aforementioned method for identifying the identity of unmanned aerial vehicles (UAVs) and surface vessels from multiple perspectives based on twin networks.

[0017] This invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method for identifying the identity of unmanned aerial vehicles (UAVs) and surface vessels from multiple perspectives based on twin networks.

[0018] The beneficial effects of this invention are: (1) This invention overcomes the problem of inconsistent appearance caused by the difference between air-to-sea and sea-to-sea perspectives, and achieves cross-perspective feature matching; (2) This invention improves the robustness of the algorithm in complex sea environments and can effectively cope with interference such as waves, fog and haze and changes in light. (3) This invention avoids the precise dependence on the relative pose of UAVs and unmanned surface vessels by using deep learning to adaptively learn perspective differences; (4) The present invention adopts a lightweight network design and embedded deployment scheme to meet the real-time requirements in practical applications.

[0019] In summary, in cross-view scenarios involving air and sea, traditional methods achieve only 81% accuracy and 80% recall for ship target identity determination; while the proposed method achieves over 92% accuracy and 86% recall. Secondly, the algorithm effectively handles complex environmental interference, significantly improving robustness and maintaining stable accuracy and recall. Furthermore, it can achieve real-time processing of over 30 frames per second on embedded devices (such as NVIDIA Jetson NX), meeting the needs of real-world maritime target scenarios. Simultaneously, this method is applicable to various ship types (such as cargo ships, unmanned surface vessels, and yachts), demonstrating strong scenario adaptability. Finally, it eliminates the need for precise calibration of the relative poses of UAVs and unmanned surface vessels, reducing system deployment difficulty and cost. Through the above technical solutions, this patent provides an efficient and reliable multi-view target identity determination method for collaborative detection by UAVs and unmanned surface vessels. Attached Figure Description

[0020] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of a UAV-ship collaborative maritime reconnaissance scenario according to an embodiment of the present invention; Figure 2 This is a design diagram of the twin network structure according to an embodiment of the present invention; Figure 3 This is a structural diagram of the relationship-aware self-attention mechanism according to an embodiment of the present invention; Figure 4 This is an example of the detection results from the bow view in an embodiment of the present invention; Figure 5 This is an example of the detection result from the machine-side perspective in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0022] Example 1 The method for multi-view target identity determination of unmanned aerial vehicles and ships based on twin networks according to embodiments of the present invention includes the following steps: (1) Multi-view target detection: An air-to-sea target detection network is deployed at the UAV end, and a sea-to-sea target detection network is deployed at the UAV end. The two target detection networks output the bounding boxes and category labels of sea surface targets. The target detection network adopts a lightweight and efficient deep learning model (YOLOv8) to adapt to the high accuracy target detection requirements under different perspectives.

[0023] (2) Siamese Network Design: A lightweight Siamese network structure was designed. The target detection network at the UAV and unmanned surface vessel ends outputs the bounding boxes and category information of the targets. Then, the target bounding box images are cropped and input into the Siamese network to learn the similarity and difference features of the targets from the two perspectives. Finally, the similarity metric matrix of the targets from the two perspectives is output. Based on the similarity metric matrix, the identity discrimination and matching of sea surface targets are performed. The network design structure is as follows: Figure 2 As shown.

[0024] In the feature extraction stage, a dual-branch feature extraction method is adopted, using a lightweight convolutional neural network (MobileNetV3) as the backbone network, sharing weights to extract depth features of targets from the air-to-sea perspective (UAV) and the sea-to-sea perspective (UAV).

[0025] Simultaneously, a relation-aware self-attention mechanism (RASA module) is introduced to dynamically weight key feature regions from different perspectives (such as hull structure and deck layout) to enhance the saliency of discriminative features. For each feature location, to more compactly capture global structural information and local appearance information, the RASA module is used to stack various relationships, that is, the pairwise affinities of all feature locations are combined with the features themselves, and an attention is learned using a shallow neural network. By applying the RASA module, the feature representation capability can be significantly enhanced. This allows us to learn more discriminative features. Given an intermediate feature tensor... ,here To obtain the output feature map of a certain layer of a CNN, the RASA module needs to learn a scale of [missing information]. The spatial relationship diagram is shown. Here, the C-dimensional feature vector at each spatial location is used as a feature node according to the channel division method. All spatial locations form... A graph with 1 node .like Figure 3 As shown, the spatial locations are scanned level by level, and their identifiers are assigned as 1, ..., N. Then, the relationship between each pair of nodes is calculated, which is done using the dot product method:

[0026] here and For two embedded functions, through It consists of convolutional layers, batch normalized (BN) layers, and ReLU activation functions. Similarly, it calculates the activation function in the same way. Therefore, this invention can utilize relation pairs To describe and The two-way relationship between them. Finally, by analogy, this invention can calculate the relationship pairs between all nodes, thus obtaining an affinities matrix. Then, the present invention connects this relation pair with the original input feature nodes, and obtains the corresponding weight values ​​and weight graph through a global average pooling.

[0027] In the similarity measurement stage, feature matching scores are calculated using cosine similarity and fully connected layers, outputting the probability that two targets are the same ship. The formula for calculating cosine similarity is as follows:

[0028] in, This refers to the feature vector output by the fully connected layer of the UAV-view branch network in a Siamese network. This refers to the feature vector output by the fully connected layer of the unmanned surface vessel (USV) perspective branch network in the Siamese network. Let L2 be the vector norm.

[0029] (3) Training optimization strategy: Triplet loss is used for training. Positive samples are images of the same ship from different perspectives, and negative samples are images of different ships. This ensures that features of the same target are as close as possible in spatial location, while features of different targets are as far apart as possible in spatial location. The triplet loss function is shown in the following formula:

[0030] in, Anchor sample (drone perspective). This is a positive sample (the same vessel from the perspective of an unmanned surface vessel). For negative samples (different ships). For interval parameters, The cosine similarity between two samples.

[0031] The final output is the detection result.

[0032] Example 2 This invention employs a twin network approach to solve the problem of multi-view target identity discrimination and matching for unmanned aerial vehicles (UAVs). The first consideration is the data issue. Deep learning relies on massive amounts of data; generally, the more data available, the better the network model performs. Therefore, data collection and database establishment are crucial.

[0033] Step one involves using unmanned aerial vehicles (UAVs) and unmanned surface vessels (USVs) equipped with visible light image sensors to acquire long-range images of typical sea surface targets. During the acquisition of visible light images, the sensors on both platforms are susceptible to environmental conditions such as sea fog, sea wind, sea wave reflections, and varying lighting levels, which can affect image quality. To address these issues, the acquired visible light images are filtered, removing samples from extremely difficult targets in harsh environments or those completely obscured. The dataset is then segmented based on the capturing platform and viewing angle. Subsequently, the acquired and filtered images are labeled.

[0034] Step two involves using the image dataset processed in step one to train target detection models and Siamese networks from different perspectives. These trained air-to-sea and sea-to-sea target detection models and Siamese networks are then deployed on the UAV / UAV platform. In actual detection missions, both UAVs and UAVs are used to detect targets on the sea surface. Based on the bounding boxes and category information of the detection results, the targets are cropped, and the cropped images are scaled to a uniform size of 224 pixels.

[0035] Step three involves inputting the cropped image into a Siamese network. The Siamese network utilizes a two-branch feature extraction module and a relation-aware self-attention mechanism to refine and enhance the features. The enhanced features are then measured using cosine similarity to obtain a target identity score. Based on this identity score and a set threshold, it is determined whether the ship targets from the two viewpoints are the same target. If the score is greater than the threshold, they are considered the same target; otherwise, they are considered different targets.

[0036] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0037] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for identifying the identity of unmanned aerial vehicles (UAVs) from multiple perspectives based on twin networks, characterized in that, The method includes the following steps: Step 1: Using UAVs and unmanned surface vessels (USVs) equipped with visible light image sensors, deploy an air-to-sea target detection network on the UAVs and a sea-to-sea target detection network on the USVs; acquire images of typical sea surface targets at long distances, filter the acquired images, remove sample images from harsh environments and those with completely obscured targets, and divide the dataset according to the shooting platform and perspective; label the acquired and filtered images to obtain the image dataset. Step two: Using the image dataset processed in step one, the target bounding boxes and category information are output through the target detection network for UAVs and unmanned surface vessels. The cropped target bounding box images are then input into the constructed Siamese network. The Siamese network includes a feature extraction stage and a similarity measurement stage. In the feature extraction stage, a dual-branch feature extraction module is used, with a lightweight convolutional neural network as the backbone network, and a relationship-aware self-attention mechanism is introduced. In the similarity measurement stage, feature matching scores are calculated using cosine similarity and fully connected layers. The Siamese network is then trained. Step 3: Acquire the image to be identified, crop the image, and input it into the trained Siamese network. The Siamese network uses a two-branch feature extraction module and a relation-aware self-attention mechanism to refine and enhance the features, and then performs cosine similarity measurement on the enhanced features to obtain the target identity score. Based on the identity score and a set threshold, it is determined whether the target in the two views is the same target. If the score is greater than the threshold, it is considered to be the same target; otherwise, it is considered not to be the same target. The final detection result is output.

2. The method for multi-view target identity determination of unmanned aerial vehicles and ships based on twin networks according to claim 1, characterized in that, The target detection network uses the deep learning model YOLOv8.

3. The method for multi-view target identity determination of unmanned aerial vehicles and ships based on twin networks according to claim 1, characterized in that, The specific methods for the feature extraction stage in step two include: A dual-branch feature extraction module is adopted, using a lightweight convolutional neural network MobileNetV3 as the backbone network to extract depth features of targets from both air-to-sea and sea-to-sea perspectives by sharing weights. A relation-aware self-attention mechanism, namely the RASA module, is introduced to dynamically weight key feature regions from different perspectives. For each key feature region, the RASA module is used to stack various relationships to capture global structural information and local appearance information. In other words, the pairwise correlations of all feature regions are combined with the features themselves and a shallow neural network is used to learn the attention.

4. The method for multi-view target identity determination of unmanned aerial vehicles and ships based on twin networks according to claim 3, characterized in that, The specific implementation method of the RASA module includes: The output feature map of the lightweight convolutional neural network is , Where C represents the dimension, i.e., a C-dimensional feature vector. H and W This indicates that the RASA module needs to learn a scale of size. Spatial relationship diagram; All spatial locations form A graph with 1 node The spatial locations are scanned level by level, and their identifiers are assigned as 1, ..., N. The relationship between each pair of nodes is calculated, and the result is obtained using the dot product method. in, x i and x j yes X Spatial feature vectors in f s ( ) represents the dot product. and For two embedded functions, through It consists of convolutional layers, batch normalized (BN) layers, and ReLU activation functions; the same method is used to calculate... Using relationships To describe and The bidirectional relationships between nodes are calculated; similarly, all relationships between nodes are calculated, resulting in an affinities matrix. This relation is linked to the original input feature nodes, and the corresponding weight values ​​and weight graph are obtained through a global average pooling.

5. The method for multi-view target identity determination of unmanned aerial vehicles and ships based on twin networks according to claim 4, characterized in that, The method for the similarity measurement stage in step two specifically includes: The feature matching score is calculated using cosine similarity and a fully connected layer, outputting the probability that two targets are the same ship. The formula for calculating cosine similarity is as follows: in, This refers to the feature vector output by the fully connected layer of the UAV-view branch network in a Siamese network. This refers to the feature vector output by the fully connected layer of the unmanned surface vessel (USV) perspective branch network in the Siamese network. Let L2 be the vector norm.

6. The method for multi-view target identity determination of unmanned aerial vehicles and ships based on twin networks according to claim 5, characterized in that, The training method in step three specifically includes: Training is performed using triplet loss, where positive samples are images of the same ship target from different viewpoints, and negative samples are images of different ship targets; the triplet loss function is shown in the following equation: in, Anchor sample, i.e., the perspective of the drone; A positive sample is the same vessel target from the perspective of the unmanned surface vessel. For negative samples, different ship targets; For interval parameters, The cosine similarity between two samples.

7. The method for multi-view target identity determination of unmanned aerial vehicles and ships based on twin networks according to claim 6, characterized in that, The interval parameter Set it to 0.

2.

8. The method for multi-view target identity determination of unmanned aerial vehicles and ships based on twin networks according to claim 1, characterized in that, The method for cropping the image in step three specifically includes: The system utilizes both drones and unmanned surface vessels to detect targets on the sea surface. Based on the bounding boxes and category information of the detection results, it performs a cropping operation on the sea surface targets and scales the cropped images to a uniform pixel size.

9. A multi-view target identity determination system for unmanned aerial vehicles and surface vessels based on twin networks, characterized in that, include: Memory, used to store executable computer programs; A processor, when executing an executable computer program stored in a memory, implements the multi-view target identity determination method for unmanned aerial vehicles based on twin networks as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the method for determining the identity of unmanned aerial vehicles (UAVs) from multiple perspectives based on twin networks, as described in any one of claims 1 to 8.

Images