Fine-grained object recognition method and device based on multi-view image manifold representation learning

Abstract

A fine-grained target recognition method and apparatus based on multi-view image manifold representation learning is proposed. The method includes the following steps: S1, constructing a dataset of sea surface target images acquired from different azimuth angles; S2, constructing a dual-view spectral redundancy removal and correlation module; S3, constructing a multi-view hyperspherical consistency module; S4, constructing a multi-view image manifold representation learning network model based on the dual-view spectral redundancy removal and correlation module and the multi-view hyperspherical consistency module; S5, training and testing the multi-view image manifold representation learning network model to determine the optimal weights; and S6, performing inference on the test set based on the optimal weights to achieve fine-grained recognition of infrared sea surface targets. The model with the optimal weights is then deployed to an unmanned surface vessel's onboard perception computing platform for online recognition of real-time acquired infrared video streams. This invention offers high accuracy in recognizing sea surface targets and can adapt to scenarios with significant differences in target angles.

Description

Technical Field

[0001] This invention relates to the field of infrared sea surface target recognition technology, and in particular to a fine-grained target recognition method and apparatus based on multi-view image manifold representation learning. Background Technology

[0002] Maritime situational awareness is a fundamental support capability for safeguarding national maritime rights and ensuring navigational safety, and accurate identification of surface ships is a key link in achieving maritime situational assessment. Unmanned surface vehicles (USVs), as a new type of unmanned maritime platform, possess advantages such as autonomous navigation, low cost, and high maneuverability, and have been widely used in maritime patrol, search and rescue, and reconnaissance missions. Infrared imaging systems, due to their independence from external lighting and all-weather operation, have become important sensors for target perception on USV platforms. However, fine-grained identification of infrared targets on the sea surface still faces many technical challenges.

[0003] First, the imaging mechanism of infrared images inherently results in a lack of texture information and low contrast. Unlike the rich color and texture features of visible light images, infrared images rely primarily on the radiation temperature difference between the target and the background. This leads to blurred target shapes and outlines, and a lack of detailed information. The appearance differences between different types of ships within the same category are further compressed in infrared images, significantly reducing inter-class separability and increasing the difficulty of fine-grained identification. Second, the appearance of sea surface targets is significantly affected by the observation angle. When the same ship is observed from different azimuth angles, its infrared radiation profile, structural features, and occlusion relationships vary considerably. Features acquired from a single viewpoint are insufficient to fully characterize the target's essential attributes, easily leading to identification results that are sensitive to the observation angle and lack robustness. Third, interference factors from the complex marine environment cannot be ignored. Waves, sea clutter, solar flares, and atmospheric transmission effects all introduce background noise and clutter interference into infrared images, further reducing the signal-to-noise ratio and feature discriminability of the target signal.

[0004] Currently, mainstream methods for sea surface target recognition primarily rely on single-view images for feature extraction and classification. Traditional methods depend on manually designed feature descriptors, such as HOG and LBP features, which have limited ability to distinguish weakly textured targets in infrared images and exhibit a sharp decline in performance under changing viewpoints and complex backgrounds. While deep learning-based methods have achieved automatic feature extraction through convolutional neural networks and made significant progress in target detection and classification tasks, most existing deep learning methods focus on a single viewpoint and fail to effectively utilize multi-view information for complementary target representation. Some self-supervised representation learning methods, such as MoCo, SimCLR, Barlow Twins, and BYOL, can learn general image feature representations under unlabeled conditions, but these methods are not optimized for modeling the semantic invariance of the same target across viewpoints in multi-view scenarios and lack explicit mechanisms for eliminating redundant information between features from different viewpoints. Therefore, their performance in fine-grained sea surface target recognition tasks remains limited. Furthermore, existing methods rarely consider combining multi-view feature learning with the actual deployment requirements of unmanned surface vessels (USVs), lacking a complete technical solution from data acquisition and model training to onboard real-time inference.

[0005] In summary, how to fully utilize the complementary geometric and semantic information contained in multi-view images under the condition of multi-unmanned surface vessels (USVs) collaborative observation, learn high-quality target representations that are invariant to changes in viewpoint and separable across different categories, and effectively apply them to fine-grained identification of infrared sea surface targets and USV deployment is a technical problem that urgently needs to be solved. Summary of the Invention

[0006] To address the technical problems of insufficient texture information, low inter-class separability, and insufficient robustness of target recognition due to drastic changes in target appearance with the observation viewpoint under single-view infrared sea surface conditions, this invention provides a fine-grained target recognition method and apparatus based on multi-view image manifold representation learning.

[0007] To address the aforementioned technical problems, this invention employs the following technical method: a fine-grained target recognition method based on multi-view image manifold representation learning, comprising: Step S1: Simultaneously collect images of sea surface targets from different azimuth angles using multiple unmanned surface vessels, construct a dataset, and divide it into a training set, a validation set, and a test set; Step S2: Construct a dual-view spectrum redundancy removal and correlation module, use a deep neural network to extract the manifold features of the image, and calculate the cross-correlation of the feature dimensions. Step S3: Construct a multi-view hyperspherical consistency module to learn the semantic invariant features of the same target under different viewpoints; Step S4: Based on the dual-view spectrum redundancy removal and correlation module and the multi-view hyperspherical consistency module, construct a multi-view image manifold representation learning network model; Step S5: Train the multi-view image manifold representation learning network model based on the training set, test the model weights at each training step on the validation set, and determine the optimal model weights. Step S6: Perform inference on the test set based on the optimal weights to achieve fine-grained identification of infrared sea surface targets, and deploy the model with the optimal weights to the unmanned surface vessel's onboard perception computing platform to perform online identification of the real-time acquired infrared video stream.

[0008] Furthermore, in step S2, the construction and calculation method of the dual-view spectrum redundancy removal correlation module is as follows: [The following text appears to be a separate, unrelated section:] ...the same sea surface target... From different perspectives and perspective The image below and Feature extraction network with shared input weights To obtain high-dimensional feature vectors ,in The batch size of the input images. The high-dimensional feature vectors are standardized along the batch dimension to obtain standardized features. and ; Calculate the cross-correlation matrix of dual-view features Define the spectrum redundancy removal correlation loss function ; (1) (2) in, For batch indexing; Indexed by feature dimension; Represents high-dimensional features; As a diagonal invariant term, the diagonal elements of the constraint cross-correlation matrix approach 1; This is a non-diagonal sparsification term that constrains off-diagonal elements to approach 0. This is the balance coefficient.

[0009] Furthermore, in step S3, the construction and calculation method of the multi-view hyperspherical consistency module is as follows: different sea surface targets... and From different perspectives and perspective The image below and Feature mapping network with shared input weights To obtain high-dimensional feature vectors and Define the cosine similarity metric matrix between features. Construct a consistency loss function based on temperature-controlled InfoNCE. ; (3) (4) in, Indicates vector transpose; The modulus represents the orientation quantity; for the first in the batch Given a sample, define the features of its query sample as follows: The corresponding positive sample features are ,the remaining All samples are negative samples. ; The number of individuals on the sea surface; express of Power of 1 for and The cosine similarity metric matrix between them This refers to temperature hyperparameters.

[0010] Furthermore, in step S4, the calculation method of the multi-view image manifold representation learning network model is as follows: [Image...] and They are fed into the feature extraction network respectively. Calculate the cross-correlation matrix; convert the image and They are fed into the feature mapping network respectively. Calculate the cosine similarity metric matrix; then calculate the spectral redundancy removal correlation loss function. Consistency loss function The total loss function is obtained. The expression is: (5) in, and These are the uncertainty parameters for the corresponding associated tasks and consistency tasks, respectively; This is a regularization term.

[0011] Furthermore, the feature extraction network It consists of 4 columns of nodes. The first column includes nodes (1,1), (1,2), and (1,3); the second column includes node (2,1); nodes (1,1), (1,2), and (1,3) are connected to node (2,1); the third column includes nodes (3,1), (3,2), and (3,3); node (2,1) is connected to nodes (3,1), (3,2), and (3,3); the fourth column includes node (4,1); nodes (3,1), (3,2), and (3,3) are connected to node (4,1); node (4,1) outputs a feature map with a resolution of 128×128. The feature mapping network It consists of 6 columns of nodes. The first column includes nodes (1,1), (1,2), and (1,3); the second column includes nodes (2,1) and (2,2); nodes (1,1), (1,2), and (1,3) are connected to node (2,1); nodes (1,1), (1,2), and (1,3) are connected to node (2,2); the third column includes node (3,1); nodes (2,1) and (2,2) are connected to node (3,1); the fourth column includes nodes (4,1) and (4,2); node (3,1) is connected to nodes (4,1) and (4,2); nodes (2,1) and (4,1) are connected to each other. The nodes (2,2) and (4,2) are connected; the fifth column includes nodes (5,1), (5,2), and (5,3); node (4,1) is connected to nodes (5,1), (5,2), and (5,3) respectively; node (4,2) is connected to nodes (5,1), (5,2), and (5,3) respectively; node (,1,1) is connected to node (5,1); node (1,3) is connected to node (5,3); the sixth column includes node (6,1); nodes (5,1), (5,2), and (5,3) are connected to node (6,1) respectively; node (6,1) outputs a feature map with a resolution of 128×128; The feature extraction network and feature mapping network The image processing steps at each node are as follows: The input feature map first enters a 1×1 convolutional layer for processing; then it flows sequentially through a BN layer and a ReLU layer to complete preliminary feature transformation, normalization, and activation; next, it continues processing in three parallel branches. Branch 1 first goes through a 5×5 convolutional layer, then through a max pooling layer to extract global abstract features; Branch 2 first goes through a 3×3 depthwise separable convolutional layer, then through an attention layer to enhance fine-grained texture and key features; Branch 3 first goes through a 1×1 convolutional layer, then through a global average pooling layer to compress semantics and focus on the overall distribution; finally, the feature maps output from the three branches are aligned through multi-resolution upsampling and then integrated to output the final feature map.

[0012] Furthermore, step S5 includes: S501, based on the training set, performs self-supervised joint training on the multi-view image manifold representation learning network model, and obtains the model weights under different training steps by iteratively updating the network model weights and uncertainty parameters by minimizing the total loss function. S502, test the model weights at each training step on the validation set, and calculate the Accuracy, Precision, Recall and F1 Score metrics respectively. S503: Select the weights corresponding to the highest values ​​of Accuracy, Precision, Recall, and F1 Score, or the highest value of Accuracy, as the optimal weights for the model. (6) (7) (8) (9) Where TP, FP, TN, and FN represent the true samples, false positive samples, true negative samples, and false negative samples corresponding to all images, respectively.

[0013] Furthermore, in step S501, when training the multi-view image manifold representation learning network model, the upper limit of the number of training steps is set to 200, and the network model weights and uncertainty parameters are saved every 20 steps.

[0014] Furthermore, in step S1, four unmanned surface vessels navigate in an encircling pattern around the detected target, acquiring real-time images of the sea surface target covering a field of view from 0° to 360°, and constructing an infrared sea surface target dataset.

[0015] Preferably, in step S1, the dataset is randomly divided into a training set, a validation set, and a test set in a ratio of 5:2:3.

[0016] As another aspect of the present invention, a fine-grained target recognition device based on multi-view image manifold representation learning includes a shore-based display and control platform, a shore-based radio station, and an unmanned surface vessel (USV). The USV is equipped with an optoelectronic pod, a marine radar, a GPS antenna, an inertial measurement unit (IMU), a network switch, an onboard perception computing platform, a USV core control board, a weather sensor, an onboard power supply, and a motor driver. The shore-based display and control platform is connected to the shore-based radio station via Ethernet and serial ports, and the shore-based radio station is connected to the network switch on the USV. The onboard power supply powers all components on the USV. The optoelectronic pod, marine radar, and GPS antenna are connected to the network switch, as are the inertial measurement unit, the USV core control board, and the onboard perception computing platform. The weather sensor and the motor driver are respectively connected to the USV core control board. The multi-view image manifold representation learning network model is deployed on the onboard perception computing platform.

[0017] This invention provides a fine-grained target recognition method based on multi-view image manifold representation learning. It designs a dual-view spectral redundancy removal and correlation module and a multi-view hyperspherical consistency module, which can explicitly eliminate redundant information between different feature dimensions, maximizing the expressive capacity of features. This enables the network to extract more discriminative target representations from limited infrared image information, effectively improving the model's recognition stability under different observation angles. Compared to existing mainstream self-supervised representation learning methods such as MoCo, SimCLR, Barlow Twins, and BYOL, this invention has higher accuracy, precision, recall, and F1 score, and can learn effective multi-view features of sea surface targets, achieving excellent target recognition performance and demonstrating significant performance advantages. The fine-grained target recognition device based on multi-view image manifold representation learning provided by this invention deploys a multi-view image manifold representation learning network model based on the dual-view spectral redundancy removal and correlation module and the multi-view hyperspherical consistency module on a shipboard perception computing platform. It achieves high recognition accuracy for sea surface targets and can adapt to scenarios with large differences in target angles. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the sea surface target recognition method based on multi-view image manifold representation learning provided by the present invention. Figure 2 This is a schematic diagram of the feature extraction network in the dual-view spectrum redundancy removal and correlation module of this invention; Figure 3 This is a schematic diagram of the feature mapping network in the multi-view hyperspherical consistency module of this invention; Figure 4 This is a schematic diagram of the multi-view image manifold representation learning network model structure in this invention; Figure 5 This is a schematic diagram of the structure of nodes in the feature extraction network and feature mapping network of this invention; Figure 6 This is a schematic diagram of the connection structure of the sea surface target recognition device for multi-view image manifold representation learning in this invention; Figure 7 This is a schematic diagram of a task scenario in an embodiment of the present invention; Figure 8 This is a comparison diagram of confusion matrices using the method of this invention and other methods in embodiments of this invention (where (a) is a schematic diagram of the confusion matrix structure using the MoCo method, (b) is a schematic diagram of the confusion matrix structure using the SimCLR method, (c) is a schematic diagram of the confusion matrix structure using the Barlow Twins method, (d) is a schematic diagram of the confusion matrix structure using the BYOL method, and (e) is a schematic diagram of the confusion matrix structure using the method of this invention). Figure 9 This is a schematic diagram of the first set of multi-view recognition visualization results using the method of the present invention in an embodiment of the present invention (where (a) is the input image of view 1 in the first set, (b) is the input image of view 2 in the first set, (c) is the input image of view 3 in the first set, (d) is the input image of view 4 in the first set, and (e) is a schematic diagram of the multi-view image recognition results of the first set). Figure 10 This is a schematic diagram of the second set of multi-view recognition visualization results using the method of the present invention in an embodiment of the present invention (where (a) is the input image of view 1 in the second set, (b) is the input image of view 2 in the second set, (c) is the input image of view 3 in the second set, (d) is the input image of view 4 in the second set, and (e) is a schematic diagram of the multi-view image recognition results of the second set). Figure 11 This is a schematic diagram of the third group of multi-view recognition visualization results using the method of the present invention in an embodiment of the present invention (where (a) is the input image of view 1 in the third group, (b) is the input image of view 2 in the third group, (c) is the input image of view 3 in the third group, (d) is the input image of view 4 in the third group, and (e) is a schematic diagram of the multi-view image recognition results of the third group). Figure 12 This is a schematic diagram of the fourth group of multi-view recognition visualization results using the method of the present invention in an embodiment of the present invention (where (a) is the input image of view 1 in the fourth group, (b) is the input image of view 2 in the fourth group, (c) is the input image of view 3 in the fourth group, (d) is the input image of view 4 in the fourth group, and (e) is a schematic diagram of the multi-view image recognition results of the fourth group). Detailed Implementation

[0019] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and accompanying drawings. The content mentioned in the embodiments is not intended to limit the present invention.

[0020] like Figure 1 As shown, the fine-grained target recognition method based on multi-view image manifold representation learning provided by this invention includes: Step S1: Simultaneously acquire images of sea surface targets from different azimuth angles using multiple unmanned surface vessels (USVs) to construct an infrared sea surface target multi-view image dataset. Then, label the sea surface targets in the images with rectangular boxes. Next, perform image data preprocessing on the infrared sea surface target multi-view image dataset and divide it into training set, validation set and test set according to a preset ratio.

[0021] Preferably, there are 4 unmanned surface vessels that navigate in an encircling pattern around the target being detected, and collect images of the sea surface target in real time, covering a field of view from 0° to 360°, to construct an infrared sea surface target dataset. The dataset is randomly divided into a training set, a validation set, and a test set in a ratio of 5:2:3.

[0022] Step S2: Construct a dual-view spectrum redundancy removal and correlation module, use a deep neural network to extract the manifold features of the image, and calculate the cross-correlation of the feature dimensions.

[0023] Specifically, the main task in constructing the dual-view spectrum redundancy removal and correlation module is to build a feature extraction network. ,like Figure 2 As shown, this feature extraction network The network consists of four columns of nodes. The first column includes nodes (1,1), (1,2), and (1,3), with the input image for the feature extraction network fed into these nodes. The second column includes node (2,1), with nodes (1,1), (1,2), and (1,3) connected to node (2,1). The third column includes nodes (3,1), (3,2), and (3,3), with node (2,1) connected to nodes (3,1), (3,2), and (3,3). The fourth column includes node (4,1), with nodes (3,1), (3,2), and (3,3) connected to node (4,1). Node (4,1) outputs a feature map, which is the feature extraction network's output. The final output feature map has a resolution of 128×128.

[0024] The aforementioned feature extraction network The process of processing images by nodes in the image can be found by referring to... Figure 5The input feature map is first processed through a 1×1 convolutional layer. Then, it flows through a BN layer and a ReLU layer to complete the initial feature transformation, normalization, and activation. Next, it is processed in three parallel branches. Branch 1 first goes through a 5×5 convolutional layer and then a max pooling layer to extract global abstract features. Branch 2 first goes through a 3×3 depthwise separable convolutional layer and then an attention layer to enhance fine-grained texture and key features. Branch 3 first goes through a 1×1 convolutional layer and then a global average pooling layer to compress semantics and focus on the overall distribution. Finally, the feature maps output from the three branches are aligned through multi-resolution upsampling (using interpolation to unify the resolution) and then integrated to output the final feature map.

[0025] The calculation method for the constructed dual-view spectrum redundancy decorrelation module is as follows: Targets on the same sea surface From different perspectives and perspective The image below and Feature extraction network with shared input weights To obtain high-dimensional feature vectors ,in The batch size of the input images. The high-dimensional feature vectors are standardized along the batch dimension to obtain standardized features. and ; Calculate the cross-correlation matrix of dual-view features To avoid feature collapse and remove redundant information between features, a spectrum redundancy removal correlation loss function is defined. The loss function includes diagonal invariance terms and off-diagonal sparsification terms. (1) (2) in, For batch indexing; Indexed by feature dimension; Represents high-dimensional features; As a diagonal invariant term, the diagonal elements of the cross-correlation matrix are constrained to approach 1, ensuring that the features of the same target from different perspectives are highly correlated in the corresponding dimensions. For off-diagonal sparsification, the off-diagonal elements are constrained to approach 0, making different feature dimensions orthogonal to each other, thereby maximizing the expressive capacity of the features; This is the balance coefficient.

[0026] Step S3: Construct a multi-view hyperspherical consistency module to learn the semantic invariant features of the same target under different viewpoints.

[0027] Specifically, the main task in constructing the multi-view hyperspherical consistency module is to build a feature mapping network. ,like Figure 3 As shown, this feature mapping network The network consists of 6 columns of nodes. The first column includes nodes (1,1), (1,2), and (1,3), with the input image of the feature mapping network fed into these nodes. The second column includes nodes (2,1) and (2,2). Nodes (1,1), (1,2), and (1,3) are connected to node (2,1), and (1,2) are connected to node (2,2). The third column includes node (3,1). Nodes (2,1) and (2,2) are connected to node (3,1). The fourth column includes nodes (4,1) and (4,2). Node (3,1) is connected to nodes (4,1), (1,2), (1,3), (1,1), (1,2), and (1,3), (1,1), (1,2), ... 4,2) are connected; node (2,1) is connected to node (4,1); node (2,2) is connected to node (4,2); the 5th column includes nodes (5,1), (5,2), and (5,3); node (4,1) is connected to nodes (5,1), (5,2), and (5,3) respectively; node (4,2) is connected to nodes (5,1), (5,2), and (5,3) respectively; node (,1,1) is connected to node (5,1); node (1,3) is connected to node (5,3); the 6th column includes node (6,1); nodes (5,1), (5,2), and (5,3) are connected to node (6,1) respectively; the output feature map of node (6,1) is the feature extraction network. The final output feature map has a resolution of 128×128.

[0028] The aforementioned feature mapping network The process of processing images by nodes in the image can be found by referring to... Figure 5 The input feature map is first processed through a 1×1 convolutional layer. Then, it flows through a BN layer and a ReLU layer to complete the initial feature transformation, normalization, and activation. Next, it is processed in three parallel branches. Branch 1 first goes through a 5×5 convolutional layer and then a max pooling layer to extract global abstract features. Branch 2 first goes through a 3×3 depthwise separable convolutional layer and then an attention layer to enhance fine-grained texture and key features. Branch 3 first goes through a 1×1 convolutional layer and then a global average pooling layer to compress semantics and focus on the overall distribution. Finally, the feature maps output from the three branches are aligned through multi-resolution upsampling (using interpolation to unify the resolution) and then integrated to output the final feature map.

[0029] The calculation method for the constructed multi-view hyperspherical consistency module is as follows: Targets on different sea surfaces and From different perspectives and perspective The image below and Feature mapping network with shared input weights To obtain high-dimensional feature vectors and Define the cosine similarity metric matrix between features. Construct a consistency loss function based on temperature-controlled InfoNCE. ; (3) (4) in, Indicates vector transpose; The modulus represents the orientation quantity; for the first in the batch Given a sample, define the features of its query sample as follows: The corresponding positive samples (different perspectives on the same target) have the following features: ,the remaining All samples are negative samples. ; The number of individuals on the sea surface; express of Power of 1 for and The cosine similarity metric matrix between them The temperature hyperparameter is used to adjust the model's attention to difficult negative samples. This loss function maximizes the similarity of positive sample pairs and minimizes the similarity of negative sample pairs, so that the model forms a compact intra-class distribution and a separate inter-class distribution in the feature space.

[0030] Step S4: Based on the dual-view spectral redundancy removal and correlation module and the multi-view hyperspherical consistency module, construct a multi-view image manifold representation learning network model, such as... Figure 4 As shown, the computation method of this multi-view image manifold representation learning network model is as follows: [Image...] and They are fed into the feature extraction network respectively. Calculate the cross-correlation matrix; convert the image and They are fed into the feature mapping network respectively. Calculate the cosine similarity metric matrix; then calculate the spectral redundancy removal correlation loss function. Consistency loss function The overall consistency loss function is obtained. A multi-task uncertainty adaptive weighted strategy is designed, and the overall loss function of the network model is defined. Design two learnable variance parameters, corresponding to the uncertainties of the association task and the consistency task respectively, then the total loss function... The expression is: (5) in, and These are the uncertainty parameters for the corresponding associated tasks and consistency tasks, respectively; For regularization, prevent and It grows indefinitely; during backpropagation, the network simultaneously updates its weight parameters using gradient descent. and uncertainty parameters , This automatically balances the contributions of the two modules at different training stages.

[0031] Step S5: Train the multi-view image manifold representation learning network model based on the training set, test the model weights at each training step on the validation set, and determine the optimal model weights.

[0032] S501, based on the training set, performs self-supervised joint training on a multi-view image manifold representation learning network model, by minimizing the total loss function. The network model weights and uncertainty parameters are iteratively updated to obtain model weights under different training steps. Specifically, when training the multi-view image manifold representation learning network model, the upper limit of the training steps is set to 200, and the network model weights and uncertainty parameters are saved every 20 steps, for a total of 10 network model weights.

[0033] S502, the model weights at each training step are tested on the validation set, and the metrics Accuracy, Precision, Recall, and F1 Score are calculated respectively. Accuracy, Precision, Recall, and F1 Score represent accuracy, precision, recall, and F1 score, respectively. The calculation methods for these four metrics are as follows: (6) (7) (8) (9) Where TP, FP, TN, and FN represent the true samples, false positive samples, true negative samples, and false negative samples corresponding to all images, respectively.

[0034] S503 selects the weights corresponding to the highest values ​​of Accuracy, Precision, Recall, and F1 Score as the optimal weights of the model; if not all metrics are at their highest values, the weights corresponding to the highest Accuracy are selected as the optimal weights of the model.

[0035] Step S6: Perform inference on the test set based on the optimal weights to achieve fine-grained identification of infrared sea surface targets, and deploy the model with the optimal weights to the unmanned surface vessel's onboard perception computing platform to perform online identification of the real-time acquired infrared video stream.

[0036] like Figure 6 As shown, the fine-grained target recognition device based on multi-view image manifold representation learning provided by this invention includes a shore-based display and control platform, a shore-based radio station, and an unmanned surface vessel (USV). The USV is equipped with an optoelectronic pod, a marine radar, a GPS antenna, an inertial measurement unit (IMU), a network switch, an onboard perception computing platform, an USV core control board, a weather sensor, an onboard power supply, and a motor driver. The shore-based display and control platform is connected to the shore-based radio station via Ethernet and serial ports, and the shore-based radio station is connected to the network switch on the USV. The onboard power supply provides power to all components on the USV. The optoelectronic pod, marine radar, and GPS antenna are connected to the network switch, the IMU is connected to the network switch, the USV core control board is connected to the network switch, the onboard perception computing platform is connected to the network switch, and the weather sensor and motor driver are connected to the USV core control board. The aforementioned multi-view image manifold representation learning network model is deployed on the onboard perception computing platform.

[0037] The core principle of the aforementioned device is that it collects data through multiple sensors on an unmanned surface vessel (USV) platform, processes the data using a deep learning model run by an onboard perception and computing platform, and then performs remote monitoring and intervention from shore, thereby achieving fine-grained target identification. Specifically: 1) Environmental perception and data acquisition: The electro-optical pods carried by the unmanned surface vessel are responsible for acquiring multi-view images, the marine radar detects the position and movement of targets on the water surface, the inertial measurement unit (IMU) records attitude and heading, and the meteorological sensor monitors real-time environmental information.

[0038] 2) Data transmission and interaction: All sensor data (including images, radar point clouds, attitude data, etc.) are aggregated through a network switch and transmitted to the onboard perception computing platform. Simultaneously, GPS antenna positioning information is also integrated, providing spatial reference for the data.

[0039] 3) Core Computing and Recognition: The onboard perception computing platform runs a "multi-view image manifold representation learning network model". This model is responsible for fusing multi-source data such as images, and extracting fine-grained features of the target by performing "manifold representation learning" on the images, thereby accurately identifying the target type or attributes.

[0040] 4) Autonomous control and decision-making: The core control board of the unmanned surface vessel receives the calculation results from the onboard perception and computing platform, combines them with its own control logic, and sends commands to the motor driver to control the course and speed of the unmanned surface vessel, so as to achieve autonomous navigation and target tracking.

[0041] 5) Remote Communication and Control: The network switch on the unmanned surface vessel (USV) maintains communication with a shore-based radio station, transmitting all status information and identification results to the shore-based control platform in real time. Operators can monitor mission progress on this shore-based control platform and intervene remotely when necessary.

[0042] To verify the effectiveness and superiority of the present invention, the present invention then built an experimental platform and used an infrared thermal imager to collect a large amount of infrared image data of sea surface targets in the Zhanjiang sea area. Referring to the existing international and domestic ship classification standards, the target categories were set as nine categories: fishing boats, speedboats, cruise ships, cargo ships, sailboats, ferries, warships, supply ships, and tugboats, and a multi-view sea surface target recognition dataset was established.

[0043] In this experiment, the fine-grained target recognition device based on multi-view image manifold representation learning provided by this invention is an unmanned surface vessel (USV) made of stainless steel with an anti-corrosion coating. The USV is 3.5 meters long, 2.5 meters wide, and 2.6 meters high. The experiment was conducted in the Zhanjiang sea area, with the USV's electro-optical pod 1 meter above sea level and its infrared imager operating at a wavelength of 8-14 μm. The experiment was conducted in sea state 1, covering a range of 3 km × 3 km. During the experiment, the electro-optical pod was positioned at a fixed angle, facing directly forward of the USV, and the infrared imager's field of view was 90 degrees. Before starting, an angle calibration device was used to calibrate and align the infrared imager with the USV at 0° forward. The relevant technical specifications of the infrared thermal imager are: 640×512 pixel uncooled detector; operating wavelength 8~12 μm; focal length: 100 mm; frame rate: 25 Hz. At an atmospheric temperature of 25℃ and a relative humidity of 80%, visibility is greater than 5km, enabling scanning and imaging of targets within a 360-degree horizontal range and an elevation angle of -40° to 65°. Infrared video and images are transmitted after compression in H.264 format. The infrared video stream is transmitted via the RTSP protocol, and the RTSP stream content can be directly read using VideoCapture.

[0044] like Figure 7The diagram illustrates the experimental scenario. Four USVs form a formation and travel in the same direction to perform collaborative target identification. In creating fine-grained infrared sea surface target identification data, four unmanned surface vessels equipped with infrared thermal imagers collected data on targets from different perspectives (different angles and distances), obtaining target images of the same target from different viewpoints. During data processing, target images of the same individual from different viewpoints were saved separately. The four USVs were arranged in an encircling pattern, collecting a total of 14,780 sea surface target images covering a field of view from 0° to 360°, constructing an infrared sea surface target dataset. Among these, the images from different viewpoints for fishing boats, speedboats, cruise ships, cargo ships, sailboats, ferries, warships, supply ships, and tugboats were 1550, 1520, 2005, 1700, 1380, 1855, 1750, 1745, and 1275, respectively, distinguishing a total of 732 different sea surface target individuals. Each sea surface target has a separate folder containing images from different perspectives. Each folder contains multiple sets of images, and each set of images is contained in a subfolder, containing four images from different perspectives. Each set of images corresponds to a category label. In this experiment, the dataset was randomly divided into a training set (7390 images), a validation set (2956 images), and a test set (4434 images) in a ratio of 5:2:3.

[0045] In this experiment, the server used for training the model was configured with 128GB of memory, an AMD Ryzen Threadripper PRO 5995WX 64-Core CPU, an NVIDIA A100 80GB PCIe graphics card, PyCharm as the development environment, and Python as the programming language. The encoder used ResNet50, the nonlinear mapper was an MLP with two fully connected layers, and the model parameters were optimized using the SGD optimizer with a momentum of 0.9 and a weight decay of 0.00005. Images were adjusted to 224×224 pixels before being input to the network.

[0046] The fine-grained target recognition method based on multi-view image manifold representation learning provided in this invention is compared with the performance of four classic self-supervised representation learning methods: Barlow Twins, Momentum Contrast Learning (MoCo), SimCLR (Simple Contrast Learning Framework), and Guided Self-Contrast Learning (BYOL). The accuracy, precision, recall, and F1 score metrics are compared. During training, unlabeled training sets are used for pre-training. In the fine-tuning stage, 10% of the model is fine-tuned using labeled training data, and max-pooling layers are used to aggregate multi-view convolutional features to obtain the final fused multi-view feature representation.

[0047] Table 1 Performance comparison of the present invention with other methods ; Table 1 compares the performance of the method of this invention with other methods. In the table, "Backbone" represents the backbone network used by each method. As shown in Table 1, compared with the four self-supervised representation learning methods—Barlow Twins, MoCo, SimCLR, and BYOL—the method of this invention has the highest accuracy, precision, recall, and F1 score, which are 95.1%, 95.2%, 95.3%, and 95.2%, respectively. The accuracy, precision, recall, and F1 score of the method of this invention are 7.1%, 6.7%, 6.8%, and 6.7% higher than those of the SimCLR algorithm, and 7.7%, 7.6%, 7.3%, and 7.4% higher than those of the BYOL algorithm, respectively. This indicates that the algorithm of this invention can learn effective multi-view features of sea surface targets and achieve excellent target recognition performance.

[0048] The confusion matrix can be used to show the overall results of each method for identifying each class of sea surface targets. Figure 8 The present invention provides the specific results of identifying nine types of sea surface targets, including fishing boats, speedboats, cruise ships, cargo ships, sailboats, ferries, warships, supply ships, and tugboats, using the method of the present invention and the Barlow Twins, MoCo, SimCLR, and BYOL methods. These nine types of sea surface targets correspond to categories 0 to 8, respectively. Figure 8 In the graph, the vertical axis represents the true target category, and the horizontal axis represents the target category predicted by the algorithm. Figure 8 (a) shows the confusion matrix results of the MoCo method. It can be seen that the MoCo method correctly identified the following numbers for the nine target categories (categories 0-8): 143, 102, 133, 148, 117, 139, 99, 120, and 113. The first row of data indicates that the MoCo method correctly identified 143 sets of data for category 0 (fishing boats), and incorrectly identified category 0 data as categories 1, 2, 3, 4, 5, 6, 7, and 8: 12, 7, 0, 0, 5, 2, 0, and 1, respectively. The method of this invention correctly identified the following numbers for the nine target categories (categories 0-8): 161, 125, 158, 162, 123, 151, 102, 138, and 121, respectively. For category 8 data, only 4 sets of data were incorrectly identified. The above comparative analysis shows that the method of the present invention can learn effective multi-view features of sea surface targets, effectively distinguish different types of sea surface targets, and achieve excellent target recognition performance.

[0049] Figure 9-12The visualization results of the multi-view recognition method of this invention are presented, with a total of 4 sets of data. Each set uses images from four different perspectives as input, and the combined recognition results from the multi-view images as output. The outputs provide the probability of identifying the target as one of 9 ship types. Figure 9-12 As can be seen, the method of the present invention can accurately identify all four sets of data, which are identified as fishing boats, speedboats, cruise ships and cargo ships, respectively, with probabilities of 0.825, 0.850, 0.900 and 0.860. The category with the highest probability is the true category of the target, and the identification is accurate.

[0050] The above embodiments are preferred implementations of the present invention. In addition, the present invention can be implemented in other ways. Any obvious substitutions without departing from the concept of the present technical solution are within the protection scope of the present invention.

[0051] To facilitate understanding by those skilled in the art of the improvements of this invention over the prior art, some of the accompanying drawings and descriptions have been simplified, and for clarity, some other elements have been omitted from this application. Those skilled in the art should realize that these omitted elements may also constitute the content of this invention.

Claims

1. A fine-grained target recognition method based on multi-view image manifold representation learning, characterized in that, include: Step S1: Simultaneously collect images of sea surface targets from different azimuth angles using multiple unmanned surface vessels, construct a dataset, and divide it into a training set, a validation set, and a test set; Step S2: Construct a dual-view spectrum redundancy removal and correlation module, use a deep neural network to extract the manifold features of the image, and calculate the cross-correlation of the feature dimensions. Step S3: Construct a multi-view hyperspherical consistency module to learn the semantic invariant features of the same target under different viewpoints; Step S4: Based on the dual-view spectrum redundancy removal and correlation module and the multi-view hyperspherical consistency module, construct a multi-view image manifold representation learning network model; Step S5: Train the multi-view image manifold representation learning network model based on the training set, test the model weights at each training step on the validation set, and determine the optimal model weights. Step S6: Perform inference on the test set based on the optimal weights to achieve fine-grained identification of infrared sea surface targets, and deploy the model with the optimal weights to the unmanned surface vessel's onboard perception computing platform to perform online identification of the real-time acquired infrared video stream.

2. The fine-grained target recognition method based on multi-view image manifold representation learning according to claim 1, characterized in that: In step S2, the construction and calculation method of the dual-view spectrum redundancy removal correlation module is as follows: [The following text appears to be a separate, unrelated section:] ...the same sea surface target... From different perspectives and perspective The image below and Feature extraction network with shared input weights To obtain high-dimensional feature vectors ,in The batch size of the input images. The high-dimensional feature vectors are standardized along the batch dimension to obtain standardized features. and ; Calculate the cross-correlation matrix of dual-view features Define the spectrum redundancy removal correlation loss function ; （1） （2） in, For batch indexing; Indexed by feature dimensions; Represents high-dimensional features; As a diagonal invariant term, the diagonal elements of the constrained cross-correlation matrix approach 1; This is a non-diagonal sparsification term that constrains non-diagonal elements to approach 0. This is the balance coefficient.

3. The fine-grained target recognition method based on multi-view image manifold representation learning according to claim 2, characterized in that: In step S3, the construction and calculation method of the multi-view hyperspherical consistency module is as follows: Different sea surface targets... and From different perspectives and perspective The image below and Feature mapping network with shared input weights To obtain high-dimensional feature vectors and Define the cosine similarity metric matrix between features. Construct a consistency loss function based on temperature-controlled InfoNCE. ; （3） （4） in, Indicates vector transpose; The modulus represents the orientation quantity; for the first in the batch Given a sample, define the features of its query sample as follows: The corresponding positive sample features are ,the remaining All samples are negative samples. ; The number of individuals on the sea surface; express of Power of 1 for and The cosine similarity metric matrix between them This refers to temperature hyperparameters.

4. The fine-grained target recognition method based on multi-view image manifold representation learning according to claim 3, characterized in that: In step S4, the calculation method of the multi-view image manifold representation learning network model is as follows: The image... and They are fed into the feature extraction network respectively. Calculate the cross-correlation matrix; convert the image and They are fed into the feature mapping network respectively. Calculate the cosine similarity metric matrix; then calculate the spectral redundancy removal correlation loss function. Consistency loss function The total loss function is obtained. The expression is: （5） in, and These are the uncertainty parameters for the corresponding associated tasks and consistency tasks, respectively; This is a regularization term.

5. The fine-grained target recognition method based on multi-view image manifold representation learning according to claim 4, characterized in that: The feature extraction network It consists of 4 columns of nodes. The first column includes nodes (1,1), (1,2), and (1,3); the second column includes node (2,1); nodes (1,1), (1,2), and (1,3) are connected to node (2,1); the third column includes nodes (3,1), (3,2), and (3,3); node (2,1) is connected to nodes (3,1), (3,2), and (3,3); the fourth column includes node (4,1); nodes (3,1), (3,2), and (3,3) are connected to node (4,1); node (4,1) outputs a feature map with a resolution of 128×128. The feature mapping network It consists of 6 columns of nodes. The first column includes nodes (1,1), (1,2), and (1,3); the second column includes nodes (2,1) and (2,2); nodes (1,1), (1,2), and (1,3) are connected to node (2,1); nodes (1,1), (1,2), and (1,3) are connected to node (2,2); the third column includes node (3,1); nodes (2,1) and (2,2) are connected to node (3,1); the fourth column includes nodes (4,1) and (4,2); node (3,1) is connected to nodes (4,1) and (4,2); nodes (2,1) and (4,1) are connected to each other. The nodes (2,2) and (4,2) are connected; the fifth column includes nodes (5,1), (5,2), and (5,3); node (4,1) is connected to nodes (5,1), (5,2), and (5,3) respectively; node (4,2) is connected to nodes (5,1), (5,2), and (5,3) respectively; node (,1,1) is connected to node (5,1); node (1,3) is connected to node (5,3); the sixth column includes node (6,1); nodes (5,1), (5,2), and (5,3) are connected to node (6,1) respectively; node (6,1) outputs a feature map with a resolution of 128×128; The feature extraction network and feature mapping network The image processing steps at each node are as follows: The input feature map first enters a 1×1 convolutional layer for processing; then it flows through a BN layer and a ReLU layer to complete the initial feature transformation, normalization, and activation; then it continues to process in three parallel branches. Branch 1 first goes through a 5×5 convolutional layer, and then through a max pooling layer to extract global abstract features; Branch 2 first goes through a 3×3 depthwise separable convolutional layer, and then through an attention layer to enhance fine-grained texture and key features; Branch 3 first goes through a 1×1 convolutional layer, and then through a global average pooling layer to compress semantics and focus on the overall distribution. Finally, the feature maps output from the three branches are aligned through multi-resolution upsampling and then integrated to output the final feature map.

6. The fine-grained target recognition method based on multi-view image manifold representation learning according to claim 5, characterized in that: Step S5 includes: S501, based on the training set, performs self-supervised joint training on the multi-view image manifold representation learning network model, and obtains the model weights under different training steps by iteratively updating the network model weights and uncertainty parameters by minimizing the total loss function. S502, test the model weights at each training step on the validation set, and calculate the Accuracy, Precision, Recall and F1 Score metrics respectively. S503 selects the weights corresponding to the highest values ​​of Accuracy, Precision, Recall, and F1 Score, or the highest value of Accuracy, as the optimal weights for the model.

7. The fine-grained target recognition method based on multi-view image manifold representation learning according to claim 6, characterized in that: In step S501, when training the multi-view image manifold representation learning network model, the upper limit of the number of training steps is set to 200, and the network model weights and uncertainty parameters are saved every 20 steps.

8. The fine-grained target recognition method based on multi-view image manifold representation learning according to claim 7, characterized in that: In step S1, four unmanned surface vessels navigate in an encircling pattern around the detected target, acquiring real-time images of the sea surface target covering a field of view from 0° to 360°, and constructing an infrared sea surface target dataset.

9. The fine-grained target recognition method based on multi-view image manifold representation learning according to claim 8, characterized in that: In step S1, the dataset is randomly divided into a training set, a validation set, and a test set in a ratio of 5:2:

3.

10. A fine-grained target recognition device based on multi-view image manifold representation learning, comprising a shore-based display and control platform, a shore-based radio station, and an unmanned surface vessel (USV). The USV is equipped with an optoelectronic pod, marine radar, GPS antenna, inertial measurement unit, network switch, onboard perception computing platform, USV core control board, meteorological sensor, onboard power supply, and motor driver. Its features include: The multi-view image manifold representation learning network model as described in any one of claims 1-9 is deployed on a shipboard perception computing platform.

Images