An intelligent multimodal marine object recognition system and method

Abstract

The present invention discloses an intelligent multimodal marine object recognition system and method. The system includes a data acquisition module that collects data and constructs a multimodal dataset; a cross-modal dataset construction module that constructs a cross-modal dataset; a feature extraction module that constructs a feature extraction model for data from different modalities and extracts features; a unimodal regularization module that uses Mahalanobis metric learning to obtain a unimodal feature extraction model with enhanced distinguishing capabilities; a feature fusion module that concatenates and fuses unimodal feature vectors; a multimodal regularization module that uses Mahalanobis metric learning to obtain a multimodal feature extraction model with enhanced distinguishing capabilities; and a recognition module that obtains the recognition probability of marine object target categories. By fusing three modal data and constructing unimodal and multimodal feature extraction models based on Mahalanobis metric learning, the system aims to overcome the limitations of existing technologies, enhance marine object recognition capabilities, reduce costs, enhance environmental adaptability, and improve recognition accuracy in complex scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent ships, and in particular to an intelligent multimodal marine object recognition system and method. Background Art

[0002] Accurate identification of maritime targets can be used to determine the safety risk levels of different encounter objects during the navigation of intelligent ships, and to make more reasonable and safer navigation decisions in a timely manner for different types of encounter objects, which plays a vital role in preventing ship collisions and planning more reasonable local paths.

[0003] To achieve autonomous path planning and collision avoidance, ships typically utilize a combination of sensors to enhance their ability to identify objects they encounter. Optical sensors capture image pixels rich in detail and semantic information; lidar (LiDAR) acquires point cloud data containing precise three-dimensional position and contour information; and navigation radar captures electromagnetic wave reflection data or radar image data from medium- and long-range targets. These sensor data exhibit typical multimodal characteristics, making multimodal data fusion difficult to achieve using existing multi-sensor data fusion methods. Furthermore, compared to land-based vehicles, the significant inertia and relatively low damping of ships sailing on the water significantly reduce their maneuverability, significantly shortening their reaction time to safety risks. Therefore, autonomous navigation demands even higher accuracy in identifying objects they encounter. Therefore, utilizing multimodal sensor data to improve the accuracy of identifying objects has always been a challenging task.

[0004] In recent years, through the efforts of researchers in related fields, this field has made exciting progress. For the problem of multimodal maritime object recognition, early research mainly relied on multi-sensor fusion to achieve target detection and tracking, focusing on identifying target attributes such as distance, direction, and speed. Although detecting and tracking the motion state of encounter objects is very necessary and critical for path planning of intelligent ships, it is far from enough to implement accurate and precise safety decisions. With the development of artificial intelligence, recognition methods based on a single data type, optical images, have made great progress in recent years. However, the quality of water surface images is often affected by weather and light, making the recognition of maritime objects based solely on single sensor data insufficiently robust in real-world applications. Utilizing multi-sensor data to further improve the perception capabilities of intelligent systems has relatively positive practical significance.

[0005] Methods for maritime target recognition: Early research has extracted multiple structured features from synthetic aperture radar (SAR) images to identify oil tankers and cargo ships. Jeong et al. used the AdaBoost method to fuse texture-based recognition results with those based on discrete Fourier transform features to improve target recognition accuracy. However, these methods, which require SAR remote sensing technology and facilities, are expensive and can only recognize a single target, making them difficult to apply to practical perception scenarios for intelligent ships. Regarding general target recognition, Li et al. proposed a data fusion method based on an improved DS evidence theory to fuse millimeter-wave radar and infrared images and applied it to an automatic target recognition system. This method, by improving the combination rules, resolves paradoxes caused by inconsistent evidence and inequalities, improving the effectiveness and accuracy of target recognition. Eitel et al. used two independent convolutional neural networks (CNNs) to fuse ordinary visible light images with depth images captured by a depth camera, achieving better recognition results than a single image. However, the algorithm was tested on only six types of structured targets. Kim et al. used a doubly weighted neural network to extract features from SAR and infrared images, and used linear and nonlinear fusion strategies to perform decision-level fusion processing on them. The experimental results showed that the nonlinear fusion method has better recognition effect.

[0006] Current limitations of multimodal maritime object recognition include: while target recognition methods based on optical remote sensing and SAR images have demonstrated capabilities for identifying maritime targets, they require expensive sensor equipment and are primarily used in the military, making them difficult to apply to intelligent ships. Object recognition technologies based on computer vision and deep learning have made significant progress in recent years. However, most rely on a single modality, images, and fail to leverage data from multiple sensors. Consequently, recognition accuracy remains insufficient to meet the practical requirements of intelligent ship navigation in complex scenarios. Therefore, a method is urgently needed that leverages the multi-sensor systems onboard intelligent ships to proactively and accurately identify encountered targets, thereby ensuring safe navigation. Summary of the Invention

[0007] The purpose of the present invention is to provide an intelligent multimodal marine object recognition system and method. The present invention integrates water surface optical images, radar images and lidar point cloud data, and uses Mahalanobis metric learning to optimize feature extraction and fusion processes, solving the problems of traditional single-sensor systems in complex marine environments, such as expensive equipment, low target recognition accuracy and poor robustness.

[0008] To achieve this goal, the present invention designs an intelligent multimodal marine object recognition system, which includes: a data acquisition module for collecting data in three modalities, namely optical images, laser point clouds and radar images, during the navigation process of the ship using a ship-borne optical camera, navigation radar and lidar, to construct a multimodal dataset;

[0009] The cross-modal dataset construction module is used to select corresponding samples from the water surface optical image, radar image and lidar point cloud data of the multimodal dataset for multimodal matching to form a cross-modal dataset;

[0010] The feature extraction module is used to construct a feature extraction model to extract features for different modal data in the cross-modal dataset;

[0011] The unimodal regularization module is used to construct a similarity metric function for each unimodality through Mahalanobis metric learning, and train the feature vectors of each unimodality as the regularization loss function of the fully connected layer to obtain a unimodal feature extraction model with enhanced discrimination ability, which is used to extract the feature vectors of each unimodality.

[0012] The feature fusion module is used to splice and fuse the single-modal feature vectors to obtain the spliced ​​and fused feature vectors;

[0013] The multimodal regularization module is used to construct a multimodal combination metric function through Mahalanobis metric learning, and uses it as the regularization loss function of the fully connected layer to train the concatenated and fused feature vectors, thereby obtaining a multimodal feature extraction model with enhanced distinguishing ability, which is used to extract multimodal feature vectors.

[0014] The recognition module model is used to map the multimodal feature vector to the specific marine object target category by adding a Softmax layer to obtain the marine object target category recognition probability.

[0015] The beneficial effects of the present invention are:

[0016] 1) By integrating data from multiple sensors, such as marine radar, lidar, and optical cameras, it can achieve active and accurate recognition of targets encountered at sea, thus making up for the shortcomings of single-modality recognition technology;

[0017] 2) Utilizing the multi-sensor system onboard smart ships eliminates the need for purchasing expensive additional sensor equipment, reducing system costs while increasing penetration and practicality in the smart ship field.

[0018] 3) The similarity measurement based on Mahalanobis distance is used to extract single-modal features and design loss functions for multi-modal feature fusion, which can effectively improve the feature representation ability of single modality and the common feature expression ability of multi-modality, which not only enhances the generalization ability of the model, but also improves the quality of fusion features. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a module workflow diagram of an intelligent multi-modal marine object recognition method of the present invention;

[0020] Figure 2 This is a flow chart of an intelligent multimodal marine object recognition method of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0022] Example 1:

[0023] like Figures 1-2 An intelligent multimodal marine object recognition system is shown, comprising:

[0024] The data acquisition module is used to collect data in three modalities: optical images, laser point clouds, and radar images during the ship's navigation process using ship-borne optical cameras, navigation radars, and lidars to construct a multimodal dataset.

[0025] The cross-modal dataset construction module is used to select corresponding samples from the water surface optical image, radar image and lidar point cloud data of the multimodal dataset for multimodal matching to form a cross-modal dataset;

[0026] The feature extraction module is used to construct a feature extraction model to extract features for different modal data in the cross-modal dataset;

[0027] The unimodal regularization module is used to construct a similarity measurement function for each unimodality through Mahalanobis metric learning. The similarity measurement function for each unimodality is used as the regularization loss function of the fully connected layer to train the feature vectors of each unimodality, thereby obtaining a unimodal feature extraction model with enhanced discrimination ability, which is used to extract the feature vectors of each unimodality.

[0028] The feature fusion module is used to splice and fuse the single-modal feature vectors to obtain the spliced ​​and fused feature vectors;

[0029] The multimodal regularization module is used to construct a multimodal combination metric function through Mahalanobis metric learning. The multimodal combination metric function is used as the regularization loss function of the fully connected layer to train the concatenated and fused feature vectors, thereby obtaining a multimodal feature extraction model with enhanced distinguishing ability, which is used to extract multimodal feature vectors.

[0030] The recognition module is used to map the multimodal feature vector to a specific marine object target category by adding a Softmax layer to obtain the marine object target category recognition probability.

[0031] In the above technical solution, in the data acquisition module, the specific process is:

[0032] The module includes a ship-borne optical camera, navigation radar, and lidar, which are used to collect water surface optical images, radar images, and lidar point cloud data while the ship is moving;

[0033] The three modal data are aligned using timestamps and spatial positions so that the three modalities describe the same scene or the same target in the time series. The aligned data constitute a multimodal dataset.

[0034] In the above technical solution, the specific construction process of the cross-modal dataset construction module is as follows: Marine object samples are selected from the multimodal dataset. Marine object categories include container ships, bulk carriers, passenger ships, yachts, and other marine objects. The LabelMe tool is used to annotate the marine objects in the optical images in the multimodal dataset with rectangular boxes and classification labels. The samples of marine objects of the three modalities form a cross-modal sample pair, and all cross-modal sample pairs are combined into a cross-modal dataset. For the modal data of marine radar and lidar, separate labeling is not required to reduce the labeling workload. The three modal data use the labeled information of the optical images for subsequent training and learning.

[0035] By using optical images with visual annotation information as labels for other modalities, weakly supervised metric learning is employed to accurately identify encountering objects, ultimately achieving target recognition results. This overcomes the inability of a single modality to leverage useful correlations between data from different modalities, while also reducing the time and labor required to manually label data from various modalities.

[0036] In the above technical solution, the feature extraction module is used to construct a feature extraction model to extract features for different modal data in the cross-modal dataset, including:

[0037] Optical image feature extraction: The backbone network ResNet50 is used as a feature extractor for optical image modal data to extract key features and obtain optical image modal features;

[0038] ResNet50 builds its architecture by stacking multiple residual blocks. Each residual unit consists of two convolutional layers and a skip connection, which directly adds the input to the output, forming residual learning. This structure enables ResNet50 to effectively solve the gradient vanishing problem in deep neural networks, allowing the network to become deeper and extract richer feature information, improving the model's expressiveness and accuracy. Specifically, the input optical image first passes through an initial convolutional layer, which uses a 7x7 convolution kernel with a stride of 2 and outputs 64 feature maps. It then undergoes batch normalization and a ReLU activation function. Next, the data passes through a 3x3 max pooling layer with a stride of 2 to reduce the spatial dimension. After that, the data enters four convolutions, each consisting of a different number of residual blocks. The number of convolutional layers and filters within these residual blocks gradually increases, starting from 64 and doubling with each stage until it reaches 512. Finally, after the optical image modality data is extracted through ResNet50, the output of each image is a feature vector with a length of 1024.

[0039] Laser point cloud feature extraction: After cleaning and denoising the laser point cloud modal data, performing coordinate conversion, depth mapping, and distortion correction, the laser point cloud modality is projected onto an image and input into a convolutional neural network for feature extraction to obtain the laser point cloud modal features.

[0040] Among them, the specific operations of preprocessing the lidar point cloud modal data and converting it into an image are: cleaning, denoising and removing outliers on the lidar point cloud modal data; using the rotation matrix and translation vector to determine the transformation relationship between the optical camera and the lidar; then using the principle of projection and the transformation relationship to map the three-dimensional coordinate depth in the optical camera coordinate system to the normalized image coordinates in the two-dimensional image coordinate system; and performing distortion correction on the depth-mapped image.

[0041] For the laser point cloud modality, in order to achieve the registration and alignment of the laser radar three-dimensional data with the optical image, it is first necessary to calibrate the camera and laser radar in the device. The calibration process includes determining the intrinsic parameters (such as focal length, optical center, etc.) and extrinsic parameters (the position and attitude of the camera relative to the reference coordinate system) of the camera, as well as the position and attitude of the laser radar relative to the same reference coordinate system. Through calibration, the relative position relationship between the camera and the laser radar in the device is obtained, that is, the transformation relationship between the laser radar coordinate system and the camera coordinate system is determined. The transformation relationship is described by the rotation matrix (Rotation Matrix) and the translation vector (Translation Vector). First, the original point cloud data is converted from the laser radar coordinate system to the camera coordinate system with the optical camera as the coordinate origin:

[0042] P camera =RP lidar +T;

[0043] Among them, P camera and P lidar They represent the position of a point in the camera coordinate system and the lidar coordinate system respectively. The rotation matrix R and translation vector T are obtained by calibration, which are a 3×3 matrix and a 3×1 vector respectively.

[0044] Subsequently, the original point cloud data is cleaned to remove noise points and outliers, and then depth mapping is performed. The camera's intrinsic parameters and the coordinates of the 3D points in the camera coordinate system are used to convert the 3D point coordinates in the camera coordinate system into normalized image coordinates in the 2D image coordinate system through the principle of projection:

[0045] Assuming that the coordinates of a point in the three-dimensional space in the camera coordinate system are (x, y, z), convert it to the normalized coordinates (u, v) in the image coordinate system:

[0046]

[0047]

[0048] Where f is the focal length of the camera, (c x ,c v ) is the principal point coordinate of the camera.

[0049] The depth-mapped image is then subjected to necessary distortion correction and further optimization to improve the quality of the two-dimensional image projected from the LiDAR data. Finally, a convolutional neural network model is constructed to extract features from the normalized image, obtaining a 1024-dimensional feature vector for the modal data.

[0050] Radar image feature extraction: De-noising is performed on the radar image modal data. After denoising, a convolutional neural network is constructed to perform feature extraction and obtain the radar image modal features.

[0051] For radar image modal data, the radar image is first denoised to remove speckle noise and background noise. In this system, the optical camera and the marine radar are installed in close proximity, and the scene they observe overlaps geographically. Therefore, the acquired image data exhibits high spatial consistency, fully preserving the mutual reference and complementarity between the two modal data. By adjusting the radar beamwidth and the optical camera's focal length, the spatial resolution of the two images is matched, bringing the radar and optical images closer together in detail, facilitating pixel-level comparison and analysis. Radar images can also use the same labels as optical images, reducing the annotation workload. For radar image data, since the image is formed from echo signals and is less information-intensive than optical images, a convolutional neural network is constructed for feature extraction to reduce the computational complexity of this modal data and improve the overall computational speed of the system. Finally, a 1024-byte feature vector is obtained for this modal data.

[0052] In the above technical solution, in the unimodal regularization module, during the feature extraction process for the three unimodal data, in order to constrain the consistency of the feature semantics of the different modal outputs, a similarity measurement function based on the Mahalanobis distance is constructed, specifically including:

[0053] The principal component analysis is used to reduce the eigenvector dimension of each modal data to 512 degrees, retaining the main components to reduce the computational complexity, and calculate the covariance matrix S and the corresponding inverse matrix S of the modal feature after dimensionality reduction. -1 , construct a single-modal feature extraction loss function based on Mahalanobis distance, and obtain each single-modal feature extraction model with enhanced discrimination ability by calculating the similarity between sample features to extract each single-modal feature vector;

[0054] Taking the optical image modality as an example, a modal similarity metric function based on Mahalanobis distance is constructed:

[0055]

[0056] Among them, S -1 It is the inverse of the covariance matrix S that is unique to the optical image modality, and x′ and y′ represent the eigenvectors extracted from the modality data;

[0057] In order to improve the distinguishability of internal features in unimodal data, a similarity metric function based on Mahalanobis distance is used as the loss function of the unimodal feature network to quantify the similarity between sample features. The more similar the unimodal samples are, the shorter the distance between the feature vectors extracted by the neural network is. The more dissimilar the samples are, the greater the distance between the feature vectors is. The similarity metric function based on Mahalanobis distance improves the feature expression of unimodal data by a single neural network by considering the distribution characteristics of unimodal features. The unimodal feature extraction loss function is:

[0058] L Mahalanobis (a,p,n)=log(1+exp(D M (a,p)-D M (a,n)+margin))+γ1·I;

[0059] Among them, a, p, and n represent a set of samples of single-modal data (such as optical image modal data, laser point cloud modal data, and radar image modal data), p and n represent positive samples that are closer to sample a and negative samples that are farther away, respectively. M (a,p),D M (a,n) represents the Mahalanobis distance between the unimodal sample a and the closer positive sample p and the farther negative sample n, respectively; margin is a hyperparameter used to ensure the distance gap between positive and negative samples; the γ1·I regularization term is added to balance the loss function of the unimodal feature extraction network, where I is the unit matrix and γ1 is a smaller hyperparameter used to ensure the effectiveness of the loss function during network propagation.

[0060] The above-mentioned unimodal network loss function is used to train three unimodal feature extraction network models respectively to obtain a unimodal feature extraction model with enhanced discrimination ability. The model can be used to extract the optical image modal feature vector, radar image modal feature vector and laser point cloud modal feature vector of the input data.

[0061] In the above technical solution, the feature fusion module integrates the unimodal feature vectors into a unified framework and trains them through a joint optimization strategy, specifically including:

[0062] The three unimodal feature vectors are concatenated through a splicing operation, and the resulting common feature vector undergoes a nonlinear transformation through a fusion layer to obtain a fused feature vector. The three unimodal feature extraction networks are able to extract discriminative feature vectors. Multimodal feature fusion and joint optimization are then performed to integrate the feature vectors extracted by network models of different modalities into a unified framework. First, the three unimodal vectors are concatenated into a feature vector of shape (512, 3) through a splicing operation. Subsequently, two fully connected layers are connected as a fusion layer to fuse the three unimodal feature vectors. The fusion layer performs a nonlinear transformation on the concatenated feature vectors.

[0063] In the above technical solution, in the multimodal regularization module, a multimodal combination metric function based on Mahalanobis distance is constructed, and it is used as a regularization loss function to fuse the concatenated feature vectors to obtain a multimodal feature extraction model with distinguishing ability. The loss function of the multimodal combination metric based on Mahalanobis distance is specifically:

[0064]

[0065] Among them, L c / r is the loss of classification and regression tasks, λ and γ2 are weights used to count the differences between different modal features, f(x) is the mapping function of the fusion layer network to the input feature x, that is, the multimodal feature after fusion, D M (f(x i ),f(x j )) is represented as the feature vector f(x) after multimodal fusion i ) and f(x j )Mahalanobis distance.

[0066] The data acquisition module collects three modal data, constructs a multimodal dataset, uses the multimodal dataset to select corresponding samples for matching, uses the optical image to annotate marine objects with rectangular boxes and classification labels, uses the annotation information of the optical image for the three modal data, and constructs samples of marine objects of the three modalities into cross-modal sample pairs. All cross-modal sample pairs are combined into a cross-modal dataset, and the cross-modal dataset is input into the feature extraction module for training. The unimodal attitude metric function regularization module is used to enhance the distinguishing ability of unimodal features, and extracts each unimodal feature vector, splices each unimodal feature vector, and inputs the spliced ​​unimodal feature vectors into the multimodal regularization module for fusion to enhance features with multimodal expression capabilities, extract multimodal feature vectors, and convert the multimodal feature vectors into probability distribution form through the added Softmax layer to obtain the target recognition result. Through the above content, a trained multimodal attitude metric learning model is obtained.

[0067] During the testing phase, different modal data in the same scenario are input into the trained multimodal attitude metric learning model to predict the recognition results of the encounter objects.

[0068] By comparing and analyzing the performance differences between single-modal and multimodal marine object recognition methods, it was found that under the same conditions, the multimodal recognition method outperformed the single-modal recognition method in terms of recognition accuracy. When affected by other factors and marine radar and lidar data were not available, the device degenerated into a single-modal target detection algorithm based on optical images. Through experimental comparison, the device's target detection accuracy in the public dataset Seaship7000, when only optical images were available, was comparable to that of mainstream target detection algorithms, with a recognition accuracy of 95.7%. In the multimodal dataset collected by this device, the recognition accuracy of marine objects reached 97.7%, and when only optical image target detection was performed, the recognition accuracy was 95.4%. The method of the present invention outperforms target detection algorithms using a single optical modality.

[0069] Example 2:

[0070] An intelligent multimodal marine object recognition method comprises the following steps:

[0071] The data acquisition module is used to collect data and build multimodal datasets;

[0072] Corresponding samples are selected from the water surface optical image, radar image and lidar point cloud data of the multimodal dataset for multimodal matching to form a cross-modal dataset;

[0073] The feature extraction module is used to construct a feature extraction model to extract features for different modal data in the cross-modal dataset;

[0074] Through Mahalanobis metric learning, a similarity measurement function is constructed for each single modality, and the feature vectors of each single modality are trained as the regularization loss function of the fully connected layer to obtain a single modality feature extraction model with enhanced discrimination ability, which is used to extract the feature vectors of each single modality.

[0075] Each single-modal feature vector is spliced ​​and fused to obtain a spliced ​​and fused feature vector;

[0076] Through Mahalanobis metric learning, a multimodal combined metric function is constructed and used as the regularization loss function of the fully connected layer to train the concatenated and fused feature vectors, thereby obtaining a multimodal feature extraction model with enhanced distinguishing ability, which is used to extract multimodal feature vectors.

[0077] By adding a Softmax layer, the multimodal feature vector is mapped to the specific marine object target category to obtain the marine object target category recognition probability.

[0078] This method collects multimodal data through three-modal data acquisition devices, and accurately identifies marine objects through a single-modal feature extraction and multimodal feature fusion framework. Unlike traditional methods that simply concatenate or weighted sum feature vectors of different modalities, this method achieves adaptive selection and weighting of different modal features by designing a specific fusion layer, thereby more effectively extracting and utilizing complementary information in multimodal data. Secondly, in the design of single-modal and multimodal measurement functions, a similarity measure based on Mahalanobis distance is used for single-modal feature extraction and multimodal feature fusion loss function design, which can effectively improve the feature representation ability of single modality and the common feature expression ability of multimodality. It not only enhances the generalization ability of the model, but also improves the quality of fused features.

[0079] This intelligent multimodal marine object recognition method integrates surface optical images, radar images, and lidar point cloud data to construct a cross-modal dataset. A feature extraction model is used to extract features from the multimodal data. Through Mahalanobis metric learning, similarity measurement functions are constructed at the unimodal and multimodal levels, respectively. These similarity measurement functions are used to construct loss functions to train the unimodal and multimodal feature extraction models, enhancing the models' ability to distinguish between different types of targets. This method, combined with a Softmax classifier, accurately classifies marine objects. This method not only improves recognition accuracy but also enhances robustness in complex environments.

[0080] By integrating multiple sensing methods, it can more comprehensively capture target characteristics and improve recognition accuracy. Using Mahalanobis metric learning to optimize feature representation, it maintains good performance even under large lighting variations and adverse weather conditions. This method provides strong technical support for fields such as ocean monitoring and navigation safety, helping to enhance the intelligence level of related industries. Therefore, this method is of great significance for promoting the effective management and protection of marine resources.

[0081] Example 3:

[0082] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0083] Example 4:

[0084] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the steps of the above method are implemented.

[0085] The contents not described in detail in this specification belong to the prior art known to those skilled in the art. It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0086] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0087] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0088] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit its scope of protection. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that after reading the present invention, those skilled in the art may still make various changes, modifications or equivalent substitutions to the specific implementation methods of the invention, but these changes, modifications or equivalent substitutions are all within the scope of protection of the pending claims of the invention.

[0090] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.

Claims

1. An intelligent multimodal marine object recognition system, characterized in that: include: The data acquisition module is used to construct a multimodal dataset using water surface optical images, radar images, and lidar point cloud data; The cross-modal dataset construction module is used to select corresponding samples from the water surface optical image, radar image and lidar point cloud data of the multimodal dataset for multimodal matching to form a cross-modal dataset; The feature extraction module is used to construct a feature extraction model to extract features for different modal data in the cross-modal dataset; The unimodal regularization module is used to construct a similarity metric function for each unimodality through Mahalanobis metric learning, and train the feature vectors of each unimodality as the regularization loss function of the fully connected layer to obtain a unimodal feature extraction model with enhanced discrimination ability, which is used to extract the feature vectors of each unimodality. The feature fusion module is used to splice the single-modal feature vectors and fuse them through the multimodal regularization module to obtain the fused feature vector; The multimodal regularization module is used to construct a multimodal combination metric function through Mahalanobis metric learning, and uses it as the regularization loss function of the fully connected layer to train the concatenated and fused feature vectors, thereby obtaining a multimodal feature extraction model with enhanced distinguishing ability, which is used to extract multimodal feature vectors. The recognition module is used to map the multimodal feature vector to a specific marine object target category by adding a Softmax layer to obtain the marine object target category recognition probability.

2. The intelligent multimodal marine object recognition system according to claim 1, characterized in that: During data collection, the specific process is as follows: The system includes ship-borne optical cameras, navigation radars, and lidars, which are used to collect water surface optical images, radar images, and lidar point cloud data while the ship is moving; The three modal data are aligned using timestamps and spatial positions so that the three modalities describe the same scene or the same target in the time series. The aligned data constitute a multimodal dataset.

3. The intelligent multimodal marine object recognition system according to claim 1, characterized in that: In the cross-modal dataset construction module, the specific construction process is as follows: in the multimodal dataset, marine object samples are selected, and the marine objects in the optical images in the multimodal dataset are annotated with rectangular frames and classification labels. The samples of marine objects of the three modalities constitute a cross-modal sample pair, and all cross-modal sample pairs are combined into a cross-modal dataset.

4. The intelligent multimodal marine object recognition system according to claim 1, characterized in that: The feature extraction module is used to build a feature extraction model for different modal data in the cross-modal dataset to extract features, including: Optical image feature extraction: The backbone network ResNet50 is used as a feature extractor for optical image modal data to extract key features and obtain optical image modal features; Laser point cloud feature extraction: After cleaning and denoising the laser point cloud modal data, performing coordinate conversion, depth mapping, and distortion correction, the laser point cloud modality is projected onto an image and input into a convolutional neural network for feature extraction to obtain the laser point cloud modal features. The specific operations for preprocessing and converting the LiDAR point cloud modal data into images are as follows: cleaning, denoising, and removing outliers on the LiDAR point cloud modal data; using the rotation matrix and translation vector to determine the transformation relationship between the optical camera and the LiDAR; then using the principle of projection and the transformation relationship to map the three-dimensional coordinate depth in the optical camera coordinate system to normalized image coordinates in the two-dimensional image coordinate system; and performing distortion correction on the depth-mapped image. Radar image feature extraction: De-noising is performed on the radar image modal data. After denoising, a convolutional neural network is constructed to perform feature extraction and obtain the radar image modal features.

5. The intelligent multimodal marine object recognition system according to claim 1, characterized in that: In the single-mode regularization module, the eigenvector of each modal data is reduced in dimension, and the covariance matrix S and the corresponding inverse matrix S of the modal feature after dimensionality reduction are calculated. -1 , construct a unimodal similarity measurement function based on Mahalanobis distance as the unimodal feature extraction loss function, calculate the similarity between sample features, enhance the distinguishability of internal features under unimodality, and obtain each unimodal feature vector with enhanced distinguishability. The loss function of the unimodal feature extraction model is: L Mahalanobis (a,p,n)=log(1+exp(D M (a,p)-D M (a,n)+margin))+γ1·I; Among them, a, p, and n represent a set of samples of unimodal data, p and n represent positive samples that are closer to sample a and negative samples that are farther away, respectively. M (a,p),D M (a,n) represents the Mahalanobis distance between the unimodal sample a and the closer positive sample p and the farther negative sample n, respectively; margin is a hyperparameter used to ensure the distance gap between positive and negative samples; the γ1·I regularization term is added to balance the loss function of the unimodal feature extraction network, where I is the unit matrix and γ1 is a small regularization parameter.

6. The intelligent multimodal marine object recognition system according to claim 1, characterized in that: In the feature fusion module, the unimodal feature vectors are integrated into a unified framework and trained through a joint optimization strategy, including: Through the splicing operation, the three single-modal feature vectors are spliced ​​together, and the spliced ​​common feature vector is nonlinearly transformed through the fusion layer to obtain the fused feature vector.

7. The intelligent multimodal marine object recognition system according to claim 1, characterized in that: In the multimodal regularization module, a multimodal combination metric function based on Mahalanobis distance is constructed and used as a regularization loss function to train the fused feature vector to obtain common features with distinguishing ability. The loss function of the multimodal combination metric based on Mahalanobis distance is specifically: Among them, L c / r is the loss of classification and regression tasks, λ and γ2 are parameters used to count the differences between different modal features, f(x) is the mapping function of the fusion layer network to the input feature x, and D M (f(x i ),f(x j )) is represented as the feature vector f(x i ) and f(x j ) is the Mahalanobis distance between them.

8. An intelligent multimodal marine object recognition method, characterized in that: The steps include: The data acquisition module is used to collect data and build multimodal datasets; Corresponding samples are selected from the water surface optical image, radar image and lidar point cloud data of the multimodal dataset for multimodal matching to form a cross-modal dataset; The feature extraction module is used to construct a feature extraction model to extract features for different modal data in the cross-modal dataset; Through Mahalanobis metric learning, a unimodal feature extraction loss function based on Mahalanobis distance is constructed, and the feature vectors of each unimodality are trained to obtain a unimodal feature extraction model with enhanced discrimination ability, which is used to extract each unimodal feature vector; Each single-modal feature vector is spliced ​​and fused to obtain a spliced ​​and fused feature vector; Through Mahalanobis metric learning, a multimodal combined metric function is constructed and used as the regularization loss function of the fully connected layer to train the concatenated and fused feature vectors, thereby obtaining a multimodal feature extraction model with enhanced distinguishing ability, which is used to extract multimodal feature vectors. By adding a Softmax layer, the multimodal feature vector is mapped to the specific marine object target category to obtain the marine object target category recognition probability.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 8 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claim 8 are implemented.

Images