Multi-mode ship intelligent identification system for coping with complex marine environment

Abstract

The invention provides a multi-mode ship intelligent identification system for coping with a complex marine environment, and the system comprises a data collection and enhancement module which is used for collecting ship pictures and corresponding auxiliary information, and carrying out the enhancement of the ship pictures; the image definition improving module is used for improving the definition of the enhanced image; the image segmentation module is used for carrying out ship identification and background separation on the enhanced picture; and the multi-modal classification module is used for classifying and identifying the ship pictures and determining ship models. Through multi-modal data fusion, an advanced image processing technology, an innovative classification model design and detailed knowledge base construction, the accuracy, adaptability and efficiency of marine ship identification are greatly improved.

Description

Technical Field

[0001] The present invention belongs to target recognition technology, and in particular to a multi-mode ship intelligent recognition system for coping with complex marine environments. Background Art

[0002] Ship identification not only directly supports intelligence surveillance and reconnaissance missions, but also plays a key role in assessing maritime security situations, optimizing maritime traffic management, and protecting marine resources. However, the complex maritime environment, such as changes in lighting, weather conditions, wave interference, and the influence of shooting distance, poses a huge challenge to accurately identifying ships from real-shot images. Traditional ship identification methods mainly rely on manual review, which is not only inefficient, but also easily interfered by environmental noise, making it difficult to ensure high accuracy and real-time performance. With the development of machine learning technologies such as convolutional neural networks (CNNs), automated image recognition has solved this problem to a certain extent. However, single-modality image data still faces problems of accuracy and reliability when processing low-resolution and high-noise maritime ship images. Summary of the invention

[0003] The present invention proposes a multi-mode ship intelligent identification system for coping with complex marine environments.

[0004] The technical solution to achieve the purpose of the present invention is: a multi-mode ship intelligent identification system for coping with complex marine environments, comprising:

[0005] Data acquisition and enhancement module, used to collect ship images and corresponding auxiliary information and enhance the ship images;

[0006] Image segmentation module, used to identify ships and separate background from enhanced images;

[0007] The multimodal classification module is used to classify and identify ship images and determine the ship models.

[0008] Preferably, the specific method for enhancing the ship image is:

[0009] All images were converted to square format by adding padding areas to the edges of the images to ensure consistent proportions of the ship while normalizing the images;

[0010] Gaussian filter, median filter and mean filter are used to enhance the filled image respectively to obtain images with different clarity and noise levels.

[0011] Preferably, the specific method for performing ship recognition and background separation on the enhanced image is: confirming the preliminary position of the hull based on the YOLO target detection algorithm, performing semantic segmentation through SAM, and separating the hull from the background.

[0012] Preferably, the multimodal classification module adopts a trained multimodal classification model to classify and identify ship pictures. The multimodal classification model includes an image feature extraction module, a text feature extraction module and a weighted fusion module. The image feature extraction module adopts a ResNeXt model to extract image features, and the text feature extraction module adopts a word embedding layer to extract text features of the hull number. The weighted fusion module adopts an attention mechanism to dynamically adjust the weights of image and text features, and performs weighted fusion of image features and text information.

[0013] Preferably, a CBAM module is embedded in each residual block of the ResNeXt model, and each CBAM module includes a channel attention submodule and a spatial attention submodule. The feature map output by the convolutional layer of each residual block is first subjected to global average pooling and global maximum pooling by the channel attention submodule to calculate the importance weight of each channel, and the feature map is weighted using the importance weight; the spatial attention submodule performs maximum pooling and average pooling operations on the feature map processed by the channel attention submodule to generate a spatial attention weight map.

[0014] Preferably, the specific formula for weighted fusion of image features and text information by the weighted fusion module is:

[0015]

[0016] In the formula, represents the image features extracted by the image feature extraction module, represents the image feature weight, represents the text features extracted by the text feature extraction module, represents the text feature weight, is the characteristic dimension.

[0017] Preferably, the specific method for training the multimodal classification model is:

[0018] Use the data acquisition and enhancement module to collect and enhance ship images;

[0019] The image segmentation module is used to obtain the segmented ship images and construct them as training sets. The hull numbers that may appear in the current ship model are randomly selected and input into the multimodal classification model in the form of text as auxiliary information.

[0020] The supervised learning method is used for training, the cross entropy loss function is used as the optimization target, and the Adam optimizer is used for training.

[0021] Preferably, the auxiliary information includes hull number, country and region.

[0022] Compared with the prior art, the present invention has the following significant advantages:

[0023] 1. Enhanced recognition accuracy: Multimodal data fusion and advanced image processing techniques enable the model to analyze ship images more comprehensively and accurately, thereby significantly improving recognition accuracy, especially in low-quality images and complex environments.

[0024] 2. Improved environmental adaptability: The combination of target detection and semantic segmentation technology enables the model to demonstrate excellent adaptability in dealing with the changing marine environment and can effectively handle various background interferences and environmental changes.

[0025] 3. Improved data processing efficiency: The model that integrates multiple technologies not only improves the processing efficiency of image data, but also optimizes the analysis process of hull number information. This improvement means that the model can process and analyze large amounts of data more quickly, especially in practical application scenarios that require rapid response.

[0026] 4. Fine-grained classification performance: By introducing the CBAM module and the attention mechanism to fuse auxiliary information, the performance of the model in the fine-grained ship classification task is significantly enhanced, and the ability to recognize tiny but critical features is improved.

[0027] The present invention is described in further detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 Schematic diagram of the CBAM structure.

[0029] Figure 2 Schematic diagram of the structure of integrating CBAM modules into ResNext basic blocks.

[0030] Figure 3 Schematic diagram of the multimodal classification model structure. DETAILED DESCRIPTION

[0031] A multi-mode ship intelligent identification system for coping with complex marine environments, comprising:

[0032] Data acquisition and enhancement module, used to collect ship images and corresponding auxiliary information and enhance the ship images;

[0033] Image segmentation module, used to identify ships and separate background from enhanced images;

[0034] The multimodal classification module is used to classify and identify ship images and determine the ship models.

[0035] In a further embodiment, the specific method for performing ship recognition and background separation on the enhanced image is:

[0036] The method of combining YOLO target detection with SAM image segmentation model is used to achieve accurate recognition and background segmentation of ship images. First, YOLO target detection technology is used to quickly detect and locate ship targets in the image and generate a bounding box containing the ship. With its fast and high-precision characteristics, YOLO can effectively extract target areas from complex images. After YOLO generates the bounding box, the SAM image segmentation model is used to perform fine segmentation within the bounding box, thereby achieving accurate recognition and background segmentation of ship images. SAM can accurately remove background noise and only retain pixels related to the ship, ensuring that details are fully preserved.

[0037] In a further embodiment, the multimodal classification module uses a trained multimodal classification model to classify and identify ship images. The multimodal classification model includes an image feature extraction module, a text feature extraction module and a weighted fusion module. The image feature extraction module uses a ResNeXt model to extract image features, and the text feature extraction module uses a word embedding layer to extract text features of the hull number. The weighted fusion module uses an attention mechanism to dynamically adjust the weights of image and text features, and weightedly fuses image features and text information. The present invention uses an attention mechanism to dynamically adjust the importance of image and text features, and optimizes classification performance based on the relationship between them. This multimodal classifier can understand and analyze data more accurately and achieve efficient classification of complex scenes.

[0038] In a further embodiment, a CBAM module is embedded in each residual block of the ResNeXt model, and the CBAM is inserted after the convolution operation to directly process the feature map output by the convolution layer, so that the CBAM can perform attention modulation on the feature map at different stages of feature extraction. Each CBAM module includes a channel attention submodule and a spatial attention submodule. The feature map output by the convolution layer of each residual block is first subjected to global average pooling and global maximum pooling by the channel attention submodule to calculate the importance weight of each channel, and the feature map is weighted using the importance weight; the spatial attention submodule performs maximum pooling and average pooling operations on the feature map processed by the channel attention submodule to generate a spatial attention weight map, further emphasizing important spatial areas. The present invention integrates the CBAM module in the ResNeXt model to enhance the ability of fine-grained image classification.

[0039] In a further embodiment, the specific formula for weighted fusion of image features and text information by the weighted fusion module is:

[0040]

[0041] In the formula, represents the image features extracted by the image feature extraction module, represents the image feature weight, represents the text features extracted by the text feature extraction module, represents the text feature weight, is the characteristic dimension.

[0042] In a further embodiment, when training the multimodal classification model, the data acquisition and enhancement module is used to collect and enhance the ship images, specifically:

[0043] By comprehensively utilizing various information sources, the current status of the Navy's ships was determined, and 131 types of ships were selected for the construction of the data set. These data record in detail the sub-class information, code, hull number and service status of the ship, and use the hull number, country and region as auxiliary information.

[0044] First, we rely on a web crawler developed based on Python, which collects ship images from search engines. In order to expand the scope of data collection, the crawler uses the Chinese and English names of ships as search keywords. The collected images are stored in a local image pool. Then, using the Baichuan big model deployed locally, the crawled image titles and search keywords are analyzed for correlation to ensure that the image content is highly relevant to the ship name. Irrelevant images will be deleted. Finally, through manual screening, ensure that the images correspond to the models, and classify the images into long-distance blurry images and close-range high-definition images according to the actual sea area shooting conditions of the ships.

[0045] The present invention designs a complete set of ship image dataset construction processes. Through the above steps, 3784 high-quality ship images are obtained as a dataset to build a comprehensive ship knowledge base.

[0046] In the training phase of a multimodal classification model, it is usually required to input images of a fixed size to ensure the consistency of feature extraction and subsequent processing. However, the original images usually have different sizes and proportions. Although traditional image conversion methods such as scaling and cropping can adjust the image size, they often cause distortion of the target shape or loss of some information. In order to unify the image size without destroying the original proportions and morphological features of the target, the present invention adopts a filling method. By adding appropriate filling areas at the edges of the image, all images are converted to a square format. This filling method not only maintains the integrity of the target, but also meets the requirements of the neural network model for the input size, thereby enhancing the generalization ability of the model.

[0047] Subsequently, in order to improve the generalization ability of the model, the present invention adopts a variety of image enhancement techniques, including Gaussian filter, median filter and mean filter, to enhance the padded image to simulate the image effects under different clarity and noise levels, providing more diverse training samples for the model.

[0048] In a further embodiment, the multimodal classification model training adopts a supervised learning method. During the training process, the cross entropy loss function is used as the main optimization target to measure the classification accuracy of the model. In order to accelerate the convergence of the model, the Adam optimizer is selected, and its adaptive learning rate helps to maintain the optimization efficiency at different training stages.

[0049] In terms of training strategy, phased training is adopted. First, the ResNeXt model integrated with CBAM is pre-trained to learn rich visual features on image data. Then, the text embedding module is combined with the image feature extraction part to perform joint training of multimodal feature fusion. In this process, the weight of the attention mechanism is dynamically adjusted to enhance the model's ability to learn the associated features of images and texts.

[0050] To further optimize the performance of the model, we introduced a cosine annealing learning rate scheduling strategy to gradually reduce the learning rate during training to ensure that the model can better stabilize and converge in the later stages of training. In addition, we used a checkpoint reset strategy to save the best state of the model during training, prevent overfitting, and ensure the stability and reliability of the model in practical applications.

[0051] In a further embodiment, when performing real-time prediction, the specific method for the data acquisition and enhancement module to enhance the ship image is:

[0052] Real-ESRGAN technology is used to enhance predicted images, which is a method of super-resolution reconstruction of images through deep learning models. This technology can process low-resolution and high-noise images into high-resolution versions, significantly enhance the details in the image, make it easier for the model to capture key features, and thus improve recognition effects in various scenarios.

[0053] After improving the clarity of the image to be predicted, the image segmentation module is used to separate the ships and input them into the trained multimodal classification model for real-time prediction.

[0054] The present invention greatly improves the accuracy, adaptability and efficiency of maritime ship identification through multimodal data fusion, advanced image processing technology, innovative classification model design and detailed knowledge base construction.

[0055] Example

[0056] When conducting fine-grained recognition experiments, the present invention uses single-class classification accuracy Acc and average classification accuracy mAc as evaluation indicators to measure the performance of the model. Single-class classification accuracy refers to the proportion of correct classifications in a single category, and its calculation formula is as follows:

[0057]

[0058] in, is the sample size of the current test set, is an indicative function, when When true, ,otherwise The mean classification accuracy (mAcc) is obtained by calculating the average classification accuracy of all categories.

[0059] The embodiments are compared with the following models:

[0060] (1) Comparison of ResNext model, ResNext-CBAM model and ResNext-CBAM-hull number model

[0061] In this experiment, the ResNext, ResNext-CBAM and ResNext-CBAM-hull number models were trained, and a series of comparative experiments were conducted to evaluate their performance in the ship recognition task.

[0062] ResNext model: ResNext is an improved convolutional neural network structure that enhances the feature extraction capability of the model by introducing residual connections and multi-path structures.

[0063] ResNext-CBAM model: The CBAM (Convolutional BlockAttention Module) module is integrated on the basis of the ResNext model, and the feature expression ability of the model is further improved through the channel attention and spatial attention mechanisms.

[0064] ResNext-CBAM-Hull Number Model: Based on the ResNext-CBAM model, the hull number information is further combined, and the dual-input network architecture and attention mechanism are used for feature fusion, which enhances the model's ability to recognize fine-grained features.

[0065] By observing the loss reduction of the three models during the training process and the accuracy performance on the test set, the results show that the ResNext-CBAM-Hull Number model is significantly better than the ordinary ResNext model in terms of loss reduction speed. At the same time, on the test set, the accuracy of the ResNext-CBAM-Hull Number model is also significantly higher than other models.

[0066] The experimental results are shown in the following table:

[0067] Model Type Test set accuracy ResNext 58.2% ResNext-CBAM 61.6% ResNext-CBAM-hull number 91.6%

[0068] The experimental results show that the ResNext-CBAM model enhances the model's feature extraction capability and thus improves recognition performance by embedding the CBAM module in the ResNext architecture. This structural improvement enables the ResNext-CBAM model to achieve an accuracy of 61.6% on the test set, which is a significant improvement over the original ResNext model.

[0069] Furthermore, the ResNext-CBAM-hull number model uses a combination of hull number information and attention mechanism, with an accuracy rate of 91.6% on the test set, far exceeding the ResNext and ResNext-CBAM models. This proves the significant contribution of the attention mechanism in fusing ship images and hull number information in improving the accuracy of ship recognition.

[0070] Model comparison on super-resolution test set:

[0071] The present invention also compares the model performance on the test set after super-resolution processing. In the experiment, the Yolo-CA ship detection model of 131 categories was used to test on the data set after super-resolution processing.

[0072] The experimental results are shown in the following table:

[0073] Model Type Test set accuracy Super-resolution-test set accuracy Yolo-CA 17.3% 21.4% ResNext 58.2% 62.1% ResNext-CBAM 61.6% 65.7% ResNext-CBAM-hull number 91.6% 97.4%

[0074] The test set accuracy of the Yolo-CA model increased from 17.3% to 21.4%. The accuracy of the ResNext model increased from 58.2% to 62.1%, and the accuracy of the ResNext-CBAM model with the integrated CBAM attention module increased from 61.6% to 65.7%, demonstrating the basic effect of super-resolution processing on performance improvement. The ResNext-CBAM-Side Number model achieved a leap forward from a high baseline accuracy of 91.6% to 97.4% after super-resolution processing.

[0075] These results not only highlight the ability of super-resolution technology in improving image recognition accuracy, but also prove the outstanding advantages of using the attention mechanism to fuse hull number information in ship recognition scenarios. Through these comparative experimental results, we can clearly see the contribution of each module to the accuracy. First, the performance improvement of the super-resolution module is overall, and various recognition methods benefit from it; second, the performance of the YOLOv8 model alone in complex scenes is significantly lower than the combined method of target detection, background segmentation and image classification; finally, the performance of the classification model that uses the attention mechanism to fuse multimodal information is further improved.

Claims

1. A multi-mode ship intelligent identification system for coping with complex marine environments, characterized in that: include: Data acquisition and enhancement module, used to collect ship images and corresponding auxiliary information and enhance the ship images; Image segmentation module, used to identify ships and separate background from enhanced images; The multimodal classification module is used to classify and identify ship images and determine the ship models.

2. The multi-mode ship intelligent identification system for coping with complex marine environments according to claim 1 is characterized in that: The specific method for ship recognition and background separation of the enhanced image is as follows: the preliminary position of the hull is confirmed based on the YOLO target detection algorithm, and the hull is separated from the background through semantic segmentation through SAM.

3. The multi-mode ship intelligent identification system for coping with complex marine environments according to claim 1 is characterized in that: The multimodal classification module uses a trained multimodal classification model to classify and identify ship pictures. The multimodal classification model includes an image feature extraction module, a text feature extraction module and a weighted fusion module. The image feature extraction module uses a ResNeXt model to extract image features, and the text feature extraction module uses a word embedding layer to extract text features of the hull number. The weighted fusion module uses an attention mechanism to dynamically adjust the weights of image and text features, and performs weighted fusion of image features and text information.

4. The multi-mode ship intelligent identification system for coping with complex marine environments according to claim 3 is characterized in that: A CBAM module is embedded in each residual block of the ResNeXt model. Each CBAM module includes a channel attention submodule and a spatial attention submodule. The feature map output by the convolutional layer of each residual block is first subjected to global average pooling and global maximum pooling by the channel attention submodule to calculate the importance weight of each channel, and the feature map is weighted using the importance weight; The spatial attention submodule performs maximum pooling and average pooling operations on the feature map processed by the channel attention submodule to generate a spatial attention weight map.

5. The multi-mode ship intelligent identification system for coping with complex marine environments according to claim 3 is characterized in that: The specific formula for weighted fusion of image features and text information by the weighted fusion module is: In the formula, represents the image features extracted by the image feature extraction module, represents the image feature weight, represents the text features extracted by the text feature extraction module, represents the text feature weight, is the characteristic dimension.

6. The multi-mode ship intelligent identification system for coping with complex marine environments according to claim 4 is characterized in that: The specific method for training the multimodal classification model is: Use the data acquisition and enhancement module to collect and enhance ship images; The image segmentation module is used to obtain the segmented ship images and construct them as training sets. The hull numbers that may appear in the current ship model are randomly selected and input into the multimodal classification model in the form of text as auxiliary information. The supervised learning method is used for training, the cross entropy loss function is used as the optimization target, and the Adam optimizer is used for training.

7. The multi-mode ship intelligent identification system for coping with complex marine environments according to claim 3 is characterized in that: When the collected images are used for model training, the specific method for the data collection and enhancement module to enhance the ship images is as follows: All images were converted to square format by adding padding areas to the edges of the images to ensure consistent proportions of the ship while normalizing the images; Gaussian filter, median filter and mean filter are used to enhance the filled image respectively to obtain images with different clarity and noise levels.

8. The multi-mode ship intelligent identification system for coping with complex marine environments according to claim 3 is characterized in that: When the collected pictures are used for real-time prediction, the specific method for the data collection and enhancement module to enhance the ship pictures is: using the Real-ESRGAN model to perform super-resolution reconstruction on the images.

9. The multi-mode ship intelligent identification system for coping with complex marine environments according to claim 1 is characterized in that: The auxiliary information includes hull number, country and region.

Images