A multimodal ship fine-grained identification method and system

Abstract

This application discloses a multimodal fine-grained ship identification method and system, relating to the technical field of fine-grained ship identification and classification. The method includes acquiring multimodal data of a target area and performing data augmentation preprocessing to construct a training set and a test set; employing a multi-source feature extraction method based on physical degradation modeling for feature extraction; employing a discriminable feature learning method based on spatiotemporal consistency alignment for node feature update; employing a multimodal hybrid contrast loss function as a loss function to train a class prediction model; and obtaining a trained class prediction model after iterative training. The updated node features of each test sample are input into the trained class prediction model to obtain a final class prediction probability. This application can improve the accuracy of fine-grained ship identification.

Description

Technical Field

[0001] The present application relates to the technical field of fine-grained ship identification and classification, and in particular to a multimodal fine-grained ship identification method and system. Background Art

[0002] Research on fine-grained ship classification uses deep learning and other technologies to accurately identify ship types. It is a core technology for intelligent ocean governance and can support multiple needs, including shipping safety. With technological advancements, single-sensor, single-modality ship target type recognition has gradually reached its performance ceiling. However, there is still significant room for exploration in multi-sensor, cross-modality, integrated ship target type recognition.

[0003] Multimodal image fusion technology can leverage the combined strengths of ISAR (Inverse Synthetic Aperture Radar), infrared, and visible light images to improve recognition performance. However, in practical applications for fine-grained ship classification, atmospheric transmission effects and the high similarity of ship shapes result in poor robustness and low accuracy for multimodal image-based fine-grained classification. Specifically, atmospheric turbulence, absorption, and scattering can affect visible and infrared imaging. The impact of atmospheric transmission effects is more pronounced in foggy conditions, and thermal interference further weakens infrared imaging, blurring or even losing target details. This reduces the consistency and reliability of multimodal data and leads to inaccurate recognition results. Furthermore, modern ships, especially those with the same purpose, adhere to similar design specifications and technical standards, resulting in less distinct characteristics between different ship classes. However, modifications and technological upgrades can lead to significant differences in characteristics within the same class, making it difficult to extract both common features within the same class and unique features between different classes, ultimately leading to inaccurate recognition results.

[0004] Therefore, how to improve the accuracy of fine-grained ship identification and solve the problem in the current existing technology that the atmospheric transmission effect interferes with the target details and even makes it difficult to extract the common features within the same class and the unique features between different classes has become a technical problem that needs to be solved urgently in this field. Summary of the Invention

[0005] The purpose of this application is to provide a multimodal ship fine-grained identification method and system, which can improve the accuracy of ship fine-grained identification.

[0006] To achieve the above objectives, this application provides the following solutions.

[0007] In a first aspect, the present application provides a multimodal fine-grained ship identification method, which includes the following steps.

[0008] Acquire multimodal data of the target area; the multimodal data includes ISAR images, infrared images and visible light images.

[0009] The multimodal data is subjected to data augmentation preprocessing, and a training set and a test set are constructed based on the preprocessed multimodal data; each training sample in the training set is provided with a corresponding true type class label, and the true type class label is used to represent the true type class of the ship in the target area.

[0010] A multi-source feature extraction method based on physical degradation modeling is adopted to extract features from each training sample in the training set and each test sample in the test set, thereby obtaining a multi-source feature map of each training sample and a multi-source feature map of each test sample.

[0011] A discriminative feature learning method based on spatiotemporal consistency alignment is adopted to update the node features of the multi-source feature graphs of each training sample and the multi-source feature graphs of each test sample, respectively, to obtain the updated node features of each training sample and the updated node features of each test sample.

[0012] The updated node features of each training sample are input into the type-level prediction model, and the multimodal mixed contrast loss function is adopted as the loss function. The loss is calculated based on the type-level prediction probability output by the type-level prediction model and the corresponding true type-level label. The model parameters are optimized based on the loss, and the trained type-level prediction model is obtained after the iterative training is completed.

[0013] The updated node features of each test sample are input into the trained type-level prediction model to obtain the final type-level prediction probability.

[0014] Optionally, data augmentation preprocessing is performed on the multimodal data, and a training set and a test set are constructed based on the preprocessed multimodal data, specifically including the following steps.

[0015] Performing synchronous geometric transformation on the multimodal data to obtain transformed multimodal data.

[0016] A modality-specific disturbance based on physics guidance is added to the transformed multimodal data to obtain multimodal data after the disturbance is added.

[0017] The multimodal data after the disturbance is added is subjected to random occlusion enhancement processing to obtain preprocessed multimodal data.

[0018] The preprocessed multimodal data is divided into samples to construct a training set and a test set.

[0019] Optionally, adding a modality-specific disturbance based on physics guidance to the transformed multimodal data to obtain the multimodal data after the disturbance is added specifically includes the following steps.

[0020] Hot spot noise is added to the infrared image in the transformed multimodal data, and the image contrast is stretched to obtain a disturbed infrared image.

[0021] HSV spatial parameter adjustment is performed on the visible light image in the transformed multimodal data to obtain a visible light image after adding disturbance.

[0022] Multiplicative noise is added to the ISAR image in the transformed multimodal data to obtain a disturbed ISAR image; the disturbed ISAR image, the disturbed visible light image, and the disturbed infrared image are used as the disturbed multimodal data.

[0023] Optionally, a multi-source feature extraction method based on physical degradation modeling is adopted to perform feature extraction on each training sample in the training set and each test sample in the test set to obtain a multi-source feature map of each training sample and a multi-source feature map of each test sample, which specifically includes the following steps.

[0024] The atmospheric transport equation is established by embedding the physical degradation model.

[0025] Each training sample and each test sample is substituted into the atmospheric transmission equation respectively, and the restoration feature of each training sample and the restoration feature of each test sample are calculated.

[0026] A degradation factor attention module based on the degradation parameter estimation subnetwork is used to calculate the attention mask of each training sample and the attention mask of each test sample respectively.

[0027] The restored features and attention masks of each training sample are multiplied point by point to obtain the multi-source feature map of each training sample; at the same time, the restored features and attention masks of each test sample are multiplied point by point to obtain the multi-source feature map of each test sample.

[0028] Optionally, the atmospheric transmission equation is expressed as follows.

[0029] .

[0030] in, Represents the recovery feature, represents the input feature map, is the transmittance, is the ambient light intensity.

[0031] The attention mask is calculated using the following formula.

[0032] .

[0033] in, represents the attention mask, is the activation function, is the weight matrix, Represents feature splicing.

[0034] Optionally, a method of discriminative feature learning based on spatiotemporal consistency alignment is adopted to update the node features of the multi-source feature graphs of each training sample and the multi-source feature graphs of each test sample, respectively, to obtain the updated node features of each training sample and the updated node features of each test sample, which specifically includes the following steps.

[0035] A spatiotemporal graph is constructed based on the multi-source feature graphs of each training sample and the multi-source feature graphs of each test sample; wherein the spatiotemporal graph includes a plurality of nodes and edges connecting every two of the nodes, and the nodes represent slices of the multi-source feature graphs of different modalities along each channel.

[0036] Based on the spatiotemporal graph, neighborhood information is aggregated using dynamic graph attention weights to calculate the inter-node similarity between every two nodes.

[0037] The node features are updated according to the inter-node similarity between every two nodes, and the updated node features of each training sample and the updated node features of each test sample are obtained respectively.

[0038] Optionally, the inter-node similarity between every two nodes is calculated using the following formula.

[0039] .

[0040] in, Representation node and nodes j The similarity between nodes, and Represents nodes respectively and nodes j Features, For nodes The set of neighboring nodes, Representation node The set of neighboring nodes midpoint k Features, and is the learnable weight matrix, is the attention weight.

[0041] Optionally, the node feature update is expressed as follows:

[0042] .

[0043] in, represents the updated node features, represents the ReLU function, For nodes The set of neighboring nodes, Representation node and nodes j The similarity between nodes, Update the weight matrix for the feature, Representation node j characteristics.

[0044] Optionally, the expression of the multimodal hybrid contrast loss function is as follows.

[0045] .

[0046] .

[0047] .

[0048] in, represents the total loss function, represents the cross entropy loss, represents the dynamic contrast loss, represents the dynamic weight, is the current sample feature, is the historical negative sample prototype feature, is the similarity calculation function, is the batch size, is the number of negative samples, is the prototype feature of the positive sample, is the distance between classes, is the temperature coefficient, is the cross entropy loss weight, is the dynamic contrast loss weight.

[0049] In a second aspect, the present application provides a multimodal ship fine-grained identification system, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multimodal ship fine-grained identification method.

[0050] According to the specific embodiments provided in this application, this application has the following technical effects.

[0051] The present application provides a multimodal ship fine-grained identification method and system, which facilitates the accurate extraction of ship features by the class prediction model during the training phase by performing data amplification preprocessing on the multimodal data, effectively improving the robustness of the class prediction model in actual scenarios, and ensuring the accuracy of fine-grained ship identification. By adopting a multi-source feature extraction method based on physical degradation modeling for feature extraction, explicitly modeling the degradation process, and improving the analysis capability of visible light and infrared image feature degradation, it is conducive to extracting more accurate ship features, thereby improving the accuracy of fine-grained ship identification of the class prediction model. By adopting a discriminable feature learning method based on spatiotemporal consistency alignment for node feature update, the class prediction model is prompted to fully understand the multimodal data, accurately grasp the intrinsic connection between ship features, and provide richer and more discriminative feature information for fine-grained ship identification, effectively improving the class prediction model's processing capability and recognition accuracy for complex multimodal data. By using the multimodal mixed contrast loss function as the loss function for model training, the type-level prediction model can pay more attention to samples that are difficult to classify, learn the subtle differences between similar types, reduce misjudgments caused by blurred type-level boundaries, and effectively improve the type-level prediction model's classification ability and recognition accuracy for complex samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 This is an application environment diagram of a multimodal fine-grained ship identification method provided in one embodiment of the present application.

[0054] Figure 2 A flowchart of a multimodal fine-grained ship identification method provided in one embodiment of the present application.

[0055] Figure 3 A schematic diagram of a multimodal fine-grained ship identification method provided in one embodiment of the present application.

[0056] Figure 4 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0058] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0059] The embodiment of the present application provides a multimodal ship fine-grained identification method, which can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send multimodal data to the server 104. After the server 104 receives the multimodal data, the server 104 performs data augmentation preprocessing on the multimodal data to construct a training set and a test set; adopts a multi-source feature extraction method based on physical degradation modeling for feature extraction, adopts a discriminable feature learning method based on spatiotemporal consistency alignment for node feature update, adopts a multimodal mixed contrast loss function as a loss function, trains the type-level prediction model, and obtains a trained type-level prediction model after iterative training; inputs the updated node features of each test sample into the trained type-level prediction model to obtain the final type-level prediction probability. The server 104 can feedback the final type-level prediction probability to the terminal 102. In addition, in some embodiments, the multimodal ship fine-grained identification method can also be implemented independently by the server 104 or the terminal 102. For example, the terminal 102 can directly perform fine-grained ship identification processing on the multimodal data, or the server 104 can obtain the multimodal data from the data storage system and perform fine-grained ship identification processing on the multimodal data.

[0060] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or a cloud server.

[0061] In an exemplary embodiment, Figure 2As shown, a multimodal ship fine-grained identification method is provided. The method is executed by a computer device, specifically a computer device such as a terminal or a server, or a terminal and a server. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in FIG. 1 is used as an example to illustrate the method, which includes the following steps S1 to S6.

[0062] Step S1: Acquire multimodal data of the target area, wherein the multimodal data includes ISAR images, infrared images, and visible light images.

[0063] Step S2: performing data augmentation preprocessing on the multimodal data, and constructing a training set and a test set based on the preprocessed multimodal data; each training sample in the training set is provided with a corresponding true type class label, and the true type class label is used to represent the true type class of the ship in the target area.

[0064] In this embodiment, step S2 performs data augmentation preprocessing on the multimodal data, and constructs a training set and a test set based on the preprocessed multimodal data, which specifically includes the following steps.

[0065] Step S21: Perform synchronous geometric transformation on the multimodal data to obtain transformed multimodal data.

[0066] Step S22: adding a modality-specific disturbance based on physics guidance to the transformed multimodal data to obtain multimodal data after the disturbance is added.

[0067] In this embodiment, step S22 adds a modality-specific disturbance based on physics guidance to the transformed multimodal data to obtain the multimodal data after the disturbance is added, which specifically includes the following steps.

[0068] Step S221 : adding hot spot noise to the infrared image in the transformed multimodal data and stretching the image contrast to obtain a disturbed infrared image.

[0069] Step S222 : performing HSV (Hue-Saturation-Value) space parameter adjustment on the visible light image in the transformed multimodal data to obtain a visible light image after adding disturbance.

[0070] Step S223: adding multiplicative noise to the ISAR image in the transformed multimodal data to obtain a disturbed ISAR image; the disturbed ISAR image, the disturbed visible light image, and the disturbed infrared image serve as the disturbed multimodal data.

[0071] Step S23: performing random occlusion enhancement processing on the multimodal data after the disturbance is added to obtain preprocessed multimodal data.

[0072] Step S24: dividing the preprocessed multimodal data into samples to construct a training set and a test set.

[0073] Step S3: adopting a multi-source feature extraction method based on physical degradation modeling to extract features from each training sample in the training set and each test sample in the test set, and obtain a multi-source feature map of each training sample and a multi-source feature map of each test sample.

[0074] In this embodiment, step S3 adopts a multi-source feature extraction method based on physical degradation modeling to perform feature extraction on each training sample in the training set and each test sample in the test set, respectively, to obtain a multi-source feature map of each training sample and a multi-source feature map of each test sample, specifically including the following steps.

[0075] Step S31: Establishing the atmospheric transmission equation by embedding the physical degradation model.

[0076] Step S32: Substitute each training sample and each test sample into the atmospheric transmission equation, and calculate and obtain the restoration features of each training sample and each test sample.

[0077] Step S33: Using the degradation factor attention module (DFAM) based on the degradation parameter estimation subnetwork, the attention mask of each training sample and the attention mask of each test sample are calculated respectively.

[0078] Step S34: multiply the restored features and attention masks of each training sample point by point to obtain a multi-source feature map of each training sample; and at the same time, multiply the restored features and attention masks of each test sample point by point to obtain a multi-source feature map of each test sample.

[0079] Step S4: adopt a discriminative feature learning method based on spatiotemporal consistency alignment to update the node features of the multi-source feature graph of each training sample and the multi-source feature graph of each test sample, respectively, to obtain the updated node features of each training sample and the updated node features of each test sample.

[0080] In this embodiment, step S4 adopts a discriminative feature learning method based on spatiotemporal consistency alignment to update the node features of the multi-source feature graphs of each training sample and the multi-source feature graphs of each test sample, respectively, to obtain the updated node features of each training sample and the updated node features of each test sample, specifically including the following steps.

[0081] Step S41: Construct a spatio-temporal graph based on the multi-source feature maps of the respective training samples and the multi-source feature maps of the respective test samples; wherein, the spatio-temporal graph includes a number of nodes and edges connecting every two of the nodes, and the nodes represent slices of the multi-source feature maps of different modalities along each channel.

[0082] Step S42: Based on the spatio-temporal graph, use dynamic graph attention weights to aggregate neighborhood information and calculate the inter-node similarity between every two nodes.

[0083] Step S43: Update the node features according to the inter-node similarity between every two nodes, and respectively obtain the updated node features of the respective training samples and the updated node features of the respective test samples.

[0084] Step S5: Input the updated node features of the respective training samples into the type-level prediction model, use the multi-modal hybrid contrast loss function as the loss function, calculate the loss according to the type-level prediction probability output by the type-level prediction model and the corresponding true type-level label, and optimize the model parameters based on the loss. After iterative training, obtain the trained type-level prediction model.

[0085] Step S6: Input the updated node features of the respective test samples into the trained type-level prediction model to obtain the final type-level prediction probability.

[0086] To make the technical solution of this embodiment clearer, the following uses an example to illustrate the specific implementation process of the technical solution of this embodiment in detail.

[0087] In response to the key challenges in multi-modal image fine-grained classification, such as the interference of atmospheric transmission effects leading to modal feature degradation, small differences between type levels and large differences within type levels, this embodiment proposes a technical route of "data augmentation - physical degradation modeling - spatio-temporal consistent alignment - multi-modal hybrid contrast loss joint optimization", and the specific steps are as follows.

[0088] Step 1: Data collection and data augmentation preprocessing.

[0089] In this embodiment, an unmanned aerial vehicle is used to collect images of three modalities, namely ISAR images, infrared images, and visible light images, of the target area, and the true type-level labels are marked to represent the true type levels of the ships, so as to construct <ISAR, infrared, visible light, type-level label> data pairs, where there are N type-level labels in total.

[0090] Changes in the drone's viewing angle lead to diverse target poses, and in real-world scenarios, atmospheric interference and sensor noise significantly reduce image quality. Traditional data enhancement methods only change geometric properties without considering physical degradation mechanisms, which limits the generalization capabilities of model-level prediction models. To improve the diversity of multimodal data and enhance the generalization capabilities of model-level prediction models for complex degradation scenarios, this embodiment designs a synchronized geometric transformation preprocessing method that ensures multimodal spatial alignment through unified random parameters to avoid registration errors. This method combines a physics-guided modal-specific perturbation preprocessing method constructed based on modal characteristics with a degradation enhancement method designed based on the sensor imaging mechanism to more closely resemble real physical scenarios.

[0091] (1) Synchronous geometric transformation.

[0092] To simulate the changes in the drone's viewpoint and enhance the adaptability of the enhanced prediction model to the target pose, we propose to simultaneously apply the same geometric deformation to the ISAR, infrared, and visible light images using unified affine transformation parameters. This simulates the simultaneous observation characteristics of multiple sensors when the drone's viewpoint changes, and avoids feature alignment errors caused by spatial misalignment between modalities. Specifically, we apply uniform random parameter rotation, translation, and scaling to the three modal images, maintaining spatial alignment between the modalities and completing the synchronous geometric transformation. The image after the synchronous geometric transformation is expressed as follows.

[0093] (1).

[0094] in, is the image after synchronous geometric transformation, represents synchronous geometric transformation, is the image to be synchronized with the geometric transformation, is the rotation angle, is the translation ratio, s is the scaling factor, m Indicates the range of the image with synchronous geometric transformation. ISAR indicates ISAR image, IR indicates infrared image, and Vis indicates visible light image.

[0095] This embodiment uses synchronous geometric transformation preprocessing to simulate images taken by drones at different perspectives, enhance the adaptability of the enhanced prediction model to different postures of ships, and ensure the spatial consistency of multimodal images, providing a basis for subsequent feature fusion.

[0096] (2) Adding physics-guided mode-specific perturbations.

[0097] Combining the characteristics of each modality, the following perturbations are added to the infrared, visible light, and ISAR images respectively.

[0098] 1) Infrared image: In order to more realistically simulate the noise or abnormalities that may occur in the actual infrared imaging process, hot spot noise is injected And stretch the image contrast. Among them, is the image after injecting hot spot noise, is the original infrared image, is the noise intensity coefficient, is the mean value of the hot spot noise, and 0.1 is the noise variance.

[0099] 2) Visible light image: To adapt the model to different lighting conditions and scenes, the HSV space parameters are adjusted to simulate lighting changes. First, the visible light image is converted from the BGR (Blue-Green-Red) color space to the HSV color space, and then the hue (H) and saturation (S) are randomly adjusted. The saturation varies randomly within the range of Finally, the adjusted HSV channels are merged and converted back to BGR color space to obtain the amplified image.

[0100] 3) ISAR Images: Coherent speckle noise is a common type of noise in ISAR images, and its essence is multiplicative noise. To improve the adaptability of the model-level prediction model to noisy environments, we propose adding multiplicative noise to complex-domain ISAR data based on the principles of radar coherent imaging. The image after adding multiplicative noise is expressed as follows.

[0101] (2).

[0102] in, is the image after adding multiplicative noise, is the original image, is the noise intensity, represents a complex Gaussian distribution.

[0103] (3) Random occlusion enhancement.

[0104] In real-world maritime scenarios, ships may be partially obscured, resulting in missing local features. To improve the robustness of the class prediction model to local feature loss, this embodiment uses random occlusion enhancement preprocessing. This adds irregular occlusions at the same location, forcing the class prediction model to learn multimodal complementary features. The image after occlusion enhancement is expressed as follows:

[0105] (3).

[0106] in, is the image after occlusion enhancement, is the original image, is the occlusion mask, the value is 0 or 1; is the mean of the image; In this way, the partial occlusion of the ship in the actual scene is simulated, which enables the class prediction model to learn the complementary features of multimodal images during training and improve its robustness to local feature loss.

[0107] Through the above steps, preprocessed multimodal data can be obtained.

[0108] This embodiment uses data acquisition and data augmentation to design modality-specific perturbations based on physics guidance, applying specific perturbations based on the imaging characteristics of each modality. For example, hotspot noise is injected into infrared images and the contrast is stretched to simulate the noise and anomalies of real infrared imaging; the HSV parameters of visible light images are adjusted to simulate illumination changes, allowing the class prediction model to adapt to different lighting environments; and multiplicative noise is added to ISAR images to align with the principles of radar coherent imaging. These operations expose the class prediction model to various real-world interferences during the training phase, enhancing its adaptability to different imaging conditions. This enables the class prediction model to accurately extract ship features even under complex weather and lighting conditions, effectively improving its robustness in real-world scenarios and ensuring the accuracy of recognition results.

[0109] Step 2: Multi-source feature extraction based on physical degradation modeling.

[0110] First, use 3 layers of ResNetBlock (residual block) to extract image features and obtain the input feature map Traditional multimodal feature fusion methods such as early fusion and feature splicing usually directly aggregate features from different modalities, ignoring the contamination of intra-modal features by degradation factors such as atmospheric turbulence and thermal radiation noise, resulting in insufficient robustness of the fused type-level discriminant features. To this end, this embodiment proposes a robust feature extraction method based on multimodal anti-degradation decoupling, which explicitly separates degradation factors by embedding an atmospheric transmission model to more specifically learn target features from multi-source images.

[0111] (1) Physical degradation modeling.

[0112] To improve the ability to analyze feature degradation processes in visible and infrared imagery, a physical degradation model embedding method was designed to explicitly model the physical mechanisms of atmospheric transport and dynamically suppress cloud obscuration and thermal radiation noise. By embedding the atmospheric degradation model (atmospheric transport equation and sensor degradation model) into the network, the degradation process is explicitly modeled. Specifically, the atmospheric transport equation is expressed as follows.

[0113] (4).

[0114] in, Represents the recovery feature, represents the input feature map, is the transmittance, For ambient light intensity, transmittance and ambient light intensity are predicted by the physical parameter estimation subnet. The physical parameter estimation subnet contains two branches (CNNs), each of which is composed of three layers of Conv (1x1), such as Figure 3 shown.

[0115] (2) Degradation factor attention module.

[0116] Existing methods for fusing multi-source heterogeneous image features directly employ methods such as splicing and overlaying to achieve rigid alignment and optimization in the feature space. This approach struggles to adapt to the spatiotemporal heterogeneity of multimodal features and fails to effectively model the spatiotemporal correlations between modalities. To enable the model-level prediction model to account for the impact of the actual atmospheric environment on features, this embodiment designs a degradation factor attention module based on the degradation parameter estimation subnetwork. This degradation factor attention module is an attention mechanism for image restoration tasks that enhances the network's ability to learn image degradation features by explicitly modeling degradation factors.

[0117] This embodiment uses the degradation factor attention module based on the degradation parameter estimation subnet to learn the transmittance through the convolution layer and the fully connected layer. and ambient light Generate attention mask modulation feature map. The attention mask generation method is expressed as follows.

[0118] (5).

[0119] in, represents the attention mask, is the activation function, is the weight matrix, Indicates feature splicing. This method uses transmittance and ambient light intensity By integrating the atmospheric degradation model into the deep learning framework and using attention mask Suppress the characteristic response of the degraded area and retain the target information.

[0120] This embodiment uses the attention mask and recovery features Multiply point by point to obtain the final multi-source feature map .

[0121] The above steps are performed on ISAR images, visible light images and infrared images respectively to obtain the multi-source feature maps of the three modes of output , including multi-source feature maps of infrared images , multi-source feature map of visible light image , multi-source feature map of ISAR image .

[0122] This embodiment utilizes a multi-source feature extraction approach based on physical degradation modeling. Based on physical degradation modeling, it innovatively embeds an atmospheric degradation model into the network, explicitly modeling the degradation process and improving the ability to analyze visible and infrared image feature degradation. By predicting transmittance and ambient light intensity, cloud obstruction and thermal radiation noise are dynamically suppressed, enabling the class prediction model to extract purer and more accurate ship target features. In adverse weather conditions such as fog and clouds, this method effectively removes interference and accurately identifies ships. Compared to traditional methods, it can significantly improve recognition accuracy in complex atmospheric environments, providing a more reliable feature foundation for fine-grained ship classification.

[0123] This embodiment also proposes a degradation factor attention module. To address the problem that traditional fusion methods are difficult to adapt to the spatiotemporal heterogeneity of multimodal features, the degradation factor attention module innovatively generates an attention mask by learning transmittance and ambient light through convolutional layers and fully connected layers. This method integrates the atmospheric degradation model into the deep learning framework, which can suppress the characteristic response of the degraded area, retain target information, and effectively model the spatiotemporal correlation between modalities. When processing different modal features, the type-class prediction model can focus more on the key features of the ship, improving the feature fusion effect and the robustness of the type-class discrimination features, so that the type-class prediction model can more accurately identify the ship type when faced with complex multimodal data.

[0124] Step 3: Discriminative feature learning based on spatiotemporal consistency alignment.

[0125] Multimodal images have spatiotemporal heterogeneity problems, such as sparse ISAR contours and blurred infrared thermal radiation. Traditional rigid spatial alignment methods are difficult to adapt to this heterogeneity. To solve this problem, the spatiotemporal correlation between modeling modalities is proposed instead of relying on rigid spatial alignment. To this end, a spatiotemporal graph is constructed. ,node Representing different modal feature maps 、 、 Slices along each channel, The edges connecting nodes in the spatiotemporal graph encode spatiotemporal correlations (such as motion consistency and thermal-texture correlation). Neighborhood information is aggregated using dynamic graph attention weights. Specifically, we first calculate the similarity between nodes, which is expressed as the following formula.

[0126] (6).

[0127] in, Representation node and nodes j The similarity between nodes, and Represents nodes respectively and nodes j Features, For nodes The set of neighboring nodes, Representation node The set of neighboring nodes midpoint k Features, and is the learnable weight matrix, is the attention weight, which is optimized by back propagation during training. and Calculate the similarity between nodes and adaptively capture cross-modal associations.

[0128] Then use the following formula to update the node features.

[0129] (7).

[0130] in, represents the updated node features, represents the ReLU function, For nodes The set of neighboring nodes, Representation node and nodes j The similarity between nodes, Update the weight matrix for the feature, Representation node j This method enhances the target spatiotemporal consistency expression by weighted aggregation of neighborhood node features. Neighborhood node set Node features in , through the similarity between nodes , feature update weight matrix After operation, the updated node features are obtained , that is, for the node Update the features.

[0131] Finally, a fully connected layer containing N nodes is added, corresponding to the predicted output of N type-level labels.

[0132] This embodiment employs a discriminative feature learning method based on spatiotemporal consistency alignment. By calculating node similarity and aggregating dynamic graph attention weights, it breaks through traditional rigid spatial alignment methods, constructs a spatiotemporal graph, and calculates inter-node similarity using learnable parameters. This approach adaptively captures cross-modal associations, fully exploiting spatiotemporal correlations such as motion consistency, thermal, and texture correlations between feature map slices from different modalities and channels. This enables the class prediction model to fully understand multimodal data and accurately grasp the inherent connections between ship features. This provides richer and more discriminative feature information for fine-grained ship recognition, effectively improving the class prediction model's processing power and recognition accuracy for complex multimodal data. Furthermore, this embodiment utilizes weighted aggregation of neighboring node features to enhance the target's spatiotemporal consistency representation, enabling the class prediction model to continuously optimize node features during the learning process, making it more stable and discriminative. In the face of complex situations such as ship posture changes and background interference, the class prediction model can continuously update features and accurately determine the ship's class, effectively improving the class prediction model's adaptability and recognition accuracy.

[0133] Step 4: Multimodal hybrid contrast loss function design.

[0134] This embodiment adds the following two loss functions on the basis of cross entropy loss to form a multimodal hybrid contrast loss function.

[0135] (1) Dynamic difficult sample mining contrast loss.

[0136] Screening difficult samples. First, input all training samples into the model-level prediction model to obtain the model-level prediction probability, and then select the 10% samples with the largest difference between the prediction probability and the label as the initial historical negative sample prototype. Assuming that there are negative samples.

[0137] To address the problem that traditional contrastive loss methods don't adequately focus on difficult samples (those with blurred class boundaries and large intra-class variations), we propose dynamically adjusting sample weights to focus on key areas. Specifically, we construct a memory queue to store the features of historical difficult samples and dynamically assign weights through similarity calculations, thereby strengthening the class-level prediction model's ability to distinguish class boundaries. The dynamic weight calculation is expressed as the following formula.

[0138] (8).

[0139] in, represents the dynamic weight, is the current sample feature, is the historical negative sample prototype feature, represents all prototype features, is the number of negative samples, is the similarity calculation function, here we use Euclidean distance, is the temperature coefficient. In this embodiment Set to 0.1. Using the temperature coefficient Control the smoothness of weight distribution to achieve adaptive focusing and avoid overfitting. Dynamic weights corresponding to high similarity negative samples The larger the value, the more difficult the sample is, thus achieving the goal of screening difficult samples. The dynamic contrast loss function is calculated as follows.

[0140] (9).

[0141] in, represents the dynamic contrast loss, is the batch size, is the current sample feature, is the prototype feature of the positive sample, representing the current sample feature The characteristics of another sample of the same type, is the historical negative sample prototype feature, is the inter-class distance, which is used to control the inter-class distance and prompt the type-level prediction model to strengthen the classification boundary to distinguish similar types. Set to 0.2.

[0142] (2) Multi-task joint optimization framework.

[0143] A single loss function is difficult to balance the decoupling of degradation, feature alignment and classification tasks, and requires collaborative optimization to improve the generalization of the model-level prediction model. To this end, a total loss function is constructed to integrate the cross entropy loss. and dynamic contrast loss , through the weight coefficient and Balancing multiple objectives is expressed as the following formula.

[0144] (10).

[0145] in, represents the total loss function, represents the cross entropy loss, represents the dynamic contrast loss, is the cross entropy loss weight, is the dynamic contrast loss weight. In this embodiment, =0.8, =0.2.

[0146] The process of model training and testing of the model-level prediction model in this embodiment specifically includes the following steps.

[0147] (1) Data preparation.

[0148] From the training set, each time e <ISAR, infrared, visible light, type-level label> data pairs are selected. The data here includes multi-modal data such as ISAR images, infrared images, and visible light images, and all have completed data augmentation preprocessing (including operations such as synchronous geometric transformation, physics-guided modality-specific perturbation, and random occlusion enhancement). The type-level label corresponds to fine-grained classification information such as the type of the ship.

[0149] (2)Model forward propagation.

[0150] Input a batch of e selected data pairs into the type-level prediction model. The type-level prediction model first processes the input multi-modal image data, extracts multi-source feature maps through operations such as embedding the physical degradation model (i.e., the atmospheric degradation model) and the degradation factor attention module; then, through a discriminative feature learning method based on spatio-temporal consistency alignment, constructs a spatio-temporal graph and calculates the similarity between nodes, and updates to obtain N node features.

[0151] (3)Calculate the loss.

[0152] Combine the N node features output by the type-level prediction model with the corresponding true type-level labels, and calculate the loss value according to the multi-modal hybrid contrast loss function.

[0153] (4)Backpropagation and model optimization

[0154] Use the calculated loss value for backpropagation, and calculate the gradients of each parameter in the type-level prediction model according to the backpropagation algorithm. Based on the calculated gradients, use the SGD optimization algorithm to update the model parameters of the type-level prediction model.

[0155] (5)Repeat training.

[0156] Repeat steps (1)-(4) 200 times. Finally, obtain the trained type-level prediction model.

[0157] (6)Model testing.

[0158] For the test samples in the test set, input them into the trained type-level prediction model. The type-level prediction model performs operations such as feature extraction and node feature update on the test samples according to the processing flow in the training stage, and outputs N node features. The results corresponding to these node features are the ship fine-grained recognition results, that is, the final type-level prediction probabilities predicted by the type-level prediction model.

[0159] This embodiment designs a multimodal hybrid contrast loss function, uses dynamic difficult sample mining contrast loss to select samples with the largest difference between predicted probabilities and labels as historical negative sample prototypes, and constructs a memory queue to dynamically assign weights. This approach effectively solves the problem of traditional contrast loss's insufficient attention to difficult samples and enhances the type-level prediction model's ability to distinguish type-level boundaries. In practical applications, the type-level prediction model can pay more attention to samples that are difficult to classify, learn subtle differences between similar types, reduce misjudgments caused by blurred type-level boundaries, and effectively improve the type-level prediction model's classification ability and recognition accuracy for complex samples. In addition, a multi-task joint optimization framework is designed. By constructing a total loss function, integrating cross-entropy loss and dynamic contrast loss, and balancing multiple objectives through weight coefficients, it avoids the limitations of a single loss function in balancing degradation decoupling, feature alignment, and classification tasks, enabling the type-level prediction model to better coordinate between different tasks to improve the generalization, stability, and reliability of the type-level prediction model.

[0160] In an exemplary embodiment, a multimodal ship fine-grained recognition system is provided. The system may be a computer device, which may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store multimodal data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multimodal ship fine-grained recognition method is implemented.

[0161] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0162] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0163] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0164] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A multimodal ship fine-grained identification method, characterized in that: The multimodal ship fine-grained identification method includes: Acquiring multimodal data of the target area; the multimodal data includes ISAR images, infrared images, and visible light images; Performing data augmentation preprocessing on the multimodal data, and constructing a training set and a test set based on the preprocessed multimodal data; each training sample in the training set is provided with a corresponding true class label, the true class label being used to indicate the true class of the ship in the target area; Performing data augmentation preprocessing on the multimodal data and constructing a training set and a test set based on the preprocessed multimodal data, specifically including: Performing synchronous geometric transformation on the multimodal data to obtain transformed multimodal data; Adding a physics-guided modality-specific disturbance to the transformed multimodal data to obtain perturbed multimodal data; Performing random occlusion enhancement processing on the multimodal data after adding disturbance to obtain preprocessed multimodal data; Performing sample division on the preprocessed multimodal data to construct a training set and a test set; A multi-source feature extraction method based on physical degradation modeling is used to extract features from each training sample in the training set and each test sample in the test set, respectively, to obtain a multi-source feature map of each training sample and a multi-source feature map of each test sample, specifically including: The atmospheric transport equation is established by embedding the physical degradation model; Substituting each training sample and each test sample into the atmospheric transmission equation, respectively, to calculate the restoration features of each training sample and the restoration features of each test sample; A degradation factor attention module based on the degradation parameter estimation subnetwork is used to calculate the attention mask of each training sample and the attention mask of each test sample respectively; Multiply the restored features and attention masks of each training sample point by point to obtain the multi-source feature map of each training sample; at the same time, multiply the restored features and attention masks of each test sample point by point to obtain the multi-source feature map of each test sample; The expression of the atmospheric transmission equation is: ； in, Represents the recovery feature, represents the input feature map, is the transmittance, is the ambient light intensity; The attention mask is calculated using the following formula: ； in, represents the attention mask, is the activation function, is the weight matrix, Represents feature splicing; Adopting a discriminative feature learning method based on spatiotemporal consistency alignment, respectively updating node features of the multi-source feature graphs of each training sample and the multi-source feature graphs of each test sample, to obtain updated node features of each training sample and updated node features of each test sample; The updated node features of each training sample are input into the type-level prediction model, a multimodal hybrid contrast loss function is used as the loss function, and the loss is calculated based on the type-level prediction probability output by the type-level prediction model and the corresponding true type-level label. The model parameters are optimized based on the loss, and a trained type-level prediction model is obtained after iterative training is completed; The updated node features of each test sample are input into the trained type-level prediction model to obtain the final type-level prediction probability.

2. The multimodal ship fine-grained identification method according to claim 1 is characterized in that: Adding a physics-guided modal-specific perturbation to the transformed multimodal data to obtain the perturbed multimodal data specifically includes: adding hot spot noise to the infrared image in the transformed multimodal data and stretching the image contrast to obtain a disturbed infrared image; Performing HSV spatial parameter adjustment on the visible light image in the transformed multimodal data to obtain a visible light image after adding disturbance; Multiplicative noise is added to the ISAR image in the transformed multimodal data to obtain a disturbed ISAR image; the disturbed ISAR image, the disturbed visible light image, and the disturbed infrared image are used as the disturbed multimodal data.

3. The multimodal ship fine-grained identification method according to claim 1 is characterized in that: A method of discriminative feature learning based on spatiotemporal consistency alignment is adopted to update node features of the multi-source feature graphs of each training sample and the multi-source feature graphs of each test sample, respectively, to obtain updated node features of each training sample and updated node features of each test sample, specifically including: Constructing a spatiotemporal graph based on the multi-source feature graphs of each training sample and the multi-source feature graphs of each test sample; wherein the spatiotemporal graph includes a plurality of nodes and edges connecting every two of the nodes, the nodes representing slices of the multi-source feature graphs of different modalities along each channel; Based on the spatiotemporal graph, neighborhood information is aggregated using dynamic graph attention weights to calculate the node similarity between every two nodes; The node features are updated according to the inter-node similarity between every two nodes, and the updated node features of each training sample and the updated node features of each test sample are obtained respectively.

4. The multimodal ship fine-grained identification method according to claim 3 is characterized in that: The following formula is used to calculate the node similarity between every two nodes: ； in, Representation node and nodes j The similarity between nodes, and Represents nodes respectively and nodes j Features, For nodes The set of neighboring nodes, Representation node The set of neighboring nodes midpoint k Features, and is the learnable weight matrix, is the attention weight.

5. The multimodal ship fine-grained identification method according to claim 3 is characterized in that: The following formula is used to express node feature update: ； in, represents the updated node features, ReLU(·) represents the ReLU function, For nodes The set of neighboring nodes, Representation node and nodes j The similarity between nodes, Update the weight matrix for the feature, Representation node j characteristics.

6. The multimodal ship fine-grained identification method according to claim 1 is characterized in that: The expression of the multimodal hybrid contrast loss function is: ； ； ； in, represents the total loss function, represents the cross entropy loss, represents the dynamic contrast loss, represents the dynamic weight, is the current sample feature, is the historical negative sample prototype feature, s (·) is the similarity calculation function, is the batch size, is the number of negative samples, is the prototype feature of the positive sample, represents all prototype features, is the distance between classes, is the temperature coefficient, is the cross entropy loss weight, is the dynamic contrast loss weight.

7. A multimodal ship fine-grained identification system comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multimodal ship fine-grained identification method according to any one of claims 1 to 6.

Images