A multi-modal ship hierarchical fine-grained collaborative recognition method and system

Abstract

The application provides a multi-modal ship hierarchical fine-grained collaborative recognition method and system, belonging to the technical field of computers. The method comprises: acquiring visible light, near-infrared and thermal infrared images of a ship under the same scene and performing pretreatment; inputting the standardized multi-modal input tensor into an HSR-Net model, first extracting independent modal features, performing semantic alignment and dynamic gate fusion through a multi-modal feature alignment module, and obtaining fusion features; then extracting multi-scale feature maps using a backbone network; then performing CBAM feature calibration, CrossAttention semantic guidance and ShipHPFO hierarchical progressive optimization in turn through an inter-layer optimization unit, and outputting semantic-enhanced hierarchical features; finally, inputting the features into three parallel branches of class level, type level and individual level respectively, and outputting hierarchical prediction probability distribution. The application realizes high-precision, fine-grained collaborative recognition from ship categories, types to individual identities.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and more specifically relates to a multimodal ship hierarchical fine-grained collaborative identification method and system. Background Technology

[0002] In recent years, with the increasing frequency of marine economic activities and the growing demand for perception capabilities in applications such as maritime supervision and maritime traffic management, accurate and reliable identification of ship targets has become a key technological support. Although deep learning methods, represented by convolutional neural networks (CNNs) and Transformers, have made significant progress in this field, existing technologies still have obvious limitations. Most methods rely on a single imaging modality such as visible light, and their recognition performance is easily affected by drastic changes in illumination, adverse weather conditions (such as fog and rain), and complex sea surface backgrounds, resulting in unstable feature extraction and limited model generalization ability, making it difficult to meet the practical application requirements of all-weather, high robustness.

[0003] Furthermore, ship targets possess an inherent hierarchical semantic structure of "category-model-individual," which poses a significant challenge to fine-grained recognition tasks: visual differences between different models or individuals are often very subtle, while intra-class differences between similar ships caused by variations in viewpoint, occlusion, or paint scheme can be very significant. Most current mainstream methods employ a flat classification paradigm, failing to explicitly model and utilize this hierarchical prior knowledge. Therefore, models struggle to learn hierarchical feature representations with strong discriminative power, particularly in fine-grained sub-category recognition with a limited sample size, where performance improvement encounters a bottleneck.

[0004] To improve the environmental robustness of models, introducing and fusing multimodal data such as infrared data is an effective approach. However, existing multimodal ship identification methods often face problems of spectral interference and modal aliasing. Since the data captured by sensors with different imaging mechanisms have significant differences in feature distribution, simple feature stitching or early fusion strategies are difficult to achieve effective alignment and complementarity of different modal features in a unified semantic space, and may even introduce noise, leading to a decrease in the discriminative power of the fused features.

[0005] Furthermore, existing methods generally lack explicit modeling and utilization of ship hierarchical logic. High-level semantic information (such as ship category) cannot effectively guide the network to focus on local discriminative regions crucial for mid- and low-level identification tasks (such as model differentiation and individual identification), leading to semantic inconsistencies or drift in feature transmission across levels. How to design a method that can effectively align multimodal data while deeply integrating hierarchical semantic constraints to guide multi-level discriminative feature learning has become a key technical problem that urgently needs to be solved in the field of fine-grained ship identification. Summary of the Invention

[0006] To address the above problems, the present invention aims to provide a multimodal ship hierarchical fine-grained collaborative identification method and system. Through cross-modal feature alignment and dynamic fusion, multi-scale feature extraction based on attention mechanism, and hierarchical progressive optimization and multi-task collaborative identification, it effectively integrates visible light, near-infrared and thermal infrared information, significantly improving the identification accuracy and robustness of ships at the category, model and individual levels under complex sea conditions.

[0007] To achieve the above objectives, the present invention employs the following technical solution: In a first aspect, embodiments of this application provide a multimodal ship hierarchical fine-grained collaborative identification method, comprising the following steps: S1: Acquire the multimodal image data of the ship to be identified, including visible light images, near-infrared images and thermal infrared images, and preprocess each modal image to generate a standardized multimodal input tensor; S2: The multimodal input tensor is input into the multimodal feature alignment module of the HSR-Net model to extract the independent features of each modality. The independent features of different modalities are transformed to a unified number of channels and resolution through semantic space alignment. A dynamic gating fusion strategy is used for weighted fusion to obtain the fused multimodal features. The HSR-Net model includes a multimodal feature alignment module, a backbone network based on a hierarchical window attention mechanism, inter-layer optimization units, and a multi-task branch module. S3: Input the fused multimodal features into the backbone network based on the hierarchical window attention mechanism to extract multi-scale feature maps, which include low-level detail feature maps, mid-level structural feature maps and high-level semantic feature maps; S4: Input the multi-scale feature map into the inter-layer optimization unit, and sequentially perform CBAM feature calibration and Cross... Attention semantic guidance and Ship HPFO hierarchical progressive optimization outputs a semantically enhanced and hierarchically aligned target feature tensor. S5: Input the target feature tensor into the three parallel branches of the multi-task branch module: class level, type level, and individual level. Output the predicted probability distribution of each level through the global average pooling layer and fully connected classification layer of each parallel branch. S6: Based on the predicted probability distribution of each level, generate the ship's category, model, and individual identification results.

[0008] In an optional implementation, step S1 includes: Acquire multimodal image data of ships to be identified in the same scene. The multimodal image data of ships includes visible light images (VIS), near-infrared images (NIR), and thermal infrared images (TIR). Store the multimodal image data of ships in the CM-Ship-ReID dataset and record the corresponding category, model, and individual annotation information. For each type of ship multimodal image data, image normalization, multi-scale cropping, rotation, and noise enhancement are performed to generate standardized multimodal input tensors. ;in, Input a tensor to the visible light image. Input tensors to near-infrared images, Input tensors for thermal infrared images.

[0009] In an optional implementation, step S2 includes: Input the visible light image into the tensor respectively Near-infrared image input tensor and thermal infrared image input tensor The input is fed into the corresponding independent branch in the multimodal feature alignment module. Continuous convolutional layers are used to extract shallow physical features from the input tensors of each modality, preserving the modality-uniqueness of the input tensors to obtain the corresponding initial feature maps. , , The independent branches include the VIS branch, the NIR branch, and the TIR branch. Alignment layer in each individual branch Convolution operations transform initial feature maps from different modalities into maps with the same number of channels. and spatial resolution Generate corresponding alignment features , , ; By employing a dynamic fusion strategy based on a gating mechanism, alignment features are... , , The concatenation is performed, and a global context descriptor is obtained through global average pooling. This descriptor is then input into the MLP layer and Softmax function of the multimodal feature alignment module to adaptively calculate the alignment features. , , Confidence weight ; Through formula Calculate the weighted fusion features , as the multimodal features after fusion.

[0010] In an optional implementation, step S3 includes: The fused multimodal features The input network is based on a hierarchical window attention mechanism backbone, and the low-level detail feature maps are output using Stage 2, Stage 3, and Stage 4 of the Swin Transformer. Mid-layer structure feature diagram and high-level semantic feature maps .

[0011] In an optional implementation, step S4 includes: The multi-scale feature map The input layer optimization unit includes a CBAM feature calibration module, a Cross-Attention semantic guidance module, and a Ship-HPFO hierarchical progressive optimization module. The CBAM feature calibration module was used to perform multi-scale feature maps. Channel attention extraction and spatial attention extraction are performed to generate saliency enhancement features at three scales, including Significance enhancement features , Significance enhancement features , Significance enhancement features ; The saliency enhancement features are semantically guided by the Cross-Attention semantic guidance module, and the enhanced features incorporating semantic guidance are calculated using the following formula. :

[0012]

[0013] in, , , These are the learnable linear transformation matrices. The normalization coefficient for the feature dimension; Incorporating semantically guided enhanced features Significance enhancement features are superimposed back to the original through residual connections. On top, perform feature updates; Through the Ship-HPFO hierarchical progressive optimization module Constructing high-level semantic benchmarks, and Mid-layer semantic injection and validation, Perform underlying texture enhancement and guidance to generate target features. , constitute the target feature tensor ; The specific formula for calculating the target feature is as follows:

[0014]

[0015]

[0016] in, express Convolution operations are used for feature reconstruction. For batch normalization, It is the Sigmoid activation function. and These are the learnable balance coefficients.

[0017] In an optional implementation, step S5 includes: target feature tensor Input the feature type into the class-level branch, type-level branch, and individual-level branch of the multi-task branching module respectively; In the three parallel branches, the space dimension is first compressed through the GAP layer. The feature vector is obtained, then mapped to the corresponding class space dimension through the FC layer, and finally the prediction probability distribution is output using the Softmax function; The predicted probability distribution includes coarse-grained category probabilities. Fine-grained model probability Individual identity probability .

[0018] In an optional implementation, step S6 includes: Based on the predicted probability distributions at each level, the predicted probability distributions at the category, type, and individual levels are obtained. The label corresponding to the maximum probability value is selected to generate the ship's category, model, and individual identification results. In an optional implementation, the training process of the HSR-Net model includes: Constructing a composite loss function ; in, It is a multi-task basic classification loss used to measure the difference between the predicted probability distribution and the true label; For logical consistency constraints, it is used to constrain the logical consistency between predictions at different levels, ensuring that they conform to the hierarchical relationships at the class, type, and individual levels; To enhance contrast, constraints are used to constrain the distribution of samples in the feature embedding space, thereby improving the discriminative power and cross-level semantic consistency of features. , These are hyperfunctions; Anchor point samples are randomly sampled from a pre-defined multimodal training set. Each sample contains paired visible light, near infrared, and thermal infrared images and their corresponding category, model, and individual annotation information. The sample is input into the HSR-Net model, and then passes through the forward computation of the multimodal feature alignment module, the backbone network based on the hierarchical window attention mechanism, the inter-layer optimization unit, and the multi-task branch module in sequence, outputting the predicted probability distributions at the class, type, and individual levels. Positive and negative sample pairs are constructed in the feature embedding space. The positive sample pairs are composed of feature vectors of the same ship target in different branches, and the negative sample pairs are composed of feature vectors of different ship targets. Using the positive and negative sample pairs, a contrast enhancement constraint is calculated to enhance the intra-class compactness and inter-class separability of features; Simultaneously calculate the total composite loss using the composite loss function; Based on the composite total loss, the gradient is calculated using the backpropagation algorithm, and all learnable parameters of the multimodal feature alignment module, the backbone network based on the hierarchical window attention mechanism, the inter-layer optimization unit, and the multi-task branch module are updated synchronously. Repeat the above process for iterative optimization until the model converges and the parameters of the trained HSR-Net model are obtained.

[0019] Secondly, embodiments of this application also provide a multimodal ship hierarchical fine-grained collaborative identification system, including: The data acquisition and preprocessing module is used to acquire multimodal image data of the ship to be identified, including visible light images, near-infrared images and thermal infrared images, and to preprocess each modal image to generate standardized multimodal input tensors; The multimodal feature alignment module is used to input the multimodal input tensor into the multimodal feature alignment module of the HSR-Net model, extract the independent features of each modality, transform the independent features of different modalities to a unified number of channels and resolution through semantic space alignment, and perform weighted fusion using a dynamic gating fusion strategy to obtain the fused multimodal features; the HSR-Net model includes the multimodal feature alignment module, a backbone network based on a hierarchical window attention mechanism, inter-layer optimization units, and a multi-task branch module; The multi-scale feature extraction module is used to input the fused multimodal features into the backbone network based on the hierarchical window attention mechanism to extract multi-scale feature maps, which include low-level detail feature maps, mid-level structural feature maps and high-level semantic feature maps. The inter-layer optimization module is used to input the multi-scale feature map into the inter-layer optimization unit and sequentially perform CBAM feature calibration and Cross-Layer Optimization. Attention semantic guidance and Ship HPFO hierarchical progressive optimization outputs a semantically enhanced and hierarchically aligned target feature tensor. The multi-task collaborative recognition module is used to input the target feature tensor into the three parallel branches of the multi-task branch module: class level, type level, and individual level. The prediction probability distribution of each level is output through the global average pooling layer and fully connected classification layer of each parallel branch. The inference output module is used to generate the ship's category, model, and individual identification results based on the predicted probability distribution of each level.

[0020] Thirdly, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the multimodal ship hierarchical fine-grained collaborative identification method as described in any of the above.

[0021] As can be seen from the above technical solutions, the present invention has the following advantages: The multimodal ship hierarchical fine-grained collaborative recognition method provided in this application constructs a multimodal feature alignment mechanism (CMAL), which achieves dynamic alignment and fusion of visible light, near-infrared, and thermal infrared images in a unified semantic space through gating mechanisms and consistency constraints. Simultaneously, based on inter-layer optimization units (IHOU) and multi-task collaborative branches, high-level semantics guide the focusing of low-level features, and hierarchical consistency and contrast enhancement constraints force fine-grained subclass predictions to be constrained by the coarse-grained parent class distribution, and narrows the distance between positive samples across levels, further improving the discriminability of features and achieving better hierarchical classification results.

[0022] The cross-modal feature alignment mechanism constructed in this application effectively solves the problems of spectral interference and distribution differences among heterogeneous sensors. Under low-light conditions such as nighttime, rain, and fog, or in severe weather conditions, the model can automatically enhance the weight of infrared thermal radiation features, achieving complementary advantages of multimodal information and significantly improving the environmental robustness of the ship identification system in all-weather, all-time scenarios.

[0023] The IHOU interlayer optimization unit designed in this application explicitly injects high-level category semantics into low-level features by constructing a top-down progressive guided path of "class level → type level → individual level". This enables the model to filter out background noise from ocean waves using high-level semantics when performing individual-level recognition, and to accurately focus on key fine-grained components such as vertical launch systems and radomes. As a result, it significantly improves the recognition accuracy for extremely similar models and specific individuals without adding an additional detection network.

[0024] The multi-task parallel prediction architecture constructed in this application can simultaneously output three levels of information: coarse-grained category, fine-grained model, and individual identity, under a unified feature extraction framework.

[0025] The dual constraint mechanism constructed in this application not only ensures the logical correctness of the recognition results, but also enhances the ability to distinguish subtle differences, greatly improving the application value of the system in high-reliability scenarios.

[0026] This application achieves significant improvements in the collaborative recognition of ship targets at the fine-grained level by iteratively optimizing the HSR-Net model based on a multi-level fine-grained collaborative recognition framework network. This optimization fully leverages the complementary characteristics of visible light and infrared images to overcome environmental interference and spectral differences caused by complex sea conditions within a unified feature space. Furthermore, it utilizes semantic guidance and logical constraint mechanisms between levels to force the model to follow the objective cognitive logic of "type-detail-individual," eliminating semantic drift and prediction conflicts in multi-grained recognition. Ultimately, this significantly improves the collaborative recognition performance of ship targets at the fine-grained level from two dimensions: data adaptability and feature discriminative power. Attached Figure Description

[0027] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A flowchart illustrating the multimodal ship hierarchical fine-grained collaborative identification method provided in this application.

[0029] Figure 2 This is a schematic diagram of the structure of the HSR-Net model provided in this application.

[0030] Figure 3 This is a schematic diagram of the cross-modal alignment mechanism provided in this application.

[0031] Figure 4 This is a schematic diagram of the CBAM feature calibration module provided in this application.

[0032] Figure 5 This is a schematic diagram of the Cross-Attention semantic guidance module provided in this application.

[0033] Figure 6 This is a schematic diagram of the Ship-HPFO hierarchical progressive optimization module provided in this application.

[0034] Figure 7 A schematic diagram of the structure of the multimodal ship hierarchical fine-grained collaborative identification system provided in this application.

[0035] Figure 8 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0036] The various embodiments of this disclosure will be described more fully in the detailed steps of the multimodal ship hierarchical fine-grained cooperative identification method described below. This disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of this disclosure to the specific embodiments disclosed herein, but rather this disclosure should be understood to cover all adjustments, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of this disclosure.

[0037] In the following, the terms “comprising” or “may include”, which may be used in various embodiments of this disclosure, indicate the presence of the disclosed functions, operations, or elements, and do not limit the addition of one or more functions, operations, or elements. Furthermore, as used in various embodiments of this disclosure, the terms “comprising,” “having,” and their cognates are intended only to indicate a particular feature, number, step, operation, element, component, or combination of the foregoing, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing, or the possibility of adding one or more combinations of the foregoing.

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] Please see Figure 1 The diagram shows a flowchart of a multimodal ship hierarchical fine-grained collaborative identification method in a specific embodiment. The method includes: S1: Acquire the multimodal image data of the ship to be identified, including visible light images, near-infrared images and thermal infrared images, and preprocess each modal image to generate a standardized multimodal input tensor.

[0040] In a specific implementation, this step aims to acquire multimodal image data of ships in the same scene and perform standardized preprocessing on each data to eliminate scale and distribution differences caused by imaging from different sensors, ensure the quality and consistency of the input data, and construct a standardized network input tensor. The specific steps are as follows: Multimodal source data acquisition: Acquire multimodal image data of ships to be identified in the same scene, including visible light images (VIS), near-infrared images (NIR), and thermal infrared images (TIR). The dataset can be the CM-Ship-ReID dataset, covering annotation information at three levels: category, model, and individual.

[0041] Data preprocessing: Systematic preprocessing and enhancement operations are performed on the data of the three modalities, including image normalization (e.g., adjusting to...). Multi-scale cropping, rotation, and noise enhancement operations are performed to generate multimodal input tensors. .in, Input a tensor to the visible light image. Input tensors to near-infrared images, Input tensors for thermal infrared images.

[0042] S2: The multimodal input tensor is input into the multimodal feature alignment module of the HSR-Net model to extract the independent features of each modality. The independent features of different modalities are transformed to a unified number of channels and resolution through semantic space alignment. A dynamic gating fusion strategy is used for weighted fusion to obtain the fused multimodal features. The HSR-Net model includes a multimodal feature alignment module, a backbone network based on a hierarchical window attention mechanism, inter-layer optimization units, and a multi-task branch module.

[0043] In this specific implementation, this step utilizes the multimodal feature alignment module of the HSR-Net model to receive standardized multimodal input tensors and employs the Multimodal Feature Alignment (CMAL) mechanism to address the significant distribution differences and spectral interference between visible light, near-infrared, and thermal infrared data. By executing a processing flow of "independent extraction - unified alignment - dynamic fusion," heterogeneous signals are mapped to a unified semantic space and complementary aggregation is achieved, ultimately outputting the fused multimodal features. It is also used as input to the backbone network of the HSR-Net model.

[0044] It is worth noting that this method proposes a multi-modal, multi-level, fine-grained collaborative identification framework for ships (HSR-Net). This framework includes a multi-modal feature alignment mechanism (CMAL), a Swing Transformer-based backbone network, inter-layer optimization units (IHOU), and multi-task collaborative branches. For example... Figure 2 As shown, the HSR-Net model built based on this framework includes a multimodal feature alignment module, a backbone network based on a hierarchical window attention mechanism, inter-layer optimization units, and a multi-task branch module.

[0045] S3: Input the fused multimodal features into the backbone network based on the hierarchical window attention mechanism to extract multi-scale feature maps, which include low-level detail feature maps, mid-level structural feature maps, and high-level semantic feature maps.

[0046] In a specific implementation, the fused multimodal features The input network is based on a hierarchical window attention mechanism backbone, and the low-level detail feature maps are output using Stage 2, Stage 3, and Stage 4 of the Swin Transformer. Mid-layer structure feature diagram and high-level semantic feature maps .

[0047] For example, a backbone network based on a hierarchical window attention mechanism (SwinTransformer) is constructed to fuse the features output from step S2. Input network. To balance the global semantics and local details required for recognition, feature maps output from Stage 2, Stage 3, and Stage 4 of the SwinTransformer are specifically selected as the basis for subsequent processing. These are denoted as follows: (Underlying details) (Middle layer structure) (High-level semantics) These three sets of features correspond to different resolutions and levels of semantic abstraction.

[0048] S4: Input the multi-scale feature map into the inter-layer optimization unit, and sequentially perform CBAM feature calibration and Cross... Attention semantic guidance and Ship HPFO hierarchical progressive optimization outputs a semantically enhanced and hierarchically aligned target feature tensor.

[0049] In a specific implementation, the inter-layer optimization unit includes a CBAM feature calibration module, a Cross-Attention semantic guidance module, and a Ship-HPFO hierarchical progressive optimization module.

[0050] Multi-scale feature maps After inputting the interlayer optimization unit, the multi-scale feature maps are first calibrated using the CBAM feature calibration module. Channel attention extraction and spatial attention extraction are performed to generate saliency enhancement features at three scales, including Significance enhancement features , Significance enhancement features , Significance enhancement features .

[0051] Then, the Cross-Attention semantic guidance module is used to semantically guide the saliency enhancement features, and the enhanced features that incorporate the semantic guidance are calculated. .

[0052] Finally, the Ship-HPFO hierarchical progressive optimization module was used to optimize the... Constructing high-level semantic benchmarks, and Mid-layer semantic injection and validation, Perform underlying texture enhancement and guidance to generate target features. , constitute the target feature tensor .

[0053] S5: Input the target feature tensor into the three parallel branches of the multi-task branch module: class level, type level, and individual level. Output the predicted probability distribution of each level through the global average pooling layer and fully connected classification layer of each parallel branch.

[0054] In a specific implementation, the target feature tensor is first... Based on the feature type, the data is input into the class-level branch, type-level branch, and individual-level branch of the multi-task branching module, respectively; then, in the three parallel branches, the data first passes through a GAP layer to compress the spatial dimension. The feature vector is obtained, then mapped to the corresponding class space dimension through a fully connected (FC) layer, and finally the predicted probability distribution is output using the Softmax function. This predicted probability distribution includes coarse-grained class probabilities. Fine-grained model probability Individual identity probability .

[0055] In this step, the multi-task branching module constructs a multi-task branching structure, including three parallel branches: class-level, type-level, and instance-level. The IHOU-optimized features are input into these three branches respectively. Each task branch consists of a global average pooling layer (GAP) and a fully connected classification layer (FC).

[0056] The feature tensor first passes through a GAP layer, compressing the spatial dimension to... The feature vector is obtained, then mapped to the corresponding class space dimension through an FC layer, and finally the predicted probability distribution is output using the Softmax function: the Class branch outputs the coarse-grained class probability of the ship. (e.g., merchant ships); the Type branch outputs fine-grained model probabilities. The Instance branch outputs the probability of an individual's identity. (e.g., specific hull number). Through parallel output, a three-level collaborative identification of ship targets ("type-details-individual") is achieved.

[0057] S6: Based on the predicted probability distribution of each level, generate the ship's category, model, and individual identification results.

[0058] In a specific implementation, the predicted probability distributions at the class, type, and individual levels are first obtained based on the predicted probability distributions at each level, and the label corresponding to the maximum probability is selected as the ship's category, model, and individual identity recognition result.

[0059] For example, this step implements the identification of unknown samples using a trained HSR-Net model. Forward inference computation is performed by retaining the CMAL module, feature extraction backbone, IHOU unit, and multi-task classification branch from the trained HSR-Net model. The multimodal ship images (visible light, near-infrared, and thermal infrared) to be tested are input into the model. The above process is executed to obtain the predicted probability distributions at three levels. The label corresponding to the maximum probability is selected as the final result, and the "type-description-individual" three-level collaborative predicted category of the sample is output.

[0060] In this embodiment, through multimodal feature alignment and dynamic fusion, multi-scale feature extraction based on attention mechanism, hierarchical semantic guidance and progressive optimization, as well as multi-task collaborative training and hierarchical loss constraints, effective complementarity and semantic alignment of visible light, near-infrared and thermal infrared information are achieved, which significantly improves the recognition accuracy, robustness and logical consistency of ships from coarse-grained category to fine-grained individual identity in complex sea conditions.

[0061] Additionally, it should be noted that the HSR-Net model used in this invention is a pre-trained model. The model training process includes: constructing a composite loss function comprising multi-task classification loss, hierarchical logical consistency constraint loss, and hierarchical contrast enhancement constraint loss; and performing end-to-end joint training of the HSR-Net model using multimodal training samples.

[0062] Specifically, a composite loss function was first constructed. ; in, It is a multi-task basic classification loss used to measure the difference between the predicted probability distribution and the true label; For logical consistency constraints, it is used to constrain the logical consistency between predictions at different levels, ensuring that they conform to the hierarchical relationships at the class, type, and individual levels; To enhance contrast, constraints are used to constrain the distribution of samples in the feature embedding space, thereby improving the discriminative power and cross-level semantic consistency of features. , These are superfunctions.

[0063] Then, the HSR-Net model is jointly trained end-to-end using multimodal training samples. The specific process is as follows: Anchor point samples are randomly sampled from the multimodal training set. Each sample contains paired visible light, near infrared, and thermal infrared images and their corresponding "category, model, and individual annotation information". The sample is input into the HSR-Net model, and then passes through the forward computation of the multimodal feature alignment module, the backbone network based on the hierarchical window attention mechanism, the inter-layer optimization unit, and the multi-task branch module in sequence, outputting the predicted probability distributions at the class, type, and individual levels. Positive and negative sample pairs are constructed in the feature embedding space. The positive sample pairs are composed of feature vectors of the same ship target in different branches, and the negative sample pairs are composed of feature vectors of different ship targets. Using the positive and negative sample pairs, a contrast enhancement constraint is calculated to enhance the intra-class compactness and inter-class separability of features; Simultaneously calculate the total composite loss using the composite loss function; Based on the composite total loss, the gradient is calculated using the backpropagation algorithm, and all learnable parameters of the multimodal feature alignment module, the backbone network based on the hierarchical window attention mechanism, the inter-layer optimization unit, and the multi-task branch module are updated synchronously. Repeat the above process for iterative optimization until the model converges and the parameters of the trained HSR-Net model are obtained.

[0064] In one embodiment of the present invention, based on step S2, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0065] refer to Figure 3 As shown, this embodiment discloses a cross-modal alignment mechanism based on a multimodal feature alignment module. The specific implementation process is as follows: S201 Independent Feature Extraction: Construct a feature extraction subnetwork containing three parallel branches to extract features from the visible light image processed in step 1. Near-infrared images and thermal infrared images Each input is a separate branch, and shallow physical features of each modality are extracted using consecutive convolutional layers, preserving modality uniqueness (such as texture of VIS and thermal profile of TIR) to obtain the initial feature map. , , .

[0066] S202 Semantic Space Alignment: The alignment layer adopts... Convolution operations convert features from different modalities into features with the same number of channels. and spatial resolution Alignment features are obtained. , , At this stage, cross-modal consistency constraints are introduced. By minimizing the feature distance of the same target in different modalities, the features of different modalities are forced to remain consistent in the embedding space, thereby eliminating "spectral interference".

[0067] S203 Dynamic Gated Fusion: Introduces a dynamic fusion strategy based on a gating mechanism. Three aligned features are concatenated, and a global average pooling (GAP) is used to obtain the global context descriptor. This descriptor is then input into an MLP layer and a Softmax function to adaptively calculate the confidence weights for each modality. .

[0068] The calculation formula is as follows:

[0069] Final output weighted fusion features: .

[0070] In one embodiment of the present invention, based on step S4, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0071] This step utilizes Inter-Layer Optimization Units (IHOUs) to guide low-level features to focus on key discriminative regions using high-level semantics. It comprises three sequentially connected modules: CBAM feature calibration, Cross-Attention semantic guidance, and Ship-HPFO hierarchical progressive optimization. refer to Figure 4 As shown, the CBAM feature calibration module is used for multi-scale feature maps. Saliency enhancement is performed separately. This module aims to address the problem of indistinct ship target features under complex sea conditions. By sequentially performing "feature selection in the channel dimension" and "region focusing in the spatial dimension," background noise such as waves and islands is suppressed, resulting in outputting significantly enhanced features. This provides a high-quality feature foundation for subsequent semantic guidance.

[0072] The specific processing flow of the CBAM feature calibration module includes the following two cascaded stages: Channel attention extraction: Spatial information is aggregated using global max pooling and average pooling, and channel weights are generated through shared MLP and sigmoid activation. For input features Channel weighting is performed to obtain .

[0073]

[0074]

[0075] 2. Spatial attention extraction: for Max pooling and average pooling are performed along the channel axis. Convolution and Sigmoid activation generate spatial weights This allows for focusing on specific spatial regions.

[0076]

[0077]

[0078] The saliency enhancement features at three scales are obtained, denoted as follows: (Corresponding to Stage 4) (Corresponding to Stage 3) and (Corresponds to Stage 2).

[0079] refer to Figure 5 As shown, the Cross-Attention semantic guidance module is used to semantically guide saliency enhancement features. Specifically, the high-level class features processed by CBAM are denoted as follows: The low-level features after CBAM processing are denoted as .

[0080] Unlike traditional designs, this module retains a wealth of details. As a query vector, it is highly abstract. As key and value vectors, this design allows low-level features to proactively "query" high-level semantic context, explicitly injecting class-level semantic priors while maintaining high-resolution type / individual-level structure.

[0081]

[0082]

[0083] in, These are high-level class features enhanced with CBAM saliency; the Type layer features are represented as follows: Low-level features enhanced by CBAM ; For feature dimensions. , , It is a learnable linear transformation matrix. is the normalization coefficient for the feature dimension. To integrate semantically guided enhanced features, residual connections are used to superimpose them back into the original features. This achieves a unity of semantics and details.

[0084] refer to Figure 6 As shown, the Ship-HPFO hierarchical progressive optimization module constructs a complete progressive feature enhancement path of "class level → type level → individual level". Through a top-down cascading fusion mechanism, it uses high-level semantics to perform implicit logical verification and texture enhancement on low-level features, ensuring the consistency of output features in multi-granularity semantic spaces.

[0085] For example, the Ship-HPFO hierarchical progressive optimization module achieves hierarchical feature reconstruction through the following three cascaded stages: 1. Construction of High-Level Semantic Baseline: First, channel dimensionality reduction and nonlinear mapping are performed on high-level features to extract robust global semantic representations, which serve as "semantic anchors" for the entire hierarchical optimization chain.

[0086] in, For optimized class-level features, express Convolution operations are used for feature reconstruction. This is for batch normalization. This is the Sigmoid activation function.

[0087] 2. Mid-level semantic injection and validation: Constructing a feature fusion path from "class level to type level". This involves integrating optimized high-level features. As a semantic prior, it is injected into the mid-level features. Introducing a learnable balance coefficient. This enables dynamic weighting and logical verification of mid-level features by high-level semantics.

[0088] Output It inherits the global semantic constraints of the Class level and retains the type-level recognition brought by Cross-Attention, thus eliminating semantic drift between levels.

[0089] 3. Bottom-level texture enhancement and guidance: Constructing a progressive guidance path from "type level to individual level". This involves optimizing the mid-layer features. Integrating into underlying features In this process, the structural information contained in the middle-layer features is used to guide the focusing of the lower-layer texture. A balance coefficient is introduced. Controlling guidance intensity:

[0090] Output It achieves progressive semantic guidance across the entire chain, ensuring that individual-level features, while possessing rich local textures (such as hull numbers and microstructures), strictly conform to the semantic logic of the upper-level categories.

[0091] After the above progressive optimization, the final output is a feature tensor that is independent in three dimensions but highly semantically coordinated. These three sets of features will be fed into the subsequent multi-task prediction branch, and used for ship category classification, model identification, and individual identity re-identification tasks, respectively.

[0092] In one embodiment of the present invention, a method for constructing a dual constraint mechanism and training the HSR-Net model is also disclosed. During the model training phase, a composite objective function is constructed, comprising multi-task classification loss, hierarchical logical consistency constraint (HCC), and hierarchical contrast enhancement constraint (HCE). A joint optimization strategy is then employed to perform end-to-end parameter updates on the HSR-Net model. Specifically, the process includes the following steps: I. Construction of the composite loss function: 1. Multi-task basic classification loss: Weighted cross-entropy loss is used as the fundamental driving force for model learning. It measures the predicted probability distribution of the three branch outputs respectively. The difference between the actual label and the real label.

[0093]

[0094] in, Measure the predicted probability distribution of the output of each of the three branches respectively. With real labels The differences between them.

[0095] 2. Logical Consistency Constraint: Based on the objectively existing semantic hierarchy of "type-detailed" relationships among ships, the prediction confidence of a child node should not be higher than the prediction confidence of its parent node (i.e., ...). A logical consistency loss is constructed to penalize samples that violate hierarchical logic, forcing the correctness of fine-grained subclass predictions to depend on the coarse-grained parent class.

[0096] 3. Contrast Enhancement Constraints: Within each branch at the class, type, and individual levels, contrastive learning tasks are constructed independently. Using the InfoNCE loss function, features of the same ship target are defined as positive samples, and features of other different ships are defined as negative samples. By maximizing the cosine similarity of positive sample pairs and minimizing the similarity of negative sample pairs, the intra-class compactness and inter-class separability of features are enhanced. The calculation formula is as follows:

[0097] To maintain semantic orientation consistency between "type-individual" and "category-entity" in the feature embedding space, cross-level feature alignment constraints are constructed. The calculation formula is as follows:

[0098] The final HCE total constraint loss is defined as: .

[0099] 4. Overall objective function: The above three parts of the loss are weighted and fused together.

[0100]

[0101] II. Joint Iterative Optimization Strategy: Adam was used as the optimizer for the model (with an initial learning rate of...). Batch size is Train the model 100 times (epochs) using the following steps: Step 1: Randomly sample anchor point samples (including paired visible light, near-infrared, and thermal infrared images) and their corresponding hierarchical labels (type-individual) from the multimodal training set. These constitute the input sample block.

[0102] Step 2: Forward computation. The sample block is input into the network, aligned by CMAL, extracted by Swing Transformer, and optimized by IHOU, outputting the prediction results. Simultaneous computation... and .

[0103] Step 3: Constructing positive and negative samples. Within the feature embedding space, features of the same ship at both the type and individual levels are defined as positive sample pairs, while features of different ships are defined as negative sample pairs.

[0104] Step 4: Backpropagation. Calculate based on positive and negative sample pairs. Calculate by combining all loss terms. The gradient is calculated using the backpropagation algorithm, and the parameters of CMAL, backbone network, IHOU and each branch are updated synchronously.

[0105] Step 5: Iterate. Repeat steps 1-4 until the model converges and output the trained HSR-Net model parameters.

[0106] like Figure 7As shown, the following are embodiments of the multimodal ship hierarchical fine-grained collaborative identification system provided in this disclosure. This system and the multimodal ship hierarchical fine-grained collaborative identification method of the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the multimodal ship hierarchical fine-grained collaborative identification system, please refer to the embodiments of the above multimodal ship hierarchical fine-grained collaborative identification method.

[0107] A multimodal ship hierarchical fine-grained collaborative identification system includes: The data acquisition and preprocessing module is used to acquire multimodal image data of the ship to be identified, including visible light images, near-infrared images and thermal infrared images, and to preprocess each modal image to generate standardized multimodal input tensors.

[0108] The multimodal feature alignment module is used to input the multimodal input tensor into the multimodal feature alignment module of the HSR-Net model, extract the independent features of each modality, transform the independent features of different modalities to a unified number of channels and resolution through semantic space alignment, and perform weighted fusion using a dynamic gating fusion strategy to obtain the fused multimodal features; the HSR-Net model includes the multimodal feature alignment module, a backbone network based on a hierarchical window attention mechanism, inter-layer optimization units, and a multi-task branch module.

[0109] The multi-scale feature extraction module is used to input the fused multimodal features into the backbone network based on the hierarchical window attention mechanism to extract multi-scale feature maps, which include low-level detail feature maps, mid-level structural feature maps, and high-level semantic feature maps.

[0110] The inter-layer optimization module is used to input the multi-scale feature map into the inter-layer optimization unit and sequentially perform CBAM feature calibration and Cross-Layer Optimization. Attention semantic guidance and Ship HPFO hierarchical progressive optimization outputs a semantically enhanced and hierarchically aligned target feature tensor.

[0111] The multi-task collaborative recognition module is used to input the target feature tensor into the three parallel branches of the multi-task branch module: class level, type level, and individual level. The predicted probability distribution of each level is output through the global average pooling layer and fully connected classification layer of each parallel branch.

[0112] The inference output module is used to generate the ship's category, model, and individual identification results based on the predicted probability distribution of each level.

[0113] The multimodal ship hierarchical fine-grained collaborative identification system provided in this embodiment effectively integrates heterogeneous information from multiple sources, including visible light, near-infrared, and thermal infrared, through cross-modal feature alignment and dynamic fusion mechanisms, significantly improving the complementarity and robustness of feature representation. It utilizes a backbone network based on hierarchical window attention for multi-scale feature extraction, taking into account both global semantics and local details. Furthermore, it enhances the discriminative power and consistency of features in multi-granularity identification tasks through progressive semantic guidance and hierarchical alignment via inter-layer optimization units (IHOUs). Combined with a multi-task collaborative prediction network and hierarchical loss constraints, it improves overall identification accuracy and generalization ability while ensuring that the three-level prediction of "type-detailed entity" conforms to real-world logic. Ultimately, under complex sea conditions and multimodal interference, it achieves high-accuracy and high-reliability collaborative identification of ship targets, from coarse-grained categories to fine-grained individual identities.

[0114] Figure 8 A schematic diagram of the hardware structure of an electronic device for implementing various embodiments of the present invention.

[0115] The multimodal ship-level fine-grained collaborative identification method provided in this application can be applied to electronic devices. Those skilled in the art will understand that the electronic device structure involved in the embodiments of this invention does not constitute a limitation on the electronic device. An electronic device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, the electronic device includes, but is not limited to, laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0116] Electronic devices may include processors, external memory interfaces, internal memory, universal serial bus (USB) interfaces, charging management modules, power management modules, batteries, wireless communication modules, audio modules, speakers, microphones, sensor modules, buttons, cameras, displays, and SIM card interfaces, etc.

[0117] A processor may include one or more processing units, such as: a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0118] The processor can serve as the nerve center and command center of an electronic device. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0119] The processor may also include memory for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can store instructions or data that the processor has just used or that are used repeatedly. If the processor needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces processor latency, and thus improves system efficiency.

[0120] An external storage interface (ESI) can be used to connect external memory cards, such as microSD cards, to expand the storage capacity of electronic devices. The external memory card communicates with the processor through the ESI to perform data storage functions, such as saving music and video files on the external memory card.

[0121] Internal memory can be used to store computer executable program code, which includes instructions. The processor executes various functional applications and data processing of electronic devices by running the instructions stored in internal memory. Internal memory can include a program storage area and a data storage area. Internal memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0122] Wireless communication functionality in electronic devices can be achieved through antennas, wireless communication modules, modem processors, and baseband processors.

[0123] Wireless communication modules can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies.

[0124] Electronic devices can implement audio functions through audio modules, speakers, receivers, microphones, headphone jacks, and application processors.

[0125] Electronic devices can achieve shooting functions through ISPs, cameras, video codecs, GPUs, displays, and application processors.

[0126] Electronic devices can achieve display functions through GPUs, displays, and application processors.

[0127] A GPU is a microprocessor for image processing, connected to the display screen and application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. A processor may include one or more GPUs, which execute program instructions to generate or modify display information.

[0128] A display screen is used to display images, videos, etc. A display screen includes a display panel.

[0129] The aforementioned electronic device realizes the multimodal ship hierarchical fine-grained collaborative recognition method of this application. By constructing the HSR-Net model and integrating multimodal feature alignment, hierarchical attention feature extraction, progressive semantic optimization and multi-task collaborative recognition mechanism, it achieves effective fusion and hierarchical semantic alignment of visible light, near-infrared and thermal infrared information, and achieves the beneficial effect of high-precision and high-robustness three-level fine-grained collaborative recognition of ship targets in complex environments.

[0130] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal ship hierarchical fine-grained collaborative identification method, characterized in that, Includes the following steps: S1: Acquire the multimodal image data of the ship to be identified, including visible light images, near-infrared images and thermal infrared images, and preprocess each modal image to generate a standardized multimodal input tensor; S2: Input the multimodal input tensor into the multimodal feature alignment module of the HSR-Net model, extract the independent features of each modality, transform the independent features of different modalities to a unified number of channels and resolution through semantic space alignment, and perform weighted fusion using a dynamic gating fusion strategy to obtain the fused multimodal features; The HSR-Net model includes a multimodal feature alignment module, a backbone network based on a hierarchical window attention mechanism, inter-layer optimization units, and a multi-task branch module. S3: Input the fused multimodal features into the backbone network based on the hierarchical window attention mechanism to extract multi-scale feature maps, which include low-level detail feature maps, mid-level structural feature maps and high-level semantic feature maps; S4: Input the multi-scale feature map into the inter-layer optimization unit, and sequentially perform CBAM feature calibration and Cross... Attention semantic guidance and Ship HPFO hierarchical progressive optimization outputs a semantically enhanced and hierarchically aligned target feature tensor. S5: Input the target feature tensor into the three parallel branches of the multi-task branch module: class level, type level, and individual level. Output the predicted probability distribution of each level through the global average pooling layer and fully connected classification layer of each parallel branch. S6: Based on the predicted probability distribution of each level, generate the ship's category, model, and individual identification results.

2. The multimodal ship hierarchical fine-grained collaborative identification method according to claim 1, characterized in that, Step S1 includes: Acquire multimodal image data of ships to be identified in the same scene. The multimodal image data of ships includes visible light images (VIS), near-infrared images (NIR), and thermal infrared images (TIR). Store the multimodal image data of ships in the CM-Ship-ReID dataset and record the corresponding category, model, and individual annotation information. For each type of ship multimodal image data, image normalization, multi-scale cropping, rotation, and noise enhancement are performed to generate standardized multimodal input tensors. ;in, Input a tensor to the visible light image. Input tensors to near-infrared images, Input tensors for thermal infrared images.

3. The multimodal ship hierarchical fine-grained collaborative identification method according to claim 2, characterized in that, Step S2 includes: Input the visible light image into the tensor respectively Near-infrared image input tensor and thermal infrared image input tensor The input is fed into the corresponding independent branch in the multimodal feature alignment module. Continuous convolutional layers are used to extract shallow physical features from the input tensors of each modality, preserving the modality-uniqueness of the input tensors to obtain the corresponding initial feature maps. , , The independent branches include the VIS branch, the NIR branch, and the TIR branch. Alignment layer in each individual branch Convolution operations transform initial feature maps from different modalities into maps with the same number of channels. and spatial resolution Generate corresponding alignment features , , ; By employing a dynamic fusion strategy based on a gating mechanism, alignment features are... , , The concatenation is performed, and a global context descriptor is obtained through global average pooling. This descriptor is then input into the MLP layer and Softmax function of the multimodal feature alignment module to adaptively calculate the alignment features. , , Confidence weight ; Through formula Calculate the weighted fusion features , as the multimodal features after fusion.

4. The multimodal ship hierarchical fine-grained collaborative identification method according to claim 3, characterized in that, Step S3 includes: The fused multimodal features The input network is based on a hierarchical window attention mechanism backbone, and the low-level detail feature maps are output using Stage 2, Stage 3, and Stage 4 of the Swin Transformer. Mid-layer structure feature diagram and high-level semantic feature maps .

5. The multimodal ship hierarchical fine-grained collaborative identification method according to claim 4, characterized in that, Step S4 includes: The multi-scale feature map The input layer optimization unit includes a CBAM feature calibration module, a Cross-Attention semantic guidance module, and a Ship-HPFO hierarchical progressive optimization module. The CBAM feature calibration module was used to perform multi-scale feature maps. Channel attention extraction and spatial attention extraction are performed to generate saliency enhancement features at three scales, including Significance enhancement features , Significance enhancement features , Significance enhancement features ; The saliency enhancement features are semantically guided by the Cross-Attention semantic guidance module, and the enhanced features incorporating semantic guidance are calculated using the following formula. : in, , , These are the learnable linear transformation matrices. The normalization coefficient for the feature dimension; Incorporating semantically guided enhanced features Significance enhancement features are superimposed back to the original through residual connections. On top, perform feature updates; Through the Ship-HPFO hierarchical progressive optimization module Constructing high-level semantic benchmarks, and Mid-layer semantic injection and validation, Perform underlying texture enhancement and guidance to generate target features. , constitute the target feature tensor ; The specific formula for calculating the target feature is as follows: in, express Convolution operations are used for feature reconstruction. For batch normalization, It is the Sigmoid activation function. and These are the learnable balance coefficients.

6. The multimodal ship hierarchical fine-grained collaborative identification method according to claim 5, characterized in that, Step S5 includes: target feature tensor Input the feature type into the class-level branch, type-level branch, and individual-level branch of the multi-task branching module respectively; In the three parallel branches, the space dimension is first compressed through the GAP layer. The feature vector is obtained, then mapped to the corresponding class space dimension through the FC layer, and finally the prediction probability distribution is output using the Softmax function; The predicted probability distribution includes coarse-grained category probabilities. Fine-grained model probability Individual identity probability .

7. The multimodal ship hierarchical fine-grained collaborative identification method according to claim 6, characterized in that, Step S6 includes: Based on the predicted probability distributions at each level, the predicted probability distributions at the category, type, and individual levels are obtained. The label corresponding to the maximum probability value is selected to generate the ship's category, model, and individual identification results.

8. The multimodal ship hierarchical fine-grained collaborative identification method according to claim 6, characterized in that, The training process of the HSR-Net model includes: Constructing a composite loss function ; in, It is a multi-task basic classification loss used to measure the difference between the predicted probability distribution and the true label; For logical consistency constraints, it is used to constrain the logical consistency between predictions at different levels, ensuring that they conform to the hierarchical relationships at the class, type, and individual levels; To enhance contrast, constraints are used to constrain the distribution of samples in the feature embedding space, thereby improving the discriminative power and cross-level semantic consistency of features. , These are hyperfunctions; Anchor point samples are randomly sampled from a pre-set multimodal training set. Each sample contains paired visible light, near infrared, and thermal infrared images and their corresponding category, model, and individual annotation information. The sample is input into the HSR-Net model, and then passes through the forward computation of the multimodal feature alignment module, the backbone network based on the hierarchical window attention mechanism, the inter-layer optimization unit, and the multi-task branch module in sequence, outputting the predicted probability distributions at the class, type, and individual levels. Positive and negative sample pairs are constructed in the feature embedding space. The positive sample pairs are composed of feature vectors of the same ship target in different branches, and the negative sample pairs are composed of feature vectors of different ship targets. Using the positive and negative sample pairs, a contrast enhancement constraint is calculated to enhance the intra-class compactness and inter-class separability of features; Simultaneously calculate the total composite loss using the composite loss function; Based on the composite total loss, the gradient is calculated using the backpropagation algorithm, and all learnable parameters of the multimodal feature alignment module, the backbone network based on the hierarchical window attention mechanism, the inter-layer optimization unit, and the multi-task branch module are updated synchronously. Repeat the above process for iterative optimization until the model converges and the parameters of the trained HSR-Net model are obtained.

9. A multimodal ship hierarchical fine-grained collaborative identification system, characterized in that, The system employs the multimodal ship hierarchical fine-grained collaborative identification method as described in any one of claims 1 to 8; The system includes: The data acquisition and preprocessing module is used to acquire multimodal image data of the ship to be identified, including visible light images, near-infrared images and thermal infrared images, and to preprocess each modal image to generate standardized multimodal input tensors; The multimodal feature alignment module is used to input the multimodal input tensor into the multimodal feature alignment module of the HSR-Net model, extract the independent features of each modality, transform the independent features of different modalities to a unified number of channels and resolution through semantic space alignment, and perform weighted fusion using a dynamic gating fusion strategy to obtain the fused multimodal features; the HSR-Net model includes the multimodal feature alignment module, a backbone network based on a hierarchical window attention mechanism, inter-layer optimization units, and a multi-task branch module; The multi-scale feature extraction module is used to input the fused multimodal features into the backbone network based on the hierarchical window attention mechanism to extract multi-scale feature maps, which include low-level detail feature maps, mid-level structural feature maps and high-level semantic feature maps. The inter-layer optimization module is used to input the multi-scale feature map into the inter-layer optimization unit and sequentially perform CBAM feature calibration and Cross-Layer Optimization. Attention semantic guidance and Ship HPFO hierarchical progressive optimization outputs a semantically enhanced and hierarchically aligned target feature tensor. The multi-task collaborative recognition module is used to input the target feature tensor into the three parallel branches of the multi-task branch module: class level, type level, and individual level. The prediction probability distribution of each level is output through the global average pooling layer and fully connected classification layer of each parallel branch. The inference output module is used to generate the ship's category, model, and individual identification results based on the predicted probability distribution of each level.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the multimodal ship hierarchical fine-grained collaborative identification method as described in any one of claims 1 to 8.

Images