Explanatable cross-modal target re-identification method and system based on prototype learning

Abstract

The invention relates to an interpretable cross-modal target re-identification method and system based on prototype learning. The method comprises the following steps: constructing a prototype cross-modal network for carrying out channel adaptation on image samples in a multi-modal image sample set, extracting multi-scale fusion features and obtaining a discriminative feature vector through channel weighting, traversing the spatial position of the discriminative feature vector and extracting local features, and carrying out multi-scale fusion on the image samples in the multi-modal image sample set; selecting a local feature with the highest similarity for each initial prototype, fusing the local feature with the initial prototype to obtain a new prototype, iteratively updating the prototype corresponding to each identity tag to obtain a dynamic prototype set, generating a prototype activation thermodynamic diagram about a discriminative feature vector and each prototype in the dynamic prototype set, and classifying and outputting a classification result of a corresponding image sample; and training the prototype cross-modal network, and carrying out cross-modal re-identification by using the trained prototype cross-modal network. By adopting the method, the cross-modal recognition precision, scene adaptability and interpretability of target re-recognition in a complex environment can be improved.

Description

Technical Field

[0001] This application relates to the field of target re-identification technology, and in particular to an interpretable cross-modal target re-identification method and system based on prototype learning. Background Technology

[0002] With the development of deep learning technology, prototype learning techniques have emerged. The core feature of this technique is that it learns one or more of the most representative feature vectors (i.e., prototypes) for each category. Classification or retrieval is completed by calculating the similarity between the input sample and the prototype. This process is transparent and traceable, effectively alleviating the "black box" problem of deep neural networks. It exhibits unique advantages in feature representation and interpretability, and has therefore achieved significant results in single-modal target re-identification tasks (such as pedestrian and vehicle re-identification). Some studies improve recognition accuracy by learning shared features from multiple perspectives, while others construct local discriminative features and map them to the original image to optimize re-identification performance.

[0003] In the critical task of ship re-identification, a multimodal visual recognition problem, current traditional methods and existing technologies are mainly divided into two categories: one focuses on single-modal or dual-modal scenarios, including using SIFT / HOG algorithms to extract features, directly performing global average pooling on multimodal features, and setting up a modality-specific feature extraction network to fuse features; the other uses generative adversarial networks (GANs) to convert one modality (such as infrared) image into another modality (such as visible light), transforming the cross-modal problem into a single-modal matching problem, or fusing visual features with manually input ship model features to improve performance.

[0004] However, current ship re-identification methods or traditional approaches suffer from significant inefficiencies and security vulnerabilities. Inefficiency manifests in two ways: first, the inability to bridge the "modal gap"—existing methods rely on feature alignment or domain adaptation strategies that are unstable under complex conditions such as day-night cycles and severe weather, leading to decreased cross-modal retrieval performance; second, a lack of dynamic adaptation mechanisms—relying on static templates or global feature mappings to represent targets, failing to automatically extract cross-modal invariant key discriminative features (such as highlighting color texture in visible light and emphasizing thermal radiation distribution in thermal infrared), thus limiting recognition accuracy and generalization ability. Security vulnerabilities stem from a lack of interpretability and transparent decision-making processes. Existing end-to-end high-dimensional feature mapping black-box models cannot clearly reveal the ship discriminative regions or structural features the model focuses on in cross-modal scenarios, lacking reliable traceability and failing to meet the reliability requirements of high-risk scenarios such as security. Furthermore, research on prototype learning for multimodal and cross-domain scenarios is relatively scarce, failing to fully utilize the advantages of prototype learning to solve core problems. Summary of the Invention

[0005] Therefore, it is necessary to provide an interpretable cross-modal target re-identification method and system based on prototype learning to address the above-mentioned technical problems.

[0006] An interpretable cross-modal target re-identification method based on prototype learning, the method comprising: Obtain a multimodal image sample set of the target; the multimodal image sample set includes image samples of at least two different modalities, and each image sample is labeled with the target's identity tag; Construct a prototype cross-modal network; the prototype cross-modal network includes a feature extraction module, a prototype management module, and an output module; The feature extraction module performs channel adaptation on image samples in the multimodal image sample set, extracts multi-scale fusion features, and obtains discriminative feature vectors through channel weighting; The prototype management module initializes the initial prototypes corresponding to each identity label, traverses the spatial location of the discriminative feature vector and extracts local features, calculates the similarity between the local features and each initial prototype, selects the local feature with the highest similarity as the optimal matching feature for each initial prototype, merges the initial prototypes with the corresponding optimal matching features to obtain a new prototype, and iteratively updates the prototypes corresponding to each identity label to obtain a dynamic prototype set. The output module generates a prototype activation heatmap of the discriminative feature vector and each prototype in the dynamic prototype set. The activation values ​​of each prototype activation heatmap are aggregated to construct a prototype activation vector. The classification results of the corresponding image samples are output based on the prototype activation vector. The activation values ​​of the prototype activation heatmap are obtained by the similarity between the local features of each spatial location of the discriminative feature vector and the corresponding prototype. Based on the identity labels of the image samples and the classification results, the prototype cross-modal network is trained using a preset loss function to obtain the trained prototype cross-modal network. Cross-modal re-identification is performed using the trained prototype cross-modal network.

[0007] An interpretable cross-modal target re-identification system based on prototype learning, the system comprising: The sample acquisition module is used to acquire a multimodal image sample set of the target; the multimodal image sample set includes image samples of at least two different modalities, and each image sample is labeled with the target's identity tag; A network construction module is used to build a prototype cross-modal network; the prototype cross-modal network includes a feature extraction module, a prototype management module, and an output module. The discriminative feature output module is used to perform channel adaptation on image samples in the multimodal image sample set through the feature extraction module, extract multi-scale fusion features, and obtain a discriminative feature vector by channel weighting; The dynamic prototype set construction module is used to initialize the initial prototypes corresponding to each identity label through the prototype management module, traverse the spatial position of the discriminative feature vector and extract local features, calculate the similarity between the local features and each initial prototype, select the local feature with the highest similarity as the optimal matching feature for each initial prototype, fuse the initial prototypes and the corresponding optimal matching features to obtain new prototypes, and iteratively update the prototypes corresponding to each identity label to obtain the dynamic prototype set. The image classification module is used to generate a prototype activation heatmap of the discriminative feature vector and each prototype in the dynamic prototype set through the output module, aggregate the activation values ​​of each prototype activation heatmap to construct a prototype activation vector, and perform classification based on the prototype activation vector to output the classification result of the corresponding image sample; the activation value of the prototype activation heatmap is obtained by the similarity between the local features of each spatial location of the discriminative feature vector and the corresponding prototype. The network training module is used to train the prototype cross-modal network using a preset loss function based on the identity labels of the image samples and the classification results, so as to obtain the trained prototype cross-modal network. The cross-modal re-identification module is used to perform cross-modal re-identification using the trained prototype cross-modal network.

[0008] The aforementioned interpretable cross-modal target re-identification method and system based on prototype learning unifies the input scale of different modalities through a feature extraction module via channel adaptation, captures fine-grained discriminative information of the target through multi-scale fusion, and strengthens key features through channel weighting to generate discriminative feature vectors with strong modality invariance. This effectively improves the consistency and matching accuracy of cross-modal features. A dynamic prototype set is formed through initial prototype initialization, local feature selection for optimal matching features, and iterative updates, continuously optimizing the prototype's ability to represent the target identity and significantly improving the model's adaptive performance in complex scenarios. The prototype activation heatmap and prototype activation vector generated by the output module transform the model's decision-making process into an intuitive spatial matching visualization result, significantly improving the interpretability and credibility of the model's decisions. This invention can improve the cross-modal recognition accuracy, scene adaptability, and interpretability of target re-identification in complex environments. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating an interpretable cross-modal target re-identification method based on prototype learning in one embodiment. Figure 2 This is a schematic diagram of the prototype cross-modal network in one embodiment; Figure 3 This is a schematic diagram of the feature extraction module in one embodiment; Figure 4 This is a schematic diagram of the structure of the multi-channel adapter submodule in another embodiment; Figure 5 This is a schematic diagram of the structure of a multi-branch extension submodule in one embodiment; Figure 6 This is a schematic diagram of the attention submodule in one embodiment. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0011] In one embodiment, such as Figure 1 As shown, an interpretable cross-modal target re-identification method based on prototype learning is provided, including the following steps: Step 102: Obtain the multimodal image sample set of the target.

[0012] A multimodal image sample set includes image samples of at least two different modalities, and each image sample is labeled with an identity tag for the target. When performing ship target re-identification, the multimodal image sample set can be a collection of ship images containing at least two different modalities (such as visible light, thermal infrared, near infrared, etc.), and the identity tag can be an identifier assigned to each image that uniquely corresponds to a specific ship individual, such as an ID number.

[0013] Step 104: Construct a prototype cross-modal network.

[0014] The Prototype Cross-Modal Network (PCMN) comprises a feature extraction module, a prototype management module, and an output module. PCMN directly learns semantically interpretable category prototypes through multimodal convolutional feature maps, achieving two core functions: robust ship identification under cross-modal conditions and providing a traceable retrieval decision-making process. The overall PCMN process consists of five stages: multimodal feature extraction → intramodal prototype construction → prototype update → cross-modal prototype retrieval → predicted classification label.

[0015] A schematic diagram of the prototype cross-modal network is shown below. Figure 2 As shown. Where "BN" represents batch normalization; in [B, C, H, W], "B" represents batch size, "C" represents the number of channels, "H" represents height, and "W" represents width; "ReLU" represents the activation function; "MaxPool" represents max pooling; "GAP" represents a global average pooling layer; "Flatten" represents flattening; "Linear" represents a fully connected layer; and "L2 Normalize" represents L2 normalization. " indicates the bitwise multiplication operation of the characteristic matrix.

[0016] PCMN's feature extraction layer is built on the standard ResNet50 architecture. Since the dataset used in this invention contains images of three modalities (visible light, thermal infrared, and near-infrared), and the input includes 1-channel and 3-channel representations, a dedicated mechanism is needed to handle this multimodal, multi-channel input. To address this issue, PCMN introduces a multi-channel adaptation module. In each multi-channel adaptation module, different channels are passed to different convolutional layers to obtain an output vector with the same number of channels. This output vector is then passed through batch normalization, ReLU activation, and max-pooling layers before being passed to the first convolutional block. A multi-branch extension module is introduced after the third convolutional block in the feature extraction layer. This module uses parallel convolutional branches with different receptive fields to capture multi-scale contextual information, enhancing the model's ability to model fine-grained local features and global semantic structures. Finally, an attention module is integrated after the multi-branch extension module to reweight the fused features by channels, enhancing the response of the discriminative channels, suppressing noise, and generating the final feature representation for subsequent tasks such as prototype learning and cross-modal retrieval.

[0017] Step 106: The feature extraction module performs channel adaptation on the image samples in the multimodal image sample set, extracts multi-scale fusion features, and obtains discriminative feature vectors through channel weighting.

[0018] The structural diagram of the feature extraction module is shown below. Figure 3 As shown, Figure 3 In this process, the input image is first processed by the multi-channel adaptation submodule to unify the channels of different modal images, and then features are extracted sequentially through multiple ResNet Blocks (Layer 1-Layer 3, etc.). After that, the features are further refined by the multi-branch expansion submodule. At the same time, the attention module generates attention weight vectors. The convolutional feature map and the attention weight vector are weighted to obtain weighted features. The weighted features are then processed by global average pooling (GAP), flattening, linear transformation, and L2 normalization to output a vector for subsequent processing.

[0019] It is understandable that channel adaptation can eliminate the input scale differences between different modalities, multi-scale fusion ensures that features cover key ship discrimination information, channel weighting can strengthen consistent identity features across modalities, and the final generated discriminative feature vector can map ship images of different modalities to a unified feature space, greatly reducing the differences in feature distribution between modalities, providing a high-quality feature foundation for subsequent prototype matching and cross-modal recognition, and avoiding the problem of inaccurate matching caused by excessive feature modality specificity.

[0020] Step 108: Initialize the initial prototypes corresponding to each identity label through the prototype management module, traverse the spatial location of the discriminative feature vector and extract local features, calculate the similarity between the local features and each initial prototype, select the local feature with the highest similarity as the optimal matching feature for each initial prototype, fuse the initial prototypes with the corresponding optimal matching features to obtain new prototypes, and iteratively update the prototypes corresponding to each identity label to obtain a dynamic prototype set.

[0021] Dynamic prototype management overcomes the limitations of rigid representations in traditional static prototypes. The initial prototype provides a basic template for each identity, while the introduction of local features allows the prototype to focus on fine-grained discriminative information about the ship, avoiding the learning of background noise. By selecting the optimal matching features, the prototype update direction is ensured to align with the identity features. Iterative updates of the prototype result in a dynamic prototype set that can adapt to complex and changing scenarios, such as changes in ship attitude and lighting conditions, maintaining accurate representation of the ship's identity and avoiding the accuracy degradation caused by changes in data distribution in static prototypes.

[0022] Step 110: The output module generates prototype activation heatmaps for the discriminative feature vector and each prototype in the dynamic prototype set. The activation values ​​from these heatmaps are aggregated to construct prototype activation vectors. Based on these activation vectors, the classification results for the corresponding image samples are output. The activation values ​​of the prototype activation heatmaps are obtained by comparing the local features at each spatial location of the discriminative feature vector with the similarity to the corresponding prototype.

[0023] Prototype activation heatmap is a visualization image generated with the spatial location of the image as the horizontal and vertical axes and the similarity between local features and the prototype as the heat value. It can intuitively show which areas of the image match the prototype the most. For example, the smokestack area of ​​a ship is darker in color, which means that the area is the key basis for identification.

[0024] The prototype activation heatmap transforms the black-box decision-making of the model into an intuitive spatial location matching relationship. Users can clearly see which local area of ​​the ship the model uses to determine its identity through the prototype activation heatmap, completely solving the technical defect of the untraceable decision-making basis of traditional methods. The prototype activation vector transforms the visualization information of the heatmap into quantifiable matching indicators, providing accurate numerical support for classification output and ensuring the accuracy of identity judgment. This not only guarantees the reliability of the model's decision-making, but also provides clear prediction results for subsequent supervised training of the loss function.

[0025] Step 112: Based on the identity labels and classification results of the image samples, train the prototype cross-modal network using a preset loss function to obtain the trained prototype cross-modal network.

[0026] Step 114: Perform cross-modal re-identification using the trained prototype cross-modal network.

[0027] In the aforementioned interpretable cross-modal target re-identification method based on prototype learning, the feature extraction module unifies the input scale of different modalities through channel adaptation, captures fine-grained discriminative information of the target through multi-scale fusion, and strengthens key features through channel weighting to generate discriminative feature vectors with strong modality invariance. This effectively improves the consistency and matching accuracy of cross-modal features. A dynamic prototype set is formed through initial prototype initialization, local feature selection to select the optimal matching features, and iterative updates, continuously optimizing the prototype's ability to represent the target identity and significantly improving the model's adaptive performance in complex scenarios. The prototype activation heatmap and prototype activation vector generated by the output module transform the model's decision-making process into an intuitive spatial matching visualization result, significantly improving the interpretability and credibility of the model's decisions. This invention can improve the cross-modal recognition accuracy, scene adaptability, and interpretability of target re-identification in complex environments.

[0028] In one embodiment, cross-modal re-identification using a trained prototype cross-modal network includes: acquiring a single-modal query image of the target to be identified, and a gallery set of another modality different from the query image; the modality of the query image and the modality of the gallery set are both modalities contained in the multimodal image sample set; the gallery set includes several gallery images labeled with identity tags; inputting each gallery image in the gallery set into the trained prototype cross-modal network to obtain the gallery prototype activation vector of each gallery image, associating all gallery prototype activation vectors with the label information of the corresponding gallery images to construct a gallery prototype activation vector library; inputting the query image into the trained prototype cross-modal network to obtain the query prototype activation vector of the query image; calculating the matching degree between the query prototype activation vector and all gallery prototype activation vectors in the gallery prototype activation vector library, sorting the gallery images in the gallery set from high to low matching degree, and outputting the sorting result as the cross-modal re-identification result.

[0029] In one embodiment, the feature extraction module includes a multi-channel adaptation submodule, a multi-branch expansion submodule, and an attention submodule. The multi-channel adaptation submodule allocates corresponding convolutional paths according to the number of channels in the input image, and outputs an initial feature vector after channel adaptation for different modal images. The multi-branch expansion submodule performs multi-scale feature extraction on the initial feature vector through multiple parallel convolutional branches with different kernel sizes, and obtains multi-scale fused features after channel compression. The attention submodule assigns channel weights to the multi-scale fused features and generates a discriminative feature vector through weighted operations.

[0030] In this embodiment, as Figure 4The diagram illustrates the structure of a multi-channel adaptation submodule, specifically designed to support dynamic input formats. This module creates a single-channel convolutional layer by obtaining the parameters of the first convolutional layer of the ResNet50 network, with weights taken as the average of the weights of the first ResNet50 convolutional layer. It automatically identifies the number of input channels and places tensors from different channels into the corresponding convolutional layers, achieving unified adaptive feature extraction across multiple channels.

[0031] like Figure 5 The diagram illustrates the structure of a multi-branch extension submodule. This module employs parallel convolutional branches with different receptive fields to capture multi-scale contextual information, enhancing the model's ability to model fine-grained local features and global semantic structures. The module contains three parallel convolutional branches with kernel sizes of 1×1, 3×3, and 5×5, each responsible for extracting features from different receptive fields. The input feature tensor is...

[0032] Where B is the batch size, C is the number of input channels, and H and W are the height and width of the feature map, respectively. The feature vectors are input into three convolutional layers to obtain the outputs of each branch. Then, they are concatenated along the channel dimension, and the concatenated tensor is subjected to a 1×1 convolution to obtain the fused information.

[0033] like Figure 6 The diagram illustrates the structure of an attention submodule. This submodule first performs global average pooling on the input feature vector, reducing dimensionality before increasing it to minimize the number of parameters. Finally, it passes the vector through a sigmoid activation function to obtain the attention weights. This module automatically learns and assigns importance weights to each feature channel, achieving channel-level adaptive recalibration of the feature map.

[0034] In one embodiment, initializing the initial prototype corresponding to each identity tag includes: selecting a preset number of initial cluster centers from the discriminative feature vectors of the same identity tag, and iteratively optimizing the cluster centers using the K-Means clustering algorithm to obtain the initial prototype corresponding to each identity tag.

[0035] In this embodiment, prototype initialization uses K-Means clustering to avoid learning random and irrelevant features. The process begins by randomly selecting three initial points from all sample features of the same category as cluster centers. Then, through iterative optimization, the following two steps are continuously executed: First, the Euclidean distance from each feature vector to each cluster center is calculated, and the feature vector is assigned to the nearest cluster. Then, the mean of all feature vectors in each cluster is recalculated as the new cluster center. This process iterates until the change in cluster centers is less than a threshold or the maximum number of iterations is reached. The three cluster centers that finally converge are the prototypes for that category, representing different feature distribution patterns of that category (such as features from different perspectives of a ship or under different lighting conditions). In the experiments of this invention, the number of prototypes for each category is set to 3. The total number of categories is 100, and the total number of prototypes = total number of categories × number of prototypes, totaling 300. After prototype initialization, the current prototype set is first copied for safe updates.

[0036] To adapt the prototype to the actual distribution of the training data, PCMN employs a dynamic update mechanism, adjusting the prototype's position during training based on the input features and corresponding labels of the current batch. This mechanism comprises two core operations: prototype update and prototype pruning.

[0037] In one embodiment, fusing the initial prototype with the corresponding optimal matching feature to obtain a new prototype includes: numerically fusing the initial prototype with the corresponding optimal matching feature according to a pre-set update rule; the update rule includes L2 normalization of the initial prototype and the optimal matching feature respectively, then calculating the fusion result by weighted average, and finally L2 normalizing the fusion result again to obtain the new prototype.

[0038] During the forward propagation process, first for each spatial location (h, w) and feature vector... Calculate its relationship with all prototypes The cosine similarity is calculated (both features and prototypes need to be L2 normalized to ensure a fair comparison). Then, across all spatial locations of these samples, the feature vector with the highest similarity to each prototype P is identified: For each prototype The feature vector (or feature mean) with the highest activation value in its corresponding category is used for updating, and then normalization is used to ensure stability.

[0039] In one embodiment, the update rule is:

[0040] in, This is the prototype before the update. For optimal matching features, For the updated prototype, This is an L2 normalization operation.

[0041] In one embodiment, the method further includes: counting the activation frequency of each prototype in the current dynamic prototype set at preset intervals; if the activation count of a prototype is lower than a threshold, it is determined to be a redundant prototype and deleted from the dynamic prototype set.

[0042] In this embodiment, an activation counter is maintained for each prototype. The number of times a model's similarity to any spatial feature exceeds a preset threshold τ (τ=0.7 in this experiment) is counted within the current batch or multiple batches. During training, the prototype activation frequency is checked periodically, and inefficient prototypes with activation counts below the preset threshold are pruned to avoid prototype redundancy and maintain model efficiency.

[0043] By alternating between Push and Prune operations, the prototype set in PCMN adaptively focuses on truly discriminative feature regions during training, avoiding prototype shift or redundancy issues. This mechanism improves the robustness and interpretability of cross-modal retrieval in ship re-identification tasks. The resulting similarity matrix is ​​used for prototype updates, and prototype activation heatmap visualization is implemented, providing interpretable evidence for model decisions.

[0044] Based on the activation map, PCMN aggregates prototype activation values ​​to obtain prototype activation vectors. Subsequently, a learnable classification layer is used to calculate category scores. These scores are used both for supervised identity classification training and to support prototype-based cross-modal retrieval.

[0045] In one embodiment, the loss function includes identity loss, cross-modal similarity loss, orthogonality loss, and prototype loss.

[0046] In this embodiment, to learn robust and discriminative cross-modal representations, the present invention employs a composite objective function to optimize the PCMN framework. This function integrates constraints such as identity discrimination, cross-modal consistency, feature orthogonality, and prototype alignment. The total loss function is defined as follows: .

[0047] in, to This is a hyperparameter used to balance the contributions of each loss term. To mitigate identity loss, we ensure that the features of each modality can be correctly classified. To achieve cross-modal similarity loss, it is necessary to ensure that the similarity matrix between samples from different modalities remains consistent. Orthogonality loss (ORT Loss) ensures that the dimensions of the feature vectors are independent of each other, reducing redundancy. Using prototype loss, prototype learning is leveraged to enhance intra-class compactness and inter-class separability, making the prototype similarity of positive samples greater than that of negative samples. This multi-objective optimization strategy enables the model to effectively address challenges in cross-modal retrieval, such as modality gaps and feature misalignment.

[0048] In one embodiment, the image samples in the multimodal image sample set include visible light image samples, thermal infrared image samples, or near-infrared image samples.

[0049] In one embodiment, the model is evaluated using Rank-k recognition rate and mean precision (mAP). First, feature vectors are extracted from all samples in the query set and gallery set, respectively, and a cosine similarity matrix is ​​calculated between them. Each row of this matrix corresponds to the similarity score between a query sample and all samples in the gallery. The similarity scores in each row are sorted in descending order to obtain the matching ranking list for each query sample in the gallery. Based on the ranking results, the Rank-k recognition rate is calculated by statistically analyzing the proportion of queries containing similar samples among the top k most similar gallery samples; while the mean precision (mAP) requires calculating the area under the precision-recall curve for each query sample. Finally, the arithmetic mean of the AP values ​​for all query samples is calculated to comprehensively measure the overall performance of the retrieval system.

[0050] The effectiveness of the method of the present invention is demonstrated through experiments: Experiments were conducted on the multimodal ship re-identification dataset CMShipReID, which includes ship images in three modalities: visible light (VIS), thermal infrared (TIR), and near-infrared (NIR). The original dataset contains approximately 3071 visible light images (covering 138 ship identities), 2326 thermal infrared images (covering 134 ship identities), and 2530 near-infrared images (covering 138 ship identities). Due to inconsistencies in ship identities across different modalities, only ship identities present in all three modalities were used for model training to ensure multimodal alignment. The testing phase primarily evaluated the following two cross-modal retrieval combinations: (1) Visible light → Near infrared (VIS → NIR): Visible light images are used as query samples, and near infrared images are used as image library samples. This combination contains 38 shared ship identities, including 860 query images and 702 image library images.

[0051] (2) Visible Light → Thermal Infrared (VIS → TIR): Visible light images are used as query samples, and thermal infrared images are used as image library samples. This combination contains 34 shared ship identities, including 860 query images and 620 image library images.

[0052] The model was trained using aligned trimodal data and a class-balanced sampling strategy: each batch contained 4 ship identities, with each identity corresponding to 4 images (batch size 16), thereby improving training stability and model generalization ability. The training process consisted of 100 epochs, using the Adam optimizer with an initial learning rate of 1×10⁻⁶. -2 A dynamic learning rate scheduling strategy is adopted: a higher learning rate is used in the early stage of training to achieve rapid convergence, a medium learning rate is maintained in the middle stage to stabilize the learned features, and the learning rate is gradually reduced in the later stage to optimize the model performance.

[0053] The data preprocessing process is as follows: Visible light images: converted to RGB format and normalized using the mean ([0.485, 0.456, 0.406]) and standard deviation ([0.229, 0.224, 0.225]) of the ImageNet dataset.

[0054] Infrared images (thermal infrared / near infrared): converted to single-channel grayscale images and normalized using a mean of 0.5 and a standard deviation of 0.5; some images were enhanced with histogram equalization to improve feature recognition.

[0055] Data augmentation: Random horizontal flipping and random cropping operations are used during the training phase; only center cropping is used during the testing phase. All images are uniformly adjusted to a fixed resolution of 224×224 pixels.

[0056] The experiment used mean precision (mAP) and rank-k (k=1,5,10) as the core evaluation metrics. The comparison results on the CMShipReID dataset are shown in Table 1.

[0057] Table 1 Comparison of Re-identification Accuracy on the CMShipReID Dataset

[0058] As shown in Table 1, the complete PCMN model performs best in the visible and near-infrared cross-modal re-identification task: the mean average accuracy (mAP) reaches 72.30%, and the Rank-1 accuracy reaches 76.80%, indicating that even with imaging differences between visible and near-infrared modes, the model can still achieve effective feature alignment. Furthermore, the model's Rank-5 and Rank-10 accuracies are 90.10% and 93.40%, respectively, further validating its robustness in retrieval performance. In the more challenging visible and thermal infrared tasks, despite interference from thermal noise and temperature fluctuations, the model still maintains good performance: mAP reaches 65.80%, and the Rank-1 accuracy reaches 61.70%, demonstrating its strong adaptability to complex environments. The comparative experimental results highlight the crucial role of the dynamic prototype management mechanism: After removing this mechanism (PCMN without a prototype module), the mAP for visible light and near-infrared tasks was only 62.58%, and the Rank-1 accuracy was 47.47%, while the mAP for visible light and thermal infrared tasks was 57.00%, and the Rank-1 accuracy was 41.53%. In contrast, the complete model improved the mAP by 9.72 and 8.80 percentage points, respectively, and significantly improved the Rank-1 accuracy by 29.33 and 20.17 percentage points, respectively, in both tasks. This demonstrates that the dynamic prototype management mechanism effectively enhances the consistency of cross-modal feature representations and retrieval performance by optimizing prototype alignment.

[0059] This invention proposes a PCMN framework, which innovatively introduces the prototype learning paradigm into multimodal ship re-identification tasks. This framework constructs an interpretable architecture of "feature extraction—prototype learning—similarity retrieval," effectively addressing the limitations of traditional methods in cross-modal alignment, feature discriminability, and model interpretability. The core technological contributions are reflected in three aspects. First, the dynamic prototype management mechanism designed in this invention automatically learns modally invariant ship features through an "update-pruning" strategy, achieving 82.3% mAP in the visible light → near-infrared cross-modal retrieval task, a 12.7% improvement over the existing best method. Second, the proposed multi-branch depth extension enhancement (DEE) module captures hierarchical features of local ship structures (such as lifeboat layout) and global morphology (such as hull curvature) through parallel multi-scale convolution; simultaneously, the channel attention mechanism enhances the response of discriminative regions, improving Rank-1 accuracy by 9.3%. The developed prototype heatmap and Top-K activation bounding box visualization tool achieve, for the first time in the field of ship re-identification, traceable verification of the decision-making process. This study provides a new technical approach for reliable ship identification in complex marine environments, and the proposed framework can also be extended to other multimodal target identification scenarios.

[0060] It should be understood that, although Figure 1The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0061] In one embodiment, an interpretable cross-modal target re-identification system based on prototype learning is provided, comprising: The sample acquisition module is used to acquire a multimodal image sample set of the target; the multimodal image sample set includes image samples of at least two different modalities, and each image sample is labeled with the target's identity tag; The network construction module is used to build prototype cross-modal networks; the prototype cross-modal network includes a feature extraction module, a prototype management module, and an output module. The discriminative feature output module is used to perform channel adaptation on image samples in the multimodal image sample set through the feature extraction module, extract multi-scale fusion features, and obtain discriminative feature vectors through channel weighting; The dynamic prototype set construction module is used to initialize the initial prototypes corresponding to each identity label through the prototype management module, traverse the spatial location of the discriminative feature vector and extract local features, calculate the similarity between the local features and each initial prototype, select the local feature with the highest similarity as the optimal matching feature for each initial prototype, fuse the initial prototypes and the corresponding optimal matching features to obtain new prototypes, and iteratively update the prototypes corresponding to each identity label to obtain the dynamic prototype set. The image classification module generates prototype activation heatmaps of discriminative feature vectors and prototypes in the dynamic prototype set through the output module. It aggregates the activation values ​​of each prototype activation heatmap to construct prototype activation vectors and performs classification based on the prototype activation vectors to output the classification results of the corresponding image samples. The activation values ​​of the prototype activation heatmaps are obtained by the similarity between the local features of each spatial location of the discriminative feature vector and the corresponding prototype. The network training module is used to train the prototype cross-modal network using a preset loss function based on the identity labels and classification results of image samples, so as to obtain the trained prototype cross-modal network. The cross-modal re-identification module is used to perform cross-modal re-identification using a pre-trained prototype cross-modal network.

[0062] Specific limitations regarding the interpretable cross-modal target re-identification system based on prototype learning can be found in the limitations of the interpretable cross-modal target re-identification method based on prototype learning mentioned above, and will not be repeated here. Each module in the aforementioned interpretable cross-modal target re-identification system based on prototype learning can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0063] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0064] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An interpretable cross-modal target re-identification method based on prototype learning, characterized in that, The method comprises: acquiring a multi-modal image sample set of a target; the multi-modal image sample set comprises image samples of at least two different modalities, and each image sample is labeled by an identity label of the target; constructing a prototype cross-modal network; the prototype cross-modal network comprises a feature extraction module, a prototype management module and an output module; channel adaptation is performed on the image samples in the multi-modal image sample set by the feature extraction module, multi-scale fusion features are extracted, and discriminative feature vectors are obtained through channel weighting; the prototype management module initializes initial prototypes corresponding to each identity label, traverses the spatial positions of the discriminative feature vectors and extracts local features, calculates the similarity between the local features and each initial prototype, selects the local feature with the highest similarity as the optimal matching feature for each initial prototype, fuses the initial prototype and the corresponding optimal matching feature to obtain a new prototype, and iteratively updates the prototypes corresponding to each identity label to obtain a dynamic prototype set; the output module generates a prototype activation heat map of each prototype in the dynamic prototype set with respect to the discriminative feature vector, aggregates the activation values of each prototype activation heat map to construct a prototype activation vector, and classifies and outputs the classification result of the corresponding image sample according to the prototype activation vector; the activation value of the prototype activation heat map is obtained by calculating the similarity between the local features of each spatial position of the discriminative feature vector and the corresponding prototype; a pre-set loss function is used to train the prototype cross-modal network according to the identity label of the image sample and the classification result, and a trained prototype cross-modal network is obtained; cross-modal re-identification is performed using the trained prototype cross-modal network.

2. The method of claim 1, wherein, Cross-modal re-identification using the trained prototype cross-modal network comprises: acquiring a single-modal query image of a target to be identified and another modal gallery set different from the query image; the modal of the query image and the modal of the gallery set are modalities included in the multi-modal image sample set; the gallery set comprises a plurality of gallery images labeled by identity labels; each gallery image in the gallery set is input into the trained prototype cross-modal network to obtain a gallery prototype activation vector of each gallery image, the label information of all gallery prototype activation vectors and corresponding gallery images is associated, and a gallery prototype activation vector library is constructed; the query image is input into the trained prototype cross-modal network to obtain a query prototype activation vector of the query image; the matching degrees of the query prototype activation vector and all gallery prototype activation vectors in the gallery prototype activation vector library are calculated, the gallery images in the gallery set are sorted in descending order of the matching degrees, and the sorting result is output as a cross-modal re-identification result.

3. The method of claim 1, wherein, The feature extraction module comprises a multi-channel adaptation submodule, a multi-branch expansion submodule and an attention submodule; the multi-channel adaptation submodule assigns a corresponding convolution path according to the number of channels of the input image, and outputs an initial feature vector after channel adaptation of different modal images; the multi-branch expansion submodule performs multi-scale feature extraction on the initial feature vector through a plurality of parallel convolution branches with different convolution kernel sizes, and obtains multi-scale fusion features after channel compression; The attention sub-module performs channel weight distribution on the multi-scale fusion features, and generates a discriminative feature vector through a weighting operation.

4. The method of claim 1, wherein, The initial prototype corresponding to each identity tag is initialized, including: A preset number of initial cluster centers are selected from the discriminative feature vectors of the same identity tag, and a K-Means clustering algorithm is used to iteratively optimize the cluster centers to obtain the initial prototype corresponding to each identity tag.

5. The method of claim 1, wherein, Fusion of the initial prototype and the corresponding optimal matching feature to obtain a new prototype includes: According to a preset update rule, the initial prototype and the corresponding optimal matching feature are numerically fused; the update rule includes L2 normalization of the initial prototype and the optimal matching feature respectively, then weighted average calculation of the fusion result, and finally L2 normalization of the fusion result again to obtain a new prototype.

6. The method of claim 5, wherein, The update rule is: wherein, is the original prototype before update, is the optimal matching feature, is the original prototype after update, is the L2 normalization operation.

7. The method of claim 1, wherein, The method further includes: The activation frequency of each prototype in the current dynamic prototype set is counted at a preset interval; If the activation frequency of a prototype is lower than a threshold, the prototype is determined as a redundant prototype and is deleted from the dynamic prototype set.

8. The method of claim 1, wherein, The loss function includes identity loss, cross-modal similarity loss, orthogonality loss, and prototype loss.

9. The method of claim 1, wherein, The image samples in the multi-modal image sample set include visible light image samples, thermal infrared image samples, or near-infrared image samples.

10. An interpretable cross-modal target re-identification system based on prototype learning, characterized in that, The system includes: A sample acquisition module is configured to acquire a multi-modal image sample set of a target; the multi-modal image sample set includes at least two different modal image samples, and each image sample is labeled by an identity tag of the target; A network construction module is configured to construct a prototype cross-modal network; the prototype cross-modal network includes a feature extraction module, a prototype management module, and an output module; A discriminative feature output module is configured to perform channel adaptation on the image samples in the multi-modal image sample set by the feature extraction module, extract multi-scale fusion features, and obtain a discriminative feature vector through channel weighting. A dynamic prototype set construction module is configured to initialize initial prototypes corresponding to each identity tag by the prototype management module, traverse the spatial positions of the discriminative feature vectors and extract local features, calculate the similarity between the local features and each initial prototype, select the local feature with the highest similarity as the optimal matching feature for each initial prototype, fuse the initial prototype and the corresponding optimal matching feature to obtain a new prototype, and iteratively update the prototypes corresponding to each identity tag to obtain a dynamic prototype set. An image classification module is configured to generate a prototype activation heat map about the discriminative feature vector and each prototype in the dynamic prototype set by the output module, aggregate the activation values of each prototype activation heat map to construct a prototype activation vector, and output a classification result of the corresponding image sample according to the prototype activation vector; the activation value of the prototype activation heat map is obtained by calculating the similarity between the local feature of each spatial position of the discriminative feature vector and the corresponding prototype. A network training module is configured to train the prototype cross-modal network using a preset loss function according to the identity tag of the image sample and the classification result, and obtain a trained prototype cross-modal network. A cross-modal re-identification module is configured to perform cross-modal re-identification using the trained prototype cross-modal network.

Images