A multi-modal target re-identification method based on feature decoupling and covariance decorrelation

Abstract

The application discloses a multi-modal target re-identification method based on feature decoupling and covariance decorrelation, comprising the following steps: S01, a decoupling-coupling learning framework is constructed, feature decoupling and coupling learning are performed on multiple heterogeneous modal images, and exclusive features of each modal and shared features across the modal are extracted; S02, a multi-scale channel attention gate expert hybrid module is designed, the exclusive features and the shared features are weighted and fused to obtain multi-modal fusion features; S03, a covariance-based decorrelation algorithm is used to perform high-dimensional projection and weighted covariance estimation on the multi-modal fusion features; and S04, target category determination is performed based on the features after fusion and decorrelation, and re-identification is completed. The application can retain the modal information which is crucial for identity recognition while suppressing redundant features, ensure effective balance of related feature interaction, and provide a more adaptive and stable solution for the practical application of the multi-modal target re-identification technology.

Description

Technical Field

[0001] This invention relates to the field of computer vision and pattern recognition technology, specifically to a target re-identification method, and more particularly to a multimodal target re-identification method based on feature decoupling and covariance decorrelation. Background Technology

[0002] Re-identification (ReID) is an important task in computer vision, aiming to identify the same target from images or videos captured by different cameras or at different times. Traditional methods mainly rely on single-modal visible light images (RGB), which have achieved significant results in well-lit scenes with clear backgrounds. However, in complex environments such as low light, strong backlight, nighttime, or inclement weather, the quality of RGB images deteriorates severely, leading to a sharp decline in recognition performance.

[0003] To address this issue, multimodal imaging technology has been introduced into the field of target re-identification. This method integrates image information from multiple modalities, including visible light (RGB), near-infrared (NIR), and thermal infrared (TIR), leveraging the complementarity between different imaging mechanisms to improve the robustness and recognition accuracy of the model under complex conditions. In recent years, multimodal target re-identification (MMReID) methods can be broadly categorized into two types based on different feature extraction and fusion strategies: those based on global features and those based on local features.

[0004] Global feature-based methods can model overall identity information and align between modalities through global descriptors or cross-modal self-attention mechanisms, but they have limitations in handling detailed features and are sensitive to complex situations such as background interference, local occlusion, and target deformation.

[0005] To enhance the model's ability to capture discriminative details, local feature-based methods have been proposed, attempting to introduce component-level or region-level features to improve discriminative power. However, these methods often rely on manual component segmentation or random region extraction, and have not yet effectively addressed the following key issues:

[0006] (1) Significant differences in local feature quality exist between different modes;

[0007] (2) Overemphasizing intermodal differences while ignoring the intrinsic correlation of features;

[0008] (3) Feature redundancy in the process of multimodal information fusion;

[0009] (4) Noise amplification caused by model complexity. These factors together restrict the discriminative power and robustness of multimodal feature representation, and also constitute the core technical problem to be solved by this invention.

[0010] In prediction tasks, feature correlation exhibits a dual nature: on the one hand, feature correlation with semantic information (such as the correspondence between clothing textures in different modalities) can effectively enhance representation learning; on the other hand, excessive correlation between irrelevant features can introduce redundancy and interfere with the model's discriminative ability. Existing methods have not systematically addressed the negative impact of such feature correlation in their feature interaction mechanism design, resulting in insufficient utilization of multimodal information and limited recognition performance.

[0011] Existing technologies, such as CN116824625A, although they supplement another modality image by generating a network, still rely on image patch stitching and attention weighting as their core methods in their fusion module. They do not fundamentally distinguish between modality-specific features and modality-shared features, which makes modality-specific information easy to be ignored during the fusion process.

[0012] In addition, existing technologies use dual attention modules for feature fusion, but their structure is relatively fixed and does not fully consider the feature interaction relationships between different scales and channels.

[0013] Existing technologies do not explicitly constrain feature redundancy after feature fusion, which may lead to a large number of model parameters and overfitting, affecting model efficiency and generalization ability.

[0014] In summary, effectively decoupling modality-specific information from shared information, and suppressing feature redundancy and reducing model complexity during the fusion process, has become a key issue in improving the performance of multimodal target re-identification. Summary of the Invention

[0015] To address the aforementioned technical problems, this invention proposes a multimodal target re-identification method based on feature decoupling and covariance decorrelation. This method suppresses redundant features while retaining modal information crucial for identity recognition, ensuring an effective balance of relevant feature interactions. This provides a more adaptable and stable solution for the practical application of multimodal target re-identification technology. It effectively solves the problem of performance degradation caused by multimodal feature redundancy and distribution shift in traditional methods, and alleviates the deficiency in model discriminative power caused by excessive feature correlation.

[0016] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0017] A multimodal target re-identification method based on feature decoupling and covariance decorrelation includes the following steps:

[0018] S01. Construct a decoupling-coupling learning framework to perform feature decoupling and coupling learning on multiple heterogeneous modal images, and extract the specific features of each modality and the shared features across modalities respectively.

[0019] S02. Design a multi-scale channel attention-gated expert hybrid module to perform weighted fusion of the exclusive features and shared features to obtain enhanced multimodal fusion features;

[0020] S03. A covariance-based decorrelation algorithm is used to perform high-dimensional projection and weighted covariance estimation on the enhanced multimodal fusion features in step S02. By calculating and optimizing the decorrelation loss, feature redundancy introduced by different sensor physical characteristics is suppressed, and the discriminative ability of multimodal feature representation under distribution offset is improved.

[0021] S04. Based on the fused and decorrelated features, determine the target category and complete the re-identification.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0023] This invention, through the synergistic effect of a decoupled-coupled learning framework, a multi-scale channel attention-gated expert hybrid module, and a covariance decorrelation algorithm, not only preserves the unique discriminative information of multimodal images but also achieves adaptive fusion and redundancy suppression of cross-modal features. This effectively alleviates the modal differences and feature redundancy problems in multimodal target re-identification, and improves the model's recognition accuracy and robustness in distributed offset scenarios. The specific synergistic mechanism and synergistic effects are as follows:

[0024] (I) Core Technological Principles of Synergistic Effect

[0025] 1. The feature decoupling and fusion mechanism is more refined, avoiding the loss of modal information. This invention separates and fuses modality-specific features and shared features through a decoupling-coupling learning framework. While retaining the unique discrimination information of each modality due to different imaging mechanisms, it learns a consistent semantic representation across modalities, thus alleviating the problem of representation misalignment caused by modal differences.

[0026] 2. A multi-scale channel attention-gated expert hybrid module enables fine-grained, adaptive feature fusion. This invention employs a multi-scale channel attention-gated expert hybrid module, which extracts features from different modal combinations through multiple expert networks and utilizes a gating mechanism to achieve adaptive weighted fusion of cross-modal features. This structure can more flexibly adapt to feature changes in different modalities and scenarios, enhancing the model's discriminative ability and robustness.

[0027] 3. A covariance-based decorrelation algorithm is designed to effectively suppress feature redundancy. This invention introduces a covariance-based decorrelation algorithm that explicitly suppresses redundant information between features through high-dimensional projection and weighted covariance estimation, thereby improving the compactness of feature representation and discriminative diversity. This method is particularly suitable for multimodal high-dimensional feature scenarios and can significantly reduce the risk of overfitting.

[0028] In summary, the three components form a closed loop of "decoupling to remove interference → fusing strong correlations → removing redundancy to improve quality," which avoids the limitations of a single module (decoupling alone can easily lead to feature expansion, fusing alone can be susceptible to modal interference, and removing redundancy alone can easily result in the loss of discriminative information), achieving a synergistic effect of 1+1>2.

[0029] Furthermore, step S01 specifically includes the following sub-steps:

[0030] S011. In the decoupling learning stage, features are extracted from each single-modal input image to obtain initial single-modal features. Then, the initial single-modal features are processed through the multi-head sliding window attention (MSWA) mechanism to generate corresponding modality-specific features.

[0031] S012, the coupling stage, uses a fusion strategy based on multi-head sliding window attention to dynamically integrate features from multiple modalities and generate bimodal shared features and trimodal shared features.

[0032] Further, step S011 specifically includes:

[0033] For each unimodal input image, a unimodal feature extraction network based on a visual transformer is used. Extract image patch markers and category tags ;

[0034] Mark image blocks Adaptive average pooling is used as the global feature, and then it is compared with the corresponding class label. The concatenated features are used as local features, and then the concatenated features are input into a fully connected layer to obtain the initial single-modal features. ;

[0035] Construct learnable query tags for each modality. And by the initial single-modal features Generate the corresponding key token Value marker ;

[0036] The initial unimodal features are processed using a multi-head sliding window attention mechanism (MSWA), outputting three decoupled unimodal-specific features. .

[0037] The beneficial effects of the above-mentioned further solutions are as follows:

[0038] 1. Multi-head sliding window attention (MSWA) constrains the attention range by using a sliding window, which can accurately focus on local discrimination regions within a single modality (such as clothing texture in RGB mode and contour details in NIR mode), maximizing the preservation of key information unique to each modality due to differences in imaging mechanisms, and avoiding the dilution of modality-specific features by global attention.

[0039] 2. The sliding window restricts self-attention computation to a local neighborhood. Compared with global self-attention, it effectively reduces computational complexity and avoids global redundant computation. It reduces redundant interactions of long-range irrelevant features, reduces memory usage, and improves training efficiency and stability.

[0040] 3. The local neighborhood mechanism can filter out interference from weakly related information such as distant background and noise, making the output single-modal exclusive features more discriminative and providing a better input basis for subsequent cross-modal fusion.

[0041] Further, step S012 specifically includes:

[0042] S0121, Dual-modal feature fusion: fusing the initial single-modal features of any two modalities. Pairing and splicing into dual-modal features ;

[0043] Construct learnable query tags And by dual-modal features Generate key tags Value marker Multi-head sliding window attention MSWA is used to extract three coupled bimodal shared features. ;

[0044] S0122, Trimodal Shared Feature Fusion: Aggregating all three initial single-modal features Obtain aggregated features Use learnable query tags And generate key tags from this aggregation feature. Value marker A coupled trimodal shared feature is extracted using multi-head sliding window attention MSWA. .

[0045] By implementing the above-described bimodal feature fusion and trimodal feature fusion steps, this method further achieves the following beneficial effects:

[0046] 1. A progressive and hierarchical feature representation system was constructed through a progressive fusion of "single-modal → bimodal → trimodal". The bottom layer of this system retains modality specificity, the middle layer captures the complementary relationships between pairs of modalities, and the top layer integrates globally shared semantics.

[0047] 2. The hierarchical fusion structure enables the model to utilize the unique characteristics of each modality and leverage the synergistic effect of multimodal combination to characterize shared information and cross-modal interactions, fully capturing the complex relationships generated by multi-source data fusion.

[0048] Further, step S02 includes the following sub-steps:

[0049] S021. By using the gated expert hybrid GMoE module, the hybrid expert model MoE is dynamically activated according to the input characteristics, and the multiple sets of feature representations obtained in step S01 are weighted and fused.

[0050] S022. The fused features are recalibrated across multiple receptive fields using the multi-scale channel attention (MSCA) module, and the enhanced features are projected back to the original feature space through a lightweight output projection layer.

[0051] Further, step S021 dynamically activates the hybrid expert model MoE based on the input characteristics, and performs weighted fusion on the multiple sets of feature representations obtained in step S01, specifically including the following sub-steps:

[0052] S0211, concatenate multiple sets of feature representations to generate a query vector, and stack them to generate key tags;

[0053] S0212, the query vector and key tags are transformed in dimension by a learnable linear projection matrix and then batch normalized. The projected tensors are then reshaped and split according to the number of attention heads to form multiple attention subspaces for parallel computation. In each subspace, the relevance score between the query and the key is calculated independently to capture feature interaction patterns in different dimensions.

[0054] S0213, based on the partitioned multiple sets of queries and key representations, uses a multi-head cross-attention mechanism to calculate the corresponding gating weights. These gating weights are used to dynamically evaluate and quantify the importance of each hybrid expert network in the corresponding attention head, thereby achieving refined weighting of different expert outputs.

[0055] S0214 transforms attention weights into fusion coefficients for each expert through linear mapping, and then uses these coefficients to weight and aggregate the expert-processed features to generate a fine-grained, conditional fusion feature vector.

[0056] The beneficial effects of the above scheme are as follows:

[0057] 1. The mechanism of attention-based head reshaping and parallel computing can simultaneously capture diverse interaction modes of multimodal features in different subspaces (such as "RGB texture + TIR temperature", "NIR contour + RGB color"), avoiding the one-sided capture of feature interactions by single-space attention and improving the comprehensiveness of feature utilization.

[0058] 2. Attention-gated hybrid expert models can dynamically activate corresponding experts based on the modal attributes of input features, enabling adaptive and fine-grained cross-modal feature fusion, which significantly improves the accuracy of multimodal target re-identification.

[0059] Furthermore, step S022 specifically includes the following sub-steps:

[0060] S0221, set up multiple parallel channel attention branches, each channel attention branch simulates a different receptive field size by using different channel reduction ratios;

[0061] S0222, each channel attention branch independently compresses, nonlinearly transforms and excites the fused feature vector in step S0214 to generate single-scale channel attention weight maps for different scales.

[0062] S0223, these single-scale channel attention weight maps are normalized and weighted to obtain a multi-scale fused channel attention weight map, which is then modulated and multiplied with the fused feature vector in step S0214 to achieve recalibration of the feature channels.

[0063] The beneficial effects of the above scheme are as follows:

[0064] 1. Multi-parallel channel attention branches (with different reduction ratios) can simulate receptive fields of different scales: branches with small reduction ratios focus on local modal details (such as pedestrian accessories), while branches with large reduction ratios capture global cross-modal semantics (such as the overall outline of a pedestrian), enabling the model to simultaneously adapt to the "local fine-grainedness" and "global consistency" requirements of multimodal features, thereby enhancing the scale robustness of features.

[0065] 2. Channel recalibration after multi-scale weight fusion can specifically enhance the information-rich dimensions in multimodal features (such as RGB color channels and TIR temperature channels) while suppressing redundant dimensions (such as repeated background features), making the final output features more compact and discriminative, and meeting the "feature discrimination priority" requirement in multimodal target re-identification.

[0066] Further, step S03 includes the following sub-steps:

[0067] S031. Project the enhanced multimodal fusion features output in step S02 onto a high-dimensional space using random Fourier features to obtain a high-dimensional feature vector.

[0068] S032. Calculate weighted covariance estimation based on importance sampling weights. By modeling the differences in sample importance, the bias of traditional covariance estimation is corrected, and a refined modeling of the statistical characteristics of the feature distribution is achieved.

[0069] S033. Calculate the decorrelation loss of the multimodal combination. By constraining the off-diagonal elements of each covariance matrix, the linear correlation between any two feature dimensions is explicitly reduced, thereby improving the compactness of the feature representation.

[0070] Further, in step S033, the decorrelation loss is:

[0071] ,

[0072] in, It is the first The decorrelation loss for multiple combinations of input features is calculated using the following formula:

[0073] ,

[0074] in, It is the first In the feature combination, the _th The dimension of the random Fourier feature and the... The covariance matrix between random Fourier feature dimensions It is a matrix of all ones. It is the identity matrix. It represents the Hadamah accumulation. yes Regularization.

[0075] Furthermore, in step S04, the parameters of the entire network are optimized by combining label smoothing cross-entropy loss and triplet loss function, and the output after feature fusion is supervised. The overall loss function is represented by the following formula:

[0076] ,

[0077] in, and These are the label smoothing cross-entropy loss and the triplet loss, respectively. Attached Figure Description

[0078] Figure 1 This is a flowchart illustrating the implementation of the present invention;

[0079] Figure 2 This is a detailed structural diagram of a multimodal target re-identification method based on feature decoupling and covariance decorrelation;

[0080] Figure 3 These are the different attention mechanisms that multi-head sliding window attention (MSWA) can employ, where (a) is dense self-attention and (b) is the sliding window attention mechanism;

[0081] Figure 4 The following is a flowchart of the training process for a multimodal target re-identification model based on feature decoupling and covariance decorrelation, provided in an embodiment of the present invention. Detailed Implementation

[0082] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0083] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0084] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0085] To facilitate understanding of the embodiments of the present invention, the following will provide further explanation and description with reference to the accompanying drawings and several specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0086] like Figure 1 As shown, a multimodal target re-identification method based on feature decoupling and covariance decorrelation includes the following steps:

[0087] S01. Construct a decoupled-coupled learning framework, decomposing the input multimodal data of visible light RGB image, near-infrared NIR image and thermal infrared TIR image into two complementary feature subspaces, namely the modality-specific feature subspace and the modality-shared feature subspace. The modality-specific feature subspace is used to retain the modality-inherent information determined by the differences in the physical mechanisms of each modality imaging sensor, and the modality-shared feature subspace is used to learn a general semantic representation across modalities.

[0088] Step S01 includes the following sub-steps:

[0089] S011, For each modal input image ( (Representing one of the RGB, NIR, or TIR modalities), using a Visual Transformer (ViT)-based network as a single-modal feature extraction network. Extract image patch markers and category tags Mark the image patch Pooling is performed as a global feature, and then the feature is matched with the corresponding class label. The concatenated features are used as local features. These concatenated features are then input into a fully connected layer, and the initial unimodal features are obtained using the following formula. :

[0090] ,

[0091] in, Indicates adaptive average pooling. This indicates a splicing operation. It is a regularization layer. It is a fully connected layer. It is the GELU activation function. It constructs learnable query tags for each modality. From the initial single-modal features Generate key tags Value marker :

[0092] ;

[0093] The query tags are processed using a multi-head sliding window attention MSWA mechanism. , key mark Sum value mark The output shows the three single-mode specific features after decoupling, and the calculation process is as follows:

[0094] .

[0095] S012. In the coupling stage, features from multiple modalities are dynamically integrated through an attention-based fusion strategy and a multi-head sliding window attention-based fusion strategy to generate bimodal shared features and trimodal shared features.

[0096] Further, step S012 includes the following sub-steps:

[0097] S0121, Dual-modal feature fusion;

[0098] The initial single-mode features of any two modes Pairing and splicing into dual-modal features :

[0099] ,

[0100] in, , ,(in , (These are different modes in RGB, NIR, and TIR, respectively).

[0101] Construct learnable query tags And by dual-modal features Generate key tags Value marker :

[0102] ;

[0103] Multi-head sliding window attention MSWA is used to extract bimodal shared features of three coupled modes. :

[0104] ;

[0105] S0122, Trimodal Shared Feature Fusion: Aggregating all three initial single-modal features Obtain aggregated features :

[0106] ,

[0107] in, , This indicates a concatenation operation. Learnable query tags are used. And by aggregation features Generate key tags Value marker A coupled trimodal shared feature is extracted using multi-head sliding window attention MSWA. :

[0108] ,

[0109] Query tags By focusing on the discriminative regions shared by the three modalities, cross-attention can be used to more effectively capture consistent and informative features.

[0110] S02. Design a multi-scale channel attention-gated expert hybrid module to adaptively aggregate the modality-specific features and modality-shared features obtained in step S01, and realize adaptive weighted fusion of cross-modality features.

[0111] Step S02 includes the following sub-steps:

[0112] S021. Through the Gated Expert Hybrid (GMoE) module, the hybrid expert model (MoE) is dynamically activated according to the input characteristics to perform weighted fusion of the multiple feature representations obtained in step S01.

[0113] S022, the multi-scale channel attention (MSCA) module, recalibrates the fused features across multiple receptive fields and projects the enhanced features back to the original feature space through a lightweight output projection layer.

[0114] Furthermore, the Gated Expert Hybrid (GMoE) module in step S021 is implemented as follows:

[0115] The input contains 7 different feature representations: three unimodal specific features Three bimodal shared features A trimodal shared feature The seven sets of features are concatenated to generate a query vector. And stack them to generate key tokens. ,in It is a stacked tensor function. For the query vector and key mark Perform projection to generate and :

[0116] ,

[0117] ,

[0118] in, This represents the batch normalization function. and It is a learnable projection matrix.

[0119] Apply head-to-head reshaping to the projected query and key tensors to facilitate multi-head attention computation:

[0120] ,

[0121] ,

[0122] in, and They represent the first The query and key tensor of the size This represents the reshaping operation function. ,common One point of attention.

[0123] Gating weights are calculated using a multi-head cross-attention mechanism, which reflect the importance of each expert in each head:

[0124] ,

[0125] in, It is the dimension of each head. This represents the activation function.

[0126] Gated vectors Used to weight the outputs of each expert in each head. The weight of each expert is obtained through a linear mapping:

[0127] ,

[0128] in, It is a linear mapping function. , It refers to the number of experts.

[0129] To achieve modular specialization and conditional computation, a hybrid expert model (MoE) is employed for each attention head. Specifically, for each attention head... , using by A corresponding expert group consists of parallel expert networks. Each attention head... Receive input features A subset of, for the sake of generality, uses To represent any one of these 7 groups of features. Each Evenly divided into In terms of size:

[0130] ;

[0131] in, This represents a block function. For each header... Sub-feature representation , and As input for the corresponding expert. Head Each expert in Process the assigned mode:

[0132] ;

[0133] The outputs of these experts are passed through a set of attention-based gating weights. To merge:

[0134] ;

[0135] Finally, the fused outputs of all heads are stitched together to reconstruct the full-dimensional output:

[0136] .

[0137] Furthermore, step S022 is implemented as follows:

[0138] Given a fused feature vector Multi-scale channel attention (MSCA) module application There are 3 parallel channel attention branches, each with a different reduction ratio. ,in The number of channels in a single branch is calculated using the following formula. :

[0139] ,

[0140] in, This represents the embedding dimension of the input features. The enhanced expert features are obtained using the following formula:

[0141] ,

[0142] in, and These are learnable convolutional weights. It is the Sigmoid function. Represents the ReLU activation function. It is the natural logarithm An exponential function with base 0. This represents the summation function. It is the first One fusion weight parameter, It represents the Hadamah accumulation. It is a global average pooling operation.

[0143] S03. A covariance-based decorrelation algorithm is used to perform high-dimensional projection and weighted covariance estimation on the feature representation obtained after fusion in step S02. By calculating and optimizing the decorrelation loss, feature redundancy introduced by different sensor physical characteristics is suppressed, and the discriminative ability of multimodal feature representation under distribution offset is improved.

[0144] Step S03 includes the following sub-steps:

[0145] S031. Project the enhanced multimodal fusion feature output from step S02 into a high-dimensional space using random Fourier features.

[0146] S032. Calculate the weighted covariance estimate based on the importance sampling weight.

[0147] S033. Calculate the decorrelation loss of the multimodal combination.

[0148] Furthermore, step S031 uses random Fourier features to project the enhanced multimodal fusion features output in step S02 into a high-dimensional space. The projection process is represented by the following formula:

[0149] ,

[0150] in, Indicates the first The characteristic matrix of the combination, It is a Gaussian random projection matrix. It is a random bias. It is the number of Fourier features. It is a cosine function. It is a sine function. This represents the matrix transpose operation.

[0151] Further, step S032 calculates the weighted covariance estimate using importance sampling weights, and its weighted mean... with weighted covariance Defined as:

[0152] ,

[0153] ,

[0154] in, For importance sampling weights, Indicates from group The The feature vector of each sample Indicates the number of samples. It is a function for creating diagonal matrices.

[0155] Furthermore, the decorrelation loss for the multimodal combination in step S033 is:

[0156] ,

[0157] in, It is the first The decorrelation loss for multiple combinations of input features is calculated using the following formula:

[0158] ,

[0159] in, It is the first In the feature combination, the _th The dimension of the random Fourier feature and the... The covariance matrix between random Fourier feature dimensions It is a matrix of all ones. It is the identity matrix. It represents the Hadamah accumulation. yes Regularization.

[0160] S04. Input the image data of each modality of the target to be identified into the trained single-modal feature extraction network to obtain fused features, and process the fused features through the recognition network to map them into a category probability distribution to complete the target re-identification.

[0161] Furthermore, in step S04, label-smoothed cross-entropy loss is used. and triplet loss The combination of features supervises the fused data to optimize the overall network parameters, and its overall loss function... Characterized by the following formula:

[0162] .

[0163] Experimental data validates the synergistic effect:

[0164] To verify the technical advantages of the collaborative mechanism of this invention, a multimodal pedestrian re-identification dataset RGBNT201 and two multimodal vehicle re-identification datasets RGBNT100 and MSVR310 were selected. Four sets of comparative experiments were designed: three sets of single-module groups (only decoupling framework, only fusion module, only decorrelation algorithm), and one set of collaborative groups of the three components of this invention. The core evaluation metrics were mean accuracy (mAP) and Rank-1 accuracy. The experimental results are shown in the table below:

[0165] Data-driven analysis:

[0166] Based on experimental results from multimodal re-identification scenarios involving pedestrians and vehicles, the synergistic effect of the three-way collaborative mechanism in this invention is significant and universal, specifically reflected in the following three aspects:

[0167] 1. Accuracy is comprehensively improved, and cross-scene adaptability is strong: In the pedestrian dataset RGBNT201, the collaborative group achieves an mAP of 80.6% and a Rank-1 of 84.6%, which is an improvement of 1.5% and 2.4% respectively compared to the best single module group (multi-scale fusion module only, mAP 79.1%, Rank-1 82.2%), and an improvement of 2.9% compared to the worst single module group (decoupling-coupling learning framework only, mAP 77.7%). In the vehicle dataset RGBNT100, the collaborative group achieves an mAP of 88.5% and a Rank-1 of 97.6%, which is an improvement of 1.1% and 0.5% respectively compared to the best single module group (multi-scale fusion module only, mAP 87.4%, Rank-1 97.1%), achieving a breakthrough in high cardinality accuracy. In the vehicle dataset MSVR310, the collaborative group achieves an mAP of 51.3% and a Rank-1 of 68.2%, which is an improvement of 0.5% compared to the best single module group (covariance decorrelation algorithm only, mAP 50.8%). The accuracy of Rank-1 is better than that of other single module groups, which verifies that the collaborative mechanism can enhance feature discrimination in different difficulty and target type scenarios and break through the accuracy bottleneck of single modules.

[0168] 2. Superior performance stability and robustness: Single-module groups exhibit significant performance fluctuations across different datasets (e.g., the multi-scale fusion module alone achieves only 50.3% mAP on the MSVR310 dataset, significantly lower than its 87.4% on RGBNT100), while the collaborative group maintains a leading advantage across all three datasets without significant performance decline. Particularly on the low-precision benchmark MSVR310 dataset, the collaborative group still achieves steady improvement, indicating that the synergy of the three modules effectively mitigates the negative impacts of modal differences and changes in target characteristics, demonstrating superior robustness and wider applicability compared to single modules.

[0169] 3. Significant synergistic effects, achieving 1+1>2: Individual modules each have limitations (the decoupling-coupling learning framework alone has low accuracy on RGBNT201, the multi-scale fusion module alone has insufficient performance on MSVR310, and the covariance decorrelation algorithm alone has the worst accuracy on RGBNT100 Rank-1). However, the synergistic mechanism, through a closed loop of "decoupling and purification - precise fusion - redundancy removal and optimization," integrates the advantages of each module and compensates for the shortcomings of individual modules, achieving balanced and leading performance across all scenarios. Even on the RGBNT100 dataset, which already has a very high accuracy base (the single module's Rank-1 has reached 97.1%), it still achieves a 0.5% Rank-1 improvement, demonstrating that the synergistic mechanism is not simply an additive process, but rather achieves a fundamental doubling of efficiency while avoiding the performance redundancy that may result from the addition of multiple modules, thus balancing accuracy and stability.

[0170] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0171] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0172] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0173] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal target re-identification method based on feature decoupling and covariance decorrelation, characterized in that, Including the following steps: S01. Construct a decoupling-coupling learning framework to perform feature decoupling and coupling learning on multiple heterogeneous modal images, and extract the specific features of each modality and the shared features across modalities respectively. S02. Design a multi-scale channel attention-gated expert hybrid module to perform weighted fusion of the exclusive features and shared features to obtain enhanced multimodal fusion features; S03. A covariance-based decorrelation algorithm is used to perform high-dimensional projection and weighted covariance estimation on the enhanced multimodal fusion features in step S02. By calculating and optimizing the decorrelation loss, feature redundancy introduced by different sensor physical characteristics is suppressed, and the discriminative ability of multimodal feature representation under distribution offset is improved. S04. Based on the fused and decorrelated features, determine the target category and complete the re-identification.

2. The multimodal target re-identification method based on feature decoupling and covariance decorrelation according to claim 1, characterized in that, Step S01 specifically includes the following sub-steps: S011. In the decoupling learning stage, features are extracted from each single-modal input image to obtain initial single-modal features. Then, the initial single-modal features are processed through the multi-head sliding window attention (MSWA) mechanism to generate corresponding modality-specific features. S012, the coupling stage, uses a fusion strategy based on multi-head sliding window attention to dynamically integrate features from multiple modalities and generate bimodal shared features and trimodal shared features.

3. The multi-modal target re-identification method based on feature decoupling and covariance de-correlation according to claim 2, characterized in that, Step S011 specifically includes: using a visual transformer based single modality feature extraction network for each single modality input image, extracting image patch labels and class labels ; Labeling image patches Adaptive average pooling is performed as a global feature, and it is combined with the corresponding class label Concatenation is performed as a local feature, and the concatenated feature is input into a fully connected layer to obtain an initial single-modal feature ; constructing learnable query tokens for each modality and by the initial single modality features as corresponding key tokens value tokens ; The initial single-modal features are processed by a multi-head sliding window attention (MSWA) mechanism to output three single-modal specific features after decoupling .

4. The multi-modal target re-identification method based on feature decoupling and covariance de-correlation according to claim 2, characterized in that, Step S012 specifically includes: S0121, Dual-modal feature fusion: fusing the initial single-modal features of any two modalities. Pairing and splicing into dual-modal features ; Construct learnable query tags And by dual-modal features Generate key tags Value marker Multi-head sliding window attention MSWA is used to extract three coupled bimodal shared features. ; S0122, Trimodal Shared Feature Fusion: Aggregating all three initial single-modal features Obtain aggregated features Use learnable query tags And generate key tags from this aggregation feature. Value marker A coupled trimodal shared feature is extracted using multi-head sliding window attention MSWA. .

5. The multi-modal target re-identification method based on feature decoupling and covariance de-correlation according to claim 1, characterized in that, Step S02 includes the following sub-steps: S021. By using the gated expert hybrid GMoE module, the hybrid expert model MoE is dynamically activated according to the input characteristics, and the multiple sets of feature representations obtained in step S01 are weighted and fused. S022. The fused features are recalibrated across multiple receptive fields using the multi-scale channel attention (MSCA) module, and the enhanced features are projected back to the original feature space through a lightweight output projection layer.

6. The multi-modal target re-identification method based on feature decoupling and covariance de-correlation according to claim 5, characterized in that, Step S021 specifically includes the following sub-steps: S0211, concatenate multiple sets of feature representations to generate a query vector, and stack them to generate key tags; S0212, the query vector and key tags are transformed in dimension by a learnable linear projection matrix and then batch normalized; then the projected tensors are reshaped and split according to the number of attention heads to form multiple attention subspaces for parallel computation. Within each subspace, the relevance score between the query and the key is calculated independently to capture feature interaction patterns across different dimensions; S0213, based on the partitioned multiple sets of queries and key representations, uses a multi-head cross-attention mechanism to calculate the corresponding gating weights. These gating weights are used to dynamically evaluate and quantify the importance of each hybrid expert network in the corresponding attention head, thereby achieving refined weighting of different expert outputs. S0214 transforms attention weights into fusion coefficients for each expert through linear mapping, and then uses these coefficients to weight and aggregate the expert-processed features to generate a fine-grained, conditional fusion feature vector.

7. The multi-modal target re-identification method based on feature decoupling and covariance de-correlation according to claim 5, characterized in that, Step S022 specifically includes the following sub-steps: S0221, set up multiple parallel channel attention branches, each channel attention branch simulates a different receptive field size by using different channel reduction ratios; S0222, each channel attention branch independently compresses, nonlinearly transforms and excites the fused feature vector in step S0214 to generate single-scale channel attention weight maps for different scales. S0223, these single-scale channel attention weight maps are normalized and weighted to obtain a multi-scale fused channel attention weight map, which is then modulated and multiplied with the fused feature vector in step S0214 to achieve recalibration of the feature channels.

8. The multi-modal target re-identification method based on feature decoupling and covariance de-correlation according to claim 1, characterized in that, Step S03 includes the following sub-steps: S031. Project the enhanced multimodal fusion features output in step S02 onto a high-dimensional space using random Fourier features to obtain a high-dimensional feature vector. S032. Calculate weighted covariance estimation based on importance sampling weights. By modeling the differences in sample importance, the bias of traditional covariance estimation is corrected, and a refined modeling of the statistical characteristics of the feature distribution is achieved. S033. Calculate the decorrelation loss of the multimodal combination. By constraining the off-diagonal elements of each covariance matrix, the linear correlation between any two feature dimensions is explicitly reduced. Improve the compactness of feature representation.

9. The multi-modal target re-identification method based on feature decoupling and covariance de-correlation according to claim 8, characterized in that, In step S033, the decorrelation loss is: ， wherein, is the decorrelation loss for the mth combination of input features, calculated by: ​ ， in, It is the first In the feature combination, the _th The dimension of the random Fourier feature and the... The covariance matrix between random Fourier feature dimensions It is a matrix of all ones. It is the identity matrix. It represents the Hadamah accumulation. yes Regularization.

10. The multimodal target re-identification method based on feature decoupling and covariance decorrelation according to claim 1, characterized in that, In step S04, the parameters of the entire network are optimized by combining label smoothing cross-entropy loss and triplet loss function, and the output after feature fusion is supervised. The overall loss function is represented by the following formula: ， where, and are the label smoothed cross-entropy loss and the triplet loss, respectively.

Images