End-to-end cross-modal pedestrian re-identification method based on multi-domain feature alignment

Abstract

The invention relates to the technical field of pedestrian re-identification, and particularly provides an end-to-end cross-modal pedestrian re-identification method based on multi-domain feature alignment. The method comprises the following steps: splitting original text description into two sub-descriptions of identity description and clothing description through a text description separation module, and providing structured semantic input for cross-modal feature alignment; according to the two sub-descriptions of the identity and the clothing, the identity-clothing bidirectional decoupling alignment module utilizes an attention mechanism and a gating weighting strategy to realize cross-modal feature alignment; based on cross-modal feature alignment, introducing a Mama state space model SSM into a cross-modal pedestrian re-identification ReID task, and fusing image and text features; according to the features of the fused image and the text, a multi-target robust optimization module is designed for optimization, and a final retrieval result is output, the precision of fine-grained semantic alignment is improved, effective context collaboration is achieved, and balance between discrimination and robustness is achieved.

Description

Technical Field

[0001] This invention relates to the field of pedestrian re-identification technology, and in particular to an end-to-end cross-modal pedestrian re-identification method based on multi-domain feature alignment. Background Technology

[0002] Text-based person re-identification (TPRe-ID) aims to retrieve images of target pedestrians that match natural language descriptions across a multi-camera surveillance network. Compared to traditional image retrieval tasks, TPRe-ID faces a more severe challenge in cross-modal alignment. Due to the semantic gap between images and text, especially in complex scenes, factors such as background interference, perspective changes, and clothing alterations significantly weaken semantic consistency, making it difficult to accurately align fine-grained identity information. This severely limits the model's retrieval accuracy and generalization ability.

[0003] To address these issues, current mainstream methods primarily rely on global feature matching or visual-language attention mechanisms to bridge modal differences. However, these methods often suffer from fundamental flaws. First, natural language descriptions frequently contain a mixture of dynamic attributes (such as clothing) and static attributes (such as gender and body type), while existing models lack explicit semantic separation capabilities, leading to a sharp decline in recognition performance when handling complex scenarios such as occlusion and clothing changes. Second, typical cross-modal interaction structures (such as Transformers) suffer from high computational costs and difficulty in capturing fine-grained semantic alignment relationships when modeling long sequences of visual and linguistic dependencies, further impacting the efficiency and accuracy of cross-modal alignment. Furthermore, most methods focus only on identity consistency in their supervision strategies, neglecting the constraint of clothing distinctiveness, resulting in poor robustness to semantically ambiguous or incomplete descriptions.

[0004] In recent years, with the development of deep learning, especially the widespread application of Visual Transformer (ViT) in image recognition and ReID, TPRe-ID technology has also made rapid progress. Although existing research has achieved certain results in feature modeling and modal fusion, the core challenge remains unresolved: the difficulty of fine-grained semantic alignment still severely restricts the performance of models in real-world scenarios. When dealing with samples with changes in clothing, drastic viewpoint shifts, or heavy occlusion, models are prone to confusing dynamic attributes with key identity features, leading to a significant decrease in matching accuracy. Some works have attempted to introduce attribute decoupling networks or adversarial training mechanisms to alleviate this problem; however, existing methods generally suffer from incomplete decoupling and unclear semantic separation, and are also inefficient in the multimodal feature fusion stage, making it difficult to achieve effective contextual collaboration.

[0005] Overall, although attribute decoupling and adversarial training have shown initial success, they still have significant limitations in three aspects: First, existing methods fail to effectively distinguish between dynamic and static attributes in text, leading to semantic confusion during cross-modal alignment; second, the decoupled multimodal features lack efficient context fusion strategies, resulting in significant biases in local semantic matching; and finally, the lack of collaborative constraints on the cross-modal association between identity and clothing during training makes it difficult to achieve a balance between discriminability and robustness. Summary of the Invention

[0006] In view of this, the present invention provides an end-to-end cross-modal pedestrian re-identification method based on multi-domain feature alignment, which improves the accuracy of fine-grained semantic alignment, achieves effective contextual collaboration, and achieves a balance between discriminability and robustness.

[0007] In a first aspect, the present invention provides an end-to-end cross-modal person re-identification method based on multi-domain feature alignment, the method comprising:

[0008] Step 1: Using the Text Description Separation Module (TDSM), the original text description is split into two sub-descriptions: identity description and clothing description, providing structured semantic input for cross-modal feature alignment;

[0009] Step 2: Based on the two sub-descriptions of identity and clothing, the identity-clothing bidirectional decoupling alignment module BDAM uses an attention mechanism and a gating weighting strategy to achieve cross-modal feature alignment;

[0010] Step 3: Based on cross-modal feature alignment, the Mamba State Space Model (SSM) is introduced into the cross-modal person re-identification (ReID) task, fusing image and text features;

[0011] Step 4: Based on the features of the fused image and text, design a multi-objective robust optimization module for optimization and output the final retrieval results.

[0012] Optionally, step 1 includes:

[0013] By using the adaptive semantic classification and post-processing mechanism of the Text Description Separation Module (TDSM), combined with syntactic analysis and semantic embedding, the input text is decomposed into clothing description and identity description, and independent data structures are constructed. For semantically ambiguous or missing descriptions, TDSM adopts a dynamic threshold strategy and supplementation mechanism to ensure the integrity and consistency of the input.

[0014] Optionally, step 2 includes:

[0015] The image encoder's sequence features are processed through a dual-branch structure to extract identity features and clothing features respectively. Self-attention and cross-attention mechanisms are combined to capture semantic associations and feature interactions. A dynamic gating weighting strategy optimizes feature weighting to generate highly discriminative identity and clothing features. Text descriptions are embedded by the encoder and aligned with image features.

[0016] Optionally, step 3 includes:

[0017] First, a feature alignment network is used to unify the feature representation of images and text, and a pre-gating mechanism is used to generate modal weights. Then, the Mamba state-space model (SSM) is used for sequence modeling, and finally the features of the fused image and text are output.

[0018] Optionally, step 4 includes:

[0019] The optimization process balances the objectives of each task through dynamic weights, and collaboratively optimizes identity alignment, clothing robustness, and cross-modal association through a multi-task loss function; it also imposes constraints on decoupled features, fused features, and classification results, and outputs the final retrieval results.

[0020] First, through InfoNCE loss L InfoNCE First, image embeddings and identity text embeddings are compared and aligned to optimize cross-modal semantic consistency; second, biometric contrast loss L... Bio Loss L compared to clothing Cloth Identity features and clothing features are aligned separately through projection layers to enhance the discriminative power of decoupled features; clothing adversarial loss L ClothAdv Minimize the similarity between clothing features and unmatched clothing text embeddings to enhance the discriminative power of clothing features; clothing matching loss L ClothMatch Further enhance the fine-grained association between clothing image embedding and clothing text embedding; and combine it with the decoupling loss L based on the Hilbert-Schmidt independence criterion (HSIC). Decouple To reduce the correlation between identity features and clothing features, text-image feature projection is guided to an approximately orthogonal subspace to promote feature separation; gated regularization loss L Gate Gating weight W based on mean square error constraint id and W clo To balance the contributions of the branches; the total loss function L is obtained by weighted summation of the above multi-task loss functions. Total It collaboratively optimizes identity alignment, clothing robustness, and cross-modal association, outputting the final retrieval results. The total loss function is L. Total The expression is:

[0021] L Total =W InfoNCE L InfoNCE +W Bio L Bio+W Cloth L Cloth +W ClothAdv L ClothAdv +W Decouple L Decouple +W Gate L Gate ;

[0022] Wherein, the loss weight W InfoNCE =1.0, W Bio =0.5, W Cloth =0.5, W ClothAdv =0.1, W Decouple =1.0, W Gate =0.01.

[0023] Optionally, in the semantic space, L InfoNCE The loss aligns the image embedding with the identity text embedding, L InfoNCE The loss is based on contrastive learning, calculating the similarity matrix between image and identity text features, maximizing the similarity of positive sample pairs, and minimizing the similarity of negative sample pairs, L InfoNCE The expression for loss is:

[0024]

[0025] in, and These represent layer-normalized image embedding and identity text embedding, respectively; sim(·,·) is the dot product similarity; τ is the temperature parameter; and N is the batch size.

[0026] Optionally, biometric contrast loss L Bio It is the embedding of identity features extracted from aligned images with identity text, L Bio The loss function uses identity features, dimensionality reduced by a linear projection layer, and then normalized. A similarity matrix is ​​then calculated between this matrix and the identity text embedding. Bio The expression for loss is:

[0027]

[0028] in, Identity features that are projected and normalized layer by layer; For identity text embedding; sim(·,·) is the dot product similarity; τ is the temperature parameter; N is the batch size;

[0029] Clothing contrast loss L Cloth L is used to align clothing features extracted from images with corresponding clothing text embeddings. Cloth The expression for loss is:

[0030]

[0031] in, For the projection and layered normalization of clothing features; Embed text for clothing.

[0032] Optionally, clothing counteracts loss L ClothAdv The aim is to minimize the similarity between clothing features and unmatched clothing text embeddings, L ClothAdv The expression for loss is:

[0033]

[0034] in, The clothing features are normalized to a layered structure; For layer-normalized clothing text embedding; sim(·,·) is the dot product similarity; τ is the temperature parameter; N is the batch size;

[0035] Clothing matching loss L ClothMatch Further enhance the fine-grained alignment of clothing image embedding and clothing text embedding, L ClothMatch The loss is optimized through bidirectional contrast to ensure semantic consistency in detail between clothing image features and clothing text descriptions. ClothMatch The expression for loss is:

[0036]

[0037] in, parameter Embedding in clothing images.

[0038] Optionally, the decoupling loss L based on HSIC Decouple The aim is to minimize the extracted identity features f id and clothing features f cloth Semantic relevance between them; L Decouple The loss design is based on the core idea of ​​HSIC, which measures the independence of two random variables using a kernel method. HSIC quantifies the correlation between two feature sets by calculating the dot product of their kernel matrices; the smaller the value, the more independent the features. The decoupling loss L... Decouple Utilizing the nonlinear correlation measure of HSIC, its core objective is to focus identity features on identity discrimination information and clothing features on dynamic appearance attributes, thereby enhancing robustness and generalization ability; L Decouple The expression for loss is:

[0039]

[0040] in, and The identity and clothing features are projected and normalized; N is the batch size.

[0041] Optionally, the gating regularization loss L Gate The mean squared error (MSE) is used to constrain the distribution of gating weights to avoid bias towards 0 or 1, thus ensuring the co-optimization of the two branches; and low weights are used to ensure L Gate Loss as a regularization term; L Gate To balance the contributions of identity and clothing branches, a gating weight close to 0.5 is encouraged; L Gate The expression for loss is:

[0042]

[0043] Where M is the total number of elements in the gate vector, used for layer normalization of mean square error; g i The gating weights are obtained by passing g through the sigmoid activation function of the last linear layer. i , of which 0 <g i <1.

[0044] The technical solution provided by this invention includes a method that splits the original text description into two sub-descriptions—identity description and clothing description—using a text description separation module (TDSM) to provide structured semantic input for cross-modal feature alignment. Based on the two sub-descriptions, an identity-clothing bidirectional decoupling alignment module (BDAM) utilizes an attention mechanism and a gating weighting strategy to achieve cross-modal feature alignment. Based on this cross-modal feature alignment, the Mamba state-space model (SSM) is introduced into the cross-modal person re-identification (ReID) task, fusing image and text features. Based on the features of the fused image and text, a multi-objective robust optimization module is designed for optimization, outputting the final retrieval result. This method improves the accuracy of fine-grained semantic alignment, achieves effective contextual collaboration, and balances discriminability and robustness. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A flowchart of an end-to-end cross-modal pedestrian re-identification method based on multi-domain feature alignment provided in an embodiment of the present invention;

[0047] Figure 2 A flowchart of another end-to-end cross-modal pedestrian re-identification method based on multi-domain feature alignment provided in an embodiment of the present invention;

[0048] Figure 3This is a schematic diagram of feature fusion provided in an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0051] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention are also intended to include the plural forms unless the context clearly indicates otherwise.

[0052] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0053] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0054] This invention provides an end-to-end cross-modal person re-identification method based on multi-domain feature alignment, such as... Figures 1 to 3 As shown, the method includes:

[0055] Step 1: Using the Text Description Separation Module (TDSM), the original text description is split into two sub-descriptions: identity description and clothing description, providing structured semantic input for cross-modal feature alignment.

[0056] In this embodiment of the invention, step 1 includes:

[0057] The Text Description Separation Module (TDSM) aims to accurately separate clothing and identity-related information from mixed natural language descriptions, providing structured semantic input for cross-modal feature alignment. Through TDSM's adaptive semantic classification and post-processing mechanism, combined with syntactic analysis and semantic embedding, the input text is decomposed into clothing descriptions (e.g., purple-long sleeved dress) and identity descriptions (e.g., woman is heavy set), providing input for subsequent cross-modal feature extraction; and independent data structures are constructed. For semantically ambiguous or missing descriptions, TDSM employs a dynamic thresholding strategy and supplementation mechanism to ensure the completeness and consistency of the input.

[0058] Step 2: Based on the two sub-descriptions of identity and clothing, the identity-clothing bidirectional decoupling alignment module BDAM uses an attention mechanism and a gating weighting strategy to achieve cross-modal feature alignment.

[0059] In this embodiment of the invention, step 2 includes:

[0060] The Identity-Clothing Bidirectional Decoupling Alignment Module (BDAM) separates identity features from clothing features in an image and uses the separated text descriptions to achieve cross-modal alignment. It processes the sequence features of the image encoder through a dual-branch structure, extracting identity and clothing features separately. Self-attention and cross-attention mechanisms are combined to capture semantic associations and feature interactions. A dynamic gating weighting strategy optimizes feature weighting, generating highly discriminative identity and clothing features. The text description is embedded through the encoder and aligned with the image features, thereby enhancing the model's robustness and discrimination accuracy under clothing changes, lighting variations, and complex backgrounds.

[0061] Step 3: Based on cross-modal feature alignment, the Mamba State Space Model (MambaSSM) is introduced into the cross-modal person re-identification (ReID) task, fusing image and text features.

[0062] In this embodiment of the invention, step 3 includes:

[0063] The State-Space Model (SSM) fuses image and text features to efficiently capture long-distance semantic dependencies across modalities. First, a feature alignment network unifies image and text feature representations, and a pre-gating mechanism generates modal weights. Then, the Mamba State-Space Model (SSM) is used for sequence modeling, ultimately outputting features that fuse image and text. This design significantly improves matching accuracy in complex scenes. Compared to the high computational cost of traditional Transformers, the State-Space Model (SSM) offers advantages in linear complexity and, with its efficient dynamic context awareness, accurately captures long-distance dependencies across modalities, improving the modeling accuracy for fine-grained semantics (such as special accessories like striped scarves), and significantly enhancing matching performance in complex scenes.

[0064] Step 4: Based on the features of the fused image and text, design a multi-objective robust optimization module for optimization and output the final retrieval results.

[0065] In this embodiment of the invention, step 4 includes:

[0066] The optimization process balances the objectives of each task through dynamic weights to ensure that the model achieves accurate retrieval and robust performance in diverse scenarios. It also optimizes identity alignment, clothing robustness, and cross-modal association through multi-task loss functions. Constraints are imposed on decoupled features, fused features, and classification results to simultaneously improve the model's adaptability to fuzzy descriptions and complex scenarios, and output the final retrieval results.

[0067] First, the loss is estimated through Information Noise-Contrastive Estimation (InfoNCE). InfoNCE First, image embeddings and identity text embeddings are compared and aligned to optimize cross-modal semantic consistency; second, biometric contrast loss L... Bio Loss L compared to clothing Cloth Identity features and clothing features are aligned separately through projection layers to enhance the discriminative power of decoupled features; clothing adversarial loss L ClothAdv Minimize the similarity between clothing features and unmatched clothing text embeddings to enhance the discriminative power of clothing features; clothing matching loss L ClothMatch Further enhance the fine-grained association between clothing image embedding and clothing text embedding; and combine it with the decoupling loss L based on the Hilbert-Schmidt independence criterion (HSIC). Decouple To reduce the correlation between identity features and clothing features, text-image feature projection is guided to an approximately orthogonal subspace to promote feature separation; gated regularization loss L Gate Gating weight W based on mean square error constraint id and W clo To balance the contributions of the branches; the total loss function L is obtained by weighted summation of the above multi-task loss functions. TotalIt collaboratively optimizes identity alignment, clothing robustness, and cross-modal association, outputting the final retrieval results. The total loss function is L. Total The expression is: L Total =W InfoNCE L InfoNCE +W Bio L Bio +W Cloth L Cloth +W ClothAdv L ClothAdv +W Decouple L Decouple +W Gate L Gate ;

[0068] Wherein, the loss weight W InfoNCE =1.0, W Bio =0.5, W Cloth =0.5, W ClothAdv =0.1, W Decouple =1.0, W Gate =0.01. The above weight settings are all based on experimental verification to ensure that the contribution of each loss is balanced in the optimization process.

[0069] In this embodiment of the invention, L InfoNCE The loss function aims to align image embeddings with identity text embeddings, promoting cross-modal semantic consistency. In the semantic space, L... InfoNCE The loss function aligns the image embedding with the identity text embedding, providing an accurate matching foundation for subsequent cross-modal retrieval and ensuring that the model can understand the correspondence between pedestrian identities across different modalities. InfoNCE The loss is based on contrastive learning, calculating the similarity matrix between image and identity text features, maximizing the similarity of positive sample pairs, and minimizing the similarity of negative sample pairs, L InfoNCE The expression for loss is:

[0070]

[0071] in, and These represent layer-normalized image embedding and identity text embedding, respectively; sim(·,·) is the dot product similarity; τ is the temperature parameter; and N is the batch size.

[0072] In this embodiment of the invention, L InfoNCE The loss function is optimized through bidirectional contrast to ensure that image and text features are aligned in the semantic space, thus enhancing cross-modal matching accuracy. By comprehensively considering the semantic distributions of both modalities, the influence of single-modal bias is avoided. Furthermore, the smoothness of the similarity distribution is controlled by the temperature parameter τ, improving the discriminative power of the features.

[0073] In this embodiment of the invention, the biometric contrast loss L Bio This involves aligning extracted identity features from images with embedded identity text to enhance the consistency of identity information. In the TPRe-ID task, identity features are the core basis for retrieval, but they are easily affected by changes in clothing. Therefore, the model specifically optimizes the matching of identity features with text descriptions to reduce the influence of non-identity attributes such as clothing, thereby improving the model's sensitivity to identity-related information. Bio The loss function uses identity features, dimensionality reduced by a linear projection layer, and then normalized. A similarity matrix is ​​then calculated between this matrix and the identity text embedding. Bio The expression for loss is:

[0074]

[0075] in, Identity features that are projected and normalized layer by layer; For identity text embedding; sim(·,·) is the dot product similarity; τ is the temperature parameter; N is the batch size;

[0076] In this embodiment of the invention, L Bio The loss mechanism enhances the semantic consistency of identity features through contrastive learning, reducing the negative impact of clothing interference on identity recognition. Compared to L... InfoNCE Loss, L Bio The loss function is more focused on fine-grained alignment with identity features. Furthermore, the purpose of this loss function is to further enhance the independence and robustness of identity features within the framework of disentanglement.

[0077] Clothing contrast loss L Cloth This is used to align clothing features extracted from images with their corresponding clothing text embeddings, ensuring the model accurately captures potential clothing-related information. However, clothing features can change due to viewpoint, occlusion, or clothing changes, which increases the difficulty and accuracy of feature alignment. Therefore, L is designed... Cloth The loss function optimizes the semantic consistency of clothing features through contrastive learning, enhancing the model's ability to model dynamic attributes and thus maintaining retrieval accuracy even in clothing change scenarios. Cloth The expression for loss is:

[0078]

[0079] in, For the projection and layered normalization of clothing features; Embed text for clothing.

[0080] In this embodiment of the invention, since clothing features and identity features are often highly coupled in images, the model is designed with L... ClothThe loss function uses a dedicated projection layer to ensure that clothing features are optimized in an independent space, reducing confounding with identity features. This loss is designed to support the deentanglement module, strengthening the matching between clothing features and textual descriptions through contrastive learning, and enhancing the model's ability to model dynamic attributes.

[0081] In this embodiment of the invention, clothing resistance loss L ClothAdv The aim is to enhance the discriminative power of clothing features by minimizing the similarity between clothing features and unmatched clothing text embeddings. In the TPRe-ID task, clothing changes (such as changing clothes) may cause the model to incorrectly identify similar clothing worn by people with different identities as the same person. Therefore, L is designed... ClothAdv The loss is used for adversarial optimization to encourage the model to learn more discriminative clothing features, reducing the risk of mismatches, especially in scenarios where clothing is similar but identities are different. ClothAdv The expression for loss is:

[0082]

[0083] in, The clothing features are normalized to a layered structure; is the layer-normalized clothing text embedding; sim(·,·) is the dot product similarity; τ is the temperature parameter; N is the batch size; where the subscripts j and k represent different text vectors extracted by the encoder.

[0084] In this embodiment of the invention, the adversarial loss is inspired by the idea of ​​Generative Adversarial Networks (GANs), which enhances the uniqueness of clothing features by maximizing the prediction probability of negative samples (i.e., minimizing their similarity). The adversarial weights are dynamically adjusted (increasing with the number of training rounds, with a maximum of 1.0) to ensure that early training focuses on basic alignment, while later training gradually strengthens discriminative power.

[0085] Clothing matching loss L ClothMatch Further enhance the fine-grained alignment of clothing image embedding and clothing text embedding, similar to L InfoNCE While it suffers from loss, it focuses more on matching clothing features. In the TPRe-ID task, clothing descriptions may be key clues for retrieval (e.g., wearing a floral hat), L ClothMatch The loss is optimized through bidirectional contrast to ensure semantic consistency in detail between clothing image features and clothing text descriptions, thereby improving retrieval accuracy. ClothMatch The expression for loss is:

[0086]

[0087] in, parameter Embed for clothing images.

[0088] In this embodiment of the invention, with L Cloth The losses are different, L ClothMatch The loss function directly applies to the clothing image embedding and may originate from cropped regions or untangled features, emphasizing fine-grained matching. This loss is optimized through bidirectional contrast to enhance the semantic consistency of clothing features. The bidirectional contrast design ensures the model's sensitivity to clothing details (such as patterns and colors), making it particularly suitable for handling complex clothing descriptions in images. ClothMatch The introduction of loss enhances the model's ability to model clothing attributes and compensates for L... Cloth The loss is insufficient for fine-grained matching.

[0089] In this embodiment of the invention, the decoupling loss L based on HSIC Decouple The aim is to minimize the extracted identity features f id and clothing features f cloth Semantic relevance between them; L Decouple The loss design is based on the core idea of ​​HSIC, which measures the independence of two random variables using a kernel method. HSIC quantifies the correlation between two feature sets by calculating the dot product of their kernel matrices; the smaller the value, the more independent the features. The decoupling loss L... Decouple By utilizing the nonlinear correlation measure of HSIC, its core objective is to focus identity features on identity-discriminating information (such as body shape and gender) and clothing features on dynamic appearance attributes (such as clothing color and style), thereby enhancing the model's robustness and generalization ability in scenarios involving clothing changes or occlusion; L Decouple The expression for loss is:

[0090]

[0091] in, and The identity and clothing features are projected and normalized; N is the batch size.

[0092] In this embodiment of the invention, L Decouple The design philosophy of the loss function leverages the unique nonlinear correlation measurement capabilities of HSIC. Traditional correlation measurement methods (such as the Pearson correlation coefficient) can only capture linear relationships, while identity and clothing features in the TPRe-ID task often exhibit complex nonlinear coupling (such as the interaction between body shape and clothing style). Therefore, L... Decouple The loss function can utilize kernel functions in HSIC to capture higher-order correlations between features. Furthermore, L... Decouple The loss function can reduce computational complexity and improve stability through dediagonalization and layer normalization operations, adapting to the batch optimization requirements in deep learning training. Compared to other decoupling methods, L...Decouple The loss mechanism forces identity features and clothing features to remain independent in the semantic space, causing them to gradually become orthogonal, thus resolving the performance bottleneck caused by feature mixing in the traditional TPRe-ID model. This loss reduces semantic mixing between features and improves decoupling.

[0093] In this embodiment of the invention, the gating mechanism is generated using a sigmoid activation function in the deentanglement module, and is used to dynamically weight identity features and clothing features. The gating regularization loss L... Gate The mean squared error (MSE) is used to constrain the distribution of gate weights to avoid bias towards 0 or 1, thus ensuring the collaborative optimization of the two branches; and low weights (default 0.01) are used to ensure L Gate The loss term, used as a regularization term, does not interfere with the primary optimization objective. This design enhances the model's adaptability to different scenarios (e.g., costume changes or ambiguous descriptions); L Gate To balance the contributions of identity and clothing branches, a gating weight close to 0.5 is encouraged; L Gate The expression for loss is:

[0094]

[0095] Where M is the total number of elements in the gate vector, used for layer normalization of mean square error; g i The gating weights are obtained by passing g through the sigmoid activation function of the last linear layer. i , of which 0 <g i <1.

[0096] In embodiments of the present invention, such as Figure 2 and Figure 3As shown, firstly, the input dataset contains pedestrian images and natural language descriptions. Using the Text Description Separation Module (TDSM), the description text is segmented into clothing descriptions and identity descriptions using Natural Language Processing (NLP) tools and a pre-trained language model, generating independent JavaScript Object Notation (JSON) files. Empty description samples are filled with default values ​​to ensure completeness. Next, the images are encoded into sequence features by an image encoder. The clothing and identity text (e.g., clothing descriptions: "dress, hat, shorts, sleeves, pants, jacket, coat, skirt, sweater, sleeves, hat, scarf, belt, collar, neckline, flowers"; person description: "She is wearing a purple long-sleeved ankle-length dress. The dress has a pattern") are processed by a text encoder, supplemented by multi-head self-attention to enhance semantics, generating layer normalization (Layer Normalization) text embeddings. Subsequently, mean pooling is performed. The Identity-Clothing Bidirectional Decoupling Alignment Module (BDAM) receives image features, extracts identity and clothing features separately through a dual-branch structure, and achieves feature separation and dynamic weighting by combining cross-attention and gating mechanisms. The output decoupled features are then aligned with the text embedding. adv L Cloth This represents minimizing the clothing adversarial loss and clothing contrast loss; the decoupled identity text features e id_txt With clothing text features e cloth_txt The input is the Mamba State Space Fusion Module (SSM), which aligns modal features using a Multilayer Perceptron (MLP) and performs pre-gating to generate weights. The Mamba SSM efficiently models long-range dependencies and outputs fused features. Finally, a multi-objective robust optimization module integrates InfoNCE loss, biometric contrast loss, clothing adversarial loss, decoupling loss, and gating regularization loss to collaboratively optimize identity alignment and clothing robustness, outputting the final retrieval result. This process, through structural decoupling, efficient fusion, and multi-task optimization, ensures accurate matching and robust retrieval in complex scenarios such as clothing changes and occlusion, significantly improving performance.

[0097] This invention uses the CUHK-PEDES dataset as the training foundation and ensures the model's accurate matching and robust performance in complex scenes through phased optimization and multi-task collaborative training. The CUHK-PEDES dataset contains approximately 40,000 pedestrian images (224×224 pixels) and corresponding natural language descriptions. Each image is associated with multiple descriptions. The dataset is divided into a training set (approximately 34,000 images), a validation set, and a test set (approximately 3,000 images each) according to standard settings. The training process uses the parameters specified in the configuration file: Adam optimizer (learning rate 0.0001, weight decay 0.001), cosine annealing learning rate scheduling (80 training epochs, milestones 40 and 60 epochs), batch size 128, and warm-up steps 500.

[0098] In this embodiment of the invention, the training process is as follows: Figure 2 As shown, the model first processes the descriptive text of the CUHK-PEDES dataset using the Text Description Separation Module (TDSM), segmenting the descriptions into clothing descriptions and identity descriptions using regular expressions, providing structured input for subsequent modules. Then, the model employs a phased training strategy to improve efficiency. Initially, a separate pre-trained image encoder (based on ViT, with pre-trained weights from vit-base-patch16-224) and a text encoder (based on BERT, with pre-trained weights from bert-base-uncased) are used. InfoNCE loss is used to align image and text embeddings, optimizing cross-modal feature consistency. Next, the identity-clothing bidirectional decoupling alignment module (BDAM) is trained, using decoupling loss and gated regularization loss to optimize the independence and branch balance of identity and clothing features, respectively. Subsequently, the Mamba state space fusion model (SSM, 256 dimensions, 16 state dimensions, 2-layer structure) optimizes fine-grained semantic fusion using clothing contrast loss and adversarial loss. Finally, a multi-objective robust optimization module integrates all modules in a joint optimization phase, combining multi-task losses and improving identity discriminativeness and clothing robustness through collaborative constraints. The optimization strategy ensures efficient model convergence, achieving accurate cross-modal matching on the CUHK-PEDES dataset and significantly improving retrieval performance in scenarios such as clothing changes and occlusion.

[0099] This invention significantly improves retrieval accuracy and robustness in complex scenarios through the collaborative design of a text description separation module, an identity-clothing bidirectional decoupling alignment module using the Mamba state-space model, and a multi-objective robust optimization module. The text description separation module uses keyword-guided text segmentation to accurately separate natural language descriptions into clothing and identity descriptions, providing structured input for cross-modal alignment, eliminating semantic ambiguity, and effectively improving the model's adaptability to clothing changes and ambiguous descriptions. The identity-clothing bidirectional decoupling alignment module uses a dual-branch structure and dynamic gating mechanism to separate identity and clothing features from images and achieves accurate alignment using the separated text, enhancing the model's discriminative ability in scenarios with occlusion and viewpoint changes. The Mamba state-space model efficiently models long-distance cross-modal dependencies with linear complexity, significantly reducing computational costs compared to traditional Transformer methods, while accurately capturing fine-grained semantics and improving matching performance in complex backgrounds. The multi-objective robust optimization module integrates multiple loss functions to collaboratively optimize identity consistency, clothing discriminability, and cross-modal association, ensuring the model's robustness on semantically incomplete samples. The experiment was conducted on the CUHK-PEDES dataset. Compared with existing state-of-the-art methods (BDNet, CPCL, DualFocus, MLLM4, HAM), the present invention performed well in terms of retrieval accuracy metrics (such as mAP, Rank-1, Rank-5, Rank-10). Specific data are shown in Table 1.

[0100] Table 1 Comparison of retrieval performance with other advanced methods

[0101]

[0102] As shown in Table 1, the present invention achieves a Rank-1 accuracy of 74.96% and a mAP of 70.93% in cross-modal retrieval tasks, outperforming other schemes. Through structural decoupling, efficient fusion, and multi-task optimization, the present invention provides a high-performance and robust solution for TPRe-ID tasks, possessing broad practical application value.

[0103] Existing text-image cross-modal person re-identification (TPRe-ID) technologies face challenges in fine-grained semantic alignment, struggling to effectively bridge the inherent differences between text and image modalities. To address this, this invention proposes an end-to-end cross-modal matching model. This model innovatively introduces an identity-clothing bidirectional decoupling alignment module (BDAM) and a state-space model (Mamba SSM) to perform feature decoupling and fusion, respectively. Specifically, a text description separation module (TDSM) is designed first, splitting the original text description into two sub-descriptions: identity and clothing, which serve as inputs to the BDAM module. The BDAM module utilizes an attention mechanism and a gated weighting strategy to dynamically separate identity and clothing information from the pedestrian image, and combines this with a variant loss function based on HSIC to guide the projection of text-image features onto an approximately orthogonal subspace, thereby achieving high-quality semantic alignment. Furthermore, this invention is the first to introduce the Mamba state-space model into cross-modal ReID tasks, using it as a fusion module. Leveraging its linearly complex sequence modeling capabilities, it efficiently captures deep semantic dependencies and dynamic contextual changes across modalities, overcoming the computational efficiency bottleneck of the Transformer while improving the accuracy of fine-grained semantic alignment. Experimental results show that this invention significantly outperforms existing techniques on mainstream datasets, fully demonstrating its excellent practical prospects and research value.

[0104] Compared with the prior art, the present invention has the following advantages:

[0105] 1. This invention proposes a Text Description Separation Module (TDSM). This module separates mixed text descriptions in a dataset into clothing descriptions and identity descriptions, thus completing the preprocessing of text data without manual intervention. TDSM uses NLP tools and a pre-trained language model to separate the text, and then inputs the separated text into subsequent modules for image decoupling.

[0106] 2. This invention designs an Identity-Clothing Bidirectional Decoupling Alignment Module (BDAM). This module uses an attention mechanism to dynamically decouple image features into identity features and clothing features, and aligns them cross-modal with their corresponding text descriptions. The BDAM module further introduces a gating mechanism to perform weighted fusion of identity and clothing features, effectively improving the model's robustness and generalization ability in complex contexts.

[0107] 3. This invention innovatively introduces a text-image alignment module based on Mamba SSM for the first time in sequence modeling. As the first study to apply the Mamba state-space model to cross-modal ReID tasks, this scheme effectively overcomes the computational efficiency bottleneck of the Transformer global attention mechanism, achieves accurate capture of fine-grained visual-language dependencies, and particularly improves the alignment accuracy of semantically ambiguous regions.

[0108] The technical solution provided by this invention includes a method that splits the original text description into two sub-descriptions—identity description and clothing description—using a text description separation module (TDSM) to provide structured semantic input for cross-modal feature alignment. Based on the two sub-descriptions, an identity-clothing bidirectional decoupling alignment module (BDAM) utilizes an attention mechanism and a gating weighting strategy to achieve cross-modal feature alignment. Based on this cross-modal feature alignment, the Mamba state-space model (SSM) is introduced into the cross-modal person re-identification (ReID) task, fusing image and text features. Based on the features of the fused image and text, a multi-objective robust optimization module is designed for optimization, outputting the final retrieval result. This method improves the accuracy of fine-grained semantic alignment, achieves effective contextual collaboration, and balances discriminability and robustness.

[0109] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0110] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An end-to-end cross-modal pedestrian re-identification method based on multi-domain feature alignment, characterized in that, The method includes: Step 1: Using the Text Description Separation Module (TDSM), the original text description is split into two sub-descriptions: identity description and clothing description, providing structured semantic input for cross-modal feature alignment; Step 2: Based on the two sub-descriptions of identity and clothing, the identity-clothing bidirectional decoupling alignment module BDAM uses an attention mechanism and a gating weighting strategy to achieve cross-modal feature alignment; Step 3: Based on cross-modal feature alignment, the Mamba State Space Model (SSM) is introduced into the cross-modal person re-identification (ReID) task, fusing image and text features; Step 4: Based on the features of the fused image and text, design a multi-objective robust optimization module for optimization and output the final retrieval results.

2. The method of claim 1, wherein, Step 1 includes: By using the adaptive semantic classification and post-processing mechanism of the Text Description Separation Module (TDSM), combined with syntactic analysis and semantic embedding, the input text is decomposed into clothing description and identity description, and independent data structures are constructed. For semantically ambiguous or missing descriptions, TDSM adopts a dynamic threshold strategy and supplementation mechanism to ensure the integrity and consistency of the input.

3. The method of claim 1, wherein, Step 2 includes: The image encoder's sequence features are processed through a dual-branch structure to extract identity features and clothing features respectively. Self-attention and cross-attention mechanisms are combined to capture semantic associations and feature interactions. A dynamic gating weighting strategy optimizes feature weighting to generate highly discriminative identity and clothing features. Text descriptions are embedded by the encoder and aligned with image features.

4. The method of claim 1, wherein, Step 3 includes: First, a feature alignment network is used to unify the feature representation of images and text, and a pre-gating mechanism is used to generate modal weights. Then, the Mamba state-space model (SSM) is used for sequence modeling, and finally the features of the fused image and text are output.

5. The method according to claim 1, characterized in that, Step 4 includes: The optimization process balances the objectives of each task through dynamic weights, and collaboratively optimizes identity alignment, clothing robustness, and cross-modal association through a multi-task loss function; it also imposes constraints on decoupled features, fused features, and classification results, and outputs the final retrieval results. First, through InfoNCE loss L InfoNCE First, image embeddings and identity text embeddings are compared and aligned to optimize cross-modal semantic consistency; second, biometric contrast loss L... Bio Loss L compared to clothing Cloth Identity features and clothing features are aligned separately through projection layers to enhance the discriminative power of decoupled features; clothing adversarial loss L ClothAdv Minimize the similarity between clothing features and unmatched clothing text embeddings to enhance the discriminative power of clothing features; clothing matching loss L ClothMatch Further enhance the fine-grained association between clothing image embedding and clothing text embedding; and combine it with the decoupling loss L based on the Hilbert-Schmidt independence criterion (HSIC). Decouple To reduce the correlation between identity features and clothing features, text-image feature projection is guided to an approximately orthogonal subspace to promote feature separation; gated regularization loss L Gate Gating weight W based on mean square error constraint id and W clo To balance the contributions of the branches; the total loss function L is obtained by weighted summation of the above multi-task loss functions. Total It collaboratively optimizes identity alignment, clothing robustness, and cross-modal association, outputting the final retrieval results. The total loss function is L. Total The expression is: L Total = W InfoNCE L InfoNCE + W Bio L Bio + W Cloth L Cloth + W ClothAdv L ClothAdv + W Decouple L Decouple + W Gate L Gate ; wherein the loss weight W InfoNCE = 1.0, W Bio = 0.5, W Cloth = 0.5, W ClothAdv = 0.1, W Decouple = 1.0, W Gate = 0.

01.

6. The method according to claim 5, characterized in that, In the semantic space, L InfoNCE The loss aligns the image embedding with the identity text embedding, L InfoNCE The loss is based on contrastive learning, computing a similarity matrix of the image and identity text features, maximizing the similarity for positive pairs and minimizing the similarity for negative pairs, L InfoNCE The expression of the loss is: in, and These represent layer-normalized image embedding and identity text embedding, respectively; sim(·,·) is the dot product similarity; τ is the temperature parameter; and N is the batch size.

7. The method according to claim 5, characterized in that, biometric contrast loss L Bio is the identity feature extracted from the aligned image, and L Bio The loss uses the identity feature to calculate a similarity matrix with the identity text embedding after dimension reduction by a linear projection layer and layer normalization; L Bio The expression of the loss is: in, Identity features that are projected and normalized layer by layer; For identity text embedding; sim(·,·) is the dot product similarity; τ is the temperature parameter; N is the batch size; garment contrast loss L Cloth for aligning the garment features extracted from the image with the corresponding garment text embeddings, L Cloth The expression of the loss is: in, For the projection and layered normalization of clothing features; Embed text for clothing.

8. The method according to claim 5, characterized in that, garment adversarial loss L ClothAdv L is designed to minimize the similarity between garment features and mismatched garment text embeddings, ClothAdv The expression of loss is: in, The clothing features are normalized to a layered structure; For layer-normalized clothing text embedding; sim(·,·) is the dot product similarity; τ is the temperature parameter; N is the batch size; clothing matching loss L ClothMatch further reinforce the fine-grained alignment of clothing image embedding and clothing text embedding, L ClothMatch loss is optimized by bidirectional contrast to ensure the semantic consistency of clothing image features and clothing text description in details, L ClothMatch The expression of the loss is: in, parameter Embedding in clothing images.

9. The method according to claim 5, characterized in that, Decoupling loss L based on HSIC Decouple aims to minimize the semantic correlation between the extracted identity feature f id and the clothing feature f cloth ; L Decouple The design of the loss is based on the core idea of HSIC, which measures the independence of two random variables through the kernel method; HSIC quantifies the correlation by calculating the dot product of the kernel matrix of the two feature sets, and the smaller the value, the more independent the features are; while the decoupling loss L Decouple uses the nonlinear correlation measure of HSIC, whose core goal is to make the identity feature focus on the identity discrimination information and the clothing feature focus on the dynamic appearance attribute, thereby enhancing the robustness and generalization ability; L Decouple The expression of the loss is: in, and The identity and clothing features are projected and normalized; N is the batch size.

10. The method according to claim 5, characterized in that, Gating regularization loss L Gate The gating weights distribution is constrained by mean square error (MSE) to avoid the gating weights biasing to 0 or 1, so as to ensure the collaborative optimization of the two branches; and low weights are used to ensure that L Gate loss as a regularization term; L Gate The contributions of the identity and the clothing branches are balanced, and the gating weights are encouraged to be close to 0.5; L Gate The expression of the loss is: where M is the total number of elements in the gating vector, used for layer normalization mean squared error; g i is the gating weight, obtained by the Sigmoid activation function of the last linear layer i where 0 < g i <1.

Images