Cross-modal pedestrian re-identification method and device based on text modality alignment

Abstract

The application discloses a cross-modal pedestrian re-identification method and device based on text modal alignment, and belongs to the technical field of artificial intelligence. The method comprises the following steps: training a learnable vector to capture modal shared information related to identity in an infrared image and a visible light image; obtaining a text modal representation of each pedestrian ID in combination with the trained learnable vector, and using an image encoder to extract infrared modal representation and visible light modal representation of the pedestrian ID in the infrared image and the visible light image; training the image encoder based on mutual alignment between the text modal representation, the infrared modal representation and the visible light modal representation; and encoding pedestrians in a target infrared image or a target visible light image based on the trained image encoder, and obtaining a cross-modal pedestrian re-identification result through classification of the encoding result. The application can improve the precision of a cross-modal pedestrian re-identification model.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology. Specifically, it relates to a method and apparatus for cross-modal pedestrian re-identification based on text modality alignment. Background Technology

[0002] Pedestrian re-identification, a sub-problem of image retrieval, aims to retrieve images of a specific pedestrian from disjoint cameras. Most existing methods focus on matching images captured by visible light cameras. However, visible light cameras can become ineffective in capturing images under challenging conditions, such as at night or in low-light environments. Fortunately, modern surveillance cameras can autonomously switch modes to capture infrared modal images. This introduces the research task of infrared-visible light cross-modal pedestrian re-identification, which aims to match images of a specific person captured by cameras of different modalities.

[0003] Existing methods align visible light and infrared modal features by learning modality-shared features. For example, some generative methods utilize generative adversarial networks (GANs) for modality transformation, while non-generative methods design two-stream networks to learn modality-shared features. Despite some success, these methods discard modality-related features that help mitigate intra-modal variations, resulting in lower robustness and discriminability of the learned features. Therefore, auxiliary modality-based methods have been proposed, which use generated intermediate modalities to capture modality-shared and modality-related features in heterogeneous images. For example, X-Modality introduces an auxiliary intermediate modality using a lightweight network to facilitate cross-modal retrieval tasks. SMCL introduces a generative modality to mitigate modality gaps, guiding the generation of modality-independent feature representations. However, these methods generate auxiliary modalities by directly fusing raw pixels from visible light and infrared images, resulting in a lack of high-level semantic information in the synthesized modalities. Furthermore, modality gaps still exist between the auxiliary modality and the visible light / infrared modalities, and pairwise alignment of the three modalities leads to fluctuations during training, failing to narrow the gaps between these three modalities in the feature space. Ultimately, this will reduce the accuracy of cross-modal retrieval. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a cross-modal person re-identification method and apparatus based on text modality alignment. This method aligns pedestrian image features from both infrared and visible light modalities, thereby improving the accuracy of the cross-modal person re-identification model. Therefore, in the first stage, this invention learns a semantically informative text modal representation for each pedestrian, and then in the second stage, aligns the infrared and visible light modalities to the text modality.

[0005] To achieve the above-mentioned objectives, the technical solution of this invention includes the following:

[0006] A cross-modal person re-identification method based on text modality alignment, the method comprising:

[0007] Train a learnable vector to capture identity-related modal sharing information in infrared and visible light images;

[0008] The text modal representation of each pedestrian ID is obtained by combining the learned vectors after training, and the infrared modal representation and visible light modal representation of the pedestrian ID are extracted from the infrared image and the visible light image using an image encoder;

[0009] The image encoder is trained based on the alignment between the text modality representation, the infrared modality representation, and the visible light modality representation;

[0010] The trained image encoder encodes pedestrians in the target infrared or visible light image, and the cross-modal pedestrian re-identification results are obtained by classifying the encoding results.

[0011] Furthermore, the training of a learnable vector to capture identity-related modal sharing information in infrared and visible light images includes:

[0012] A learnable vector is randomly initialized, and the parameters of the image encoder and text encoder are fixed.

[0013] Obtain the identity information description D of each person's ID in the infrared image. infrared and the description of identity information in visible light images D visible ;

[0014] The identity information description D infrared and the identity information description D visible After vector mapping, the vectors are concatenated with the learnable vector to obtain infrared text vectors and visible light text vectors.

[0015] Infrared text vectors and visible light text vectors are input into the text encoder, and infrared images and visible light images are input into the image encoder. Optimization is then performed based on image-to-text contrast loss and text-to-image contrast loss to obtain the trained learnable vectors.

[0016] Furthermore, the step of obtaining the text modal representation of each pedestrian ID by combining the trained learnable vectors includes:

[0017] Obtain the identity information description D of each person's ID in the visible light image. visible ;

[0018] The identity information description D visible After vector mapping, it is concatenated with the learnable vector to obtain the visible light text vector;

[0019] Input the visible light text vector into the text encoder to obtain the text modal representation of the pedestrian ID.

[0020] Furthermore, the structure of the image encoder includes: p VIT layers, q pedestrian identity information enhancement modules, and a block feature extractor; wherein, p > q, and the latter q VIT layers and the pedestrian identity information enhancement modules form a stacked structure, p and q are natural numbers, and the processing flow of the pedestrian identity information enhancement modules includes:

[0021] Obtain the s local feature representations Z from the output of the VIT layer. patch And a global feature representation Z class ;

[0022] Based on s local feature representations Z patch Generate modally shared channel weights m C ;

[0023] Calculate the modal shared feature F shared =m C ⊙Z patch ; where ⊙ represents element-wise multiplication;

[0024] Calculate modal-specific features F spec =(1-m C )⊙IN(Z patch ); where IN represents instance normalization;

[0025] The global feature representation Z class and feature F patch =F spec +F shared This serves as the output feature of the pedestrian identity information enhancement module.

[0026] Furthermore, the infrared modal representation of the pedestrian ID in the infrared image is extracted using an image encoder, including:

[0027] For the infrared image, obtain the output features of the pedestrian identity information enhancement module of the last layer;

[0028] Calculate prototype Q and feature F patch The similarity matrix S between them patch ;

[0029] Using the similarity matrix S patch These features are used as weights to aggregate block features and create latent semantic partial features. Where i represents the i-th latent semantic part feature;

[0030] Based on the latent semantic features and the global feature representation Z class The infrared mode representation is obtained.

[0031] Furthermore, the loss used to train the image encoder includes: text alignment loss L TA Visible light-infrared alignment loss L VIA Cross-entropy loss L id And triple classification loss L Tri Wherein, the text alignment loss L TA Used to guide the alignment of infrared modal representation and visible light modal representation with text modal representation; the visible light-infrared alignment loss L VIA Used to guide the alignment of visible light mode representation with infrared mode representation.

[0032] Furthermore, the text alignment loss Where i represents the i-th image, which includes infrared and visible light images; C represents the number of pedestrian IDs; k represents the k-th pedestrian; V i T represents the image modal representation of the i-th image. k The text modal representation of the k-th pedestrian, s(V i ,T k ) represents the image modality representation V i and text modal representation T k The cosine similarity, where j represents the j-th pedestrian, l k This represents the value in the target distribution.

[0033] Furthermore, the optical-infrared alignment loss Where, x i Let y represent the features of the i-th image. i Indicates the yth i N represents the total number of images, including infrared images and visible light images. τ is the temperature coefficient, A[y i [Is the yth] i The class center feature of each pedestrian, I(x) i Let represent the image modal representation of the i-th image, C represent the total number of pedestrians, and k represent the k-th pedestrian.

[0034] A cross-modal person re-identification device based on text modality alignment, the device comprising:

[0035] A learnable vector acquisition module is used to train a learnable vector to capture identity-related modal sharing information in infrared and visible light images;

[0036] The image encoder training module is used to obtain the text modal representation of each pedestrian ID by combining the trained learnable vectors, and to extract the infrared modal representation and visible light modal representation of the pedestrian ID in the infrared image and the visible light image using the image encoder; the image encoder is trained based on the alignment between the text modal representation, the infrared modal representation and the visible light modal representation;

[0037] The pedestrian re-identification module is used to encode pedestrians in the target infrared image or target visible light image based on the trained image encoder, and obtain cross-modal pedestrian re-identification results by classifying the encoding results.

[0038] A computer device, characterized in that the computer device comprises: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the cross-modal person re-identification method based on text modality alignment as described above.

[0039] Compared with existing technologies, this invention proposes a cross-modal person re-identification method based on text modality alignment, named TMAL, for visible-infrared cross-modal person re-identification tasks. TMAL aims to leverage the powerful image-text alignment capabilities of the CLIP model to learn the semantic text representation of each individual pedestrian and generate a text modality. The proposed modality alignment module (MAM) explicitly guides the alignment of infrared and visible light modalities with the text modality. Furthermore, this invention introduces an identity enhancement module to extract information-rich modality-related features, further improving the performance of cross-modal retrieval. Attached Figure Description

[0040] Figure 1 This is a schematic diagram illustrating the difference between the method of the present invention and previous methods.

[0041] Figure 2 This is a block diagram of a cross-modal person re-identification method based on text modality alignment.

[0042] Figure 3 This is a schematic diagram of the ablation experiment and the visualization experiment. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below through specific implementations and in conjunction with the accompanying drawings.

[0044] The framework of the text modality-based alignment method proposed in this invention is as follows: Figure 2As shown, in the first stage, learnable vectors are used to capture semantic information descriptions in the image as text modalities. In the second stage, the proposed MAM aligns the infrared and visible light modalities with the text modalities (through text modal alignment loss), as well as the mutual alignment of the two modalities (visible light and infrared modal alignment loss). Furthermore, by introducing a pedestrian information augmentation module (IEM) within the Transformer network layers, the image encoder's ability to extract modality-specific features is further enhanced.

[0045] During the first training phase, this invention introduces learnable vectors to obtain auxiliary text modal representations, which are independent for each pedestrian ID. The CLIP model includes two encoders: an image encoder I(·) and a text encoder T(·). For the text modality, the text input to T(·) is "A infrared photo of a[X]1[X]2[X]3...[X]". M person" and "A visible photo of a[X]1[X]2[X]3...[X] M "person" refers to the image applied to infrared and visible light images, respectively, where each [X] m (m∈1,...,M) represents a learnable vector. The input to the image encoder is a visible light and an infrared image. The infrared / visible light information in the image is matched with 'infrared' / 'visible' in the given text. Therefore, the learnable vector can capture modality-shared identity-related information. At this stage, the invention keeps the parameters of I(·) and T(·) fixed and only optimizes the learnable vector [X]. m Because the infrared and visible light images of each pedestrian ID share a common learnable vector [X]. m This invention uses image-to-text contrast loss and text-to-image contrast loss to optimize the model.

[0046] (I) Modal Alignment Module:

[0047] like Figure 2 As shown in (b), during the second phase of training, this invention introduces a Modal Alignment Module (MAM) in which the learnable vectors and text encoder remain frozen, and only the image encoder is optimized. Specifically, this invention uses the text description "A photo of a[X]1[X]2[X]3...[X]" MThe input "person" is T(·) to obtain the text modal representation of each pedestrian ID. Subsequently, this invention uses an image encoder to extract features from infrared and visible light images to obtain the infrared and visible light modal representations of each pedestrian ID. Furthermore, text alignment loss and visible-infrared alignment loss are introduced to optimize the image encoder, enabling it to extract both modality-shared and modality-specific features. The text alignment loss L... TA The alignment of infrared and visible light modes with the text mode is defined as follows:

[0048]

[0049] Where i represents the i-th image, C represents the number of pedestrian IDs, k represents the k-th pedestrian, and V i Let T represent the image features of the i-th image. k The text features representing the k-th pedestrian are s(V i ,T k ) represents V i and T k The cosine similarity. j represents the j-th pedestrian, l k This represents the value in the target distribution.

[0050] In addition, such as Figure 2 As shown in (d), in order to directly align the infrared and visible light modes, this invention introduces a feature set A∈R d×C The feature set A stores class centers for all pedestrian features, with each column corresponding to a class center feature of a pedestrian shared across infrared and visible light modalities. During each iteration, feature set A is updated as follows:

[0051] A[y i ]←μA[y i ]+(1-μ)I(x i (2)

[0052] Among them, A[y i ] represents the y-th i Human-centered features, I(x) i ) represents the features of the i-th image, and μ is the weight coefficient, set to 0.5.

[0053] Visible-infrared alignment loss can be defined as:

[0054]

[0055] Where x i Let y represent the features of the i-th image. i Indicates the yth i Let N represent the total number of images, and p(y) represent the number of pedestrians. i |x i) is defined as:

[0056]

[0057] Where τ is the temperature coefficient, A[y i [Is the yth] i The class center feature of each pedestrian, I(x) i Let represent the features of the i-th image, and C represent the total number of pedestrians.

[0058] To improve performance, this invention introduces a cross-entropy loss L in the second stage. id and triplet classification loss L Tri Therefore, the final loss in the second phase is:

[0059] L VIA =L id +L Tri +L VIA +L TA (5)

[0060] (II) Pedestrian Identity Enhancement Module:

[0061] To further extract informative modality-specific features, this invention introduces a pedestrian identity information enhancement module (IEM). Specifically, for visible light or infrared images, this invention represents the features of its last layer as Z∈R. s×d Z, where s and d represent the number of image blocks and the feature dimension, respectively. class ∈R 1×d Z represents the global features. patch ∈R (s-1)×d This represents local features.

[0062] like Figure 2 As shown in (c), this invention employs a channel reweighting mechanism to implicitly separate mode sharing and mode-specific features. Specifically, this invention generates the channel weights m for mode sharing in the following manner. C :

[0063] m C =σ(W2δ(W1g(Z) patch) (6)

[0064] Where σ represents the sigmoid activation function, δ represents the ReLU activation function, g(·) represents average pooling, and W1 and W2 represent learnable parameters.

[0065] Then, it can be done through formula F shared =m C ⊙Z patch Calculate modal shared features, and through F spec =(1-mC )⊙IN(Z patch The modality-specific features are calculated, where ⊙ denotes element-wise multiplication and IN denotes instance normalization. Therefore, the output feature F of IEM is... patch It can be defined as:

[0066] F patch =F spec +F shared (7)

[0067] This invention utilizes an additional learnable prototype Q = [Q1, Q2, ..., Q...] n ]∈R n×d This invention introduces a Patch Feature Extractor (PFE) to obtain implicit semantic patch features. The invention first calculates the prototype Q and F... patch The similarity matrix S between them patch ∈R n×(s-1) :

[0068]

[0069] in, This represents matrix multiplication. The higher the similarity between local block features and the learnable prototype, the stronger the association with that prototype. Therefore, this invention can use S... patch These features are used as weights to aggregate block features and create latent semantic partial features. The specific calculation method is as follows:

[0070]

[0071] in Q represents i and The similarity between them, and then q patch With Z class The final feature F is obtained by piecing them together:

[0072]

[0073] Where n represents the number of learnable prototypes.

[0074] The following experiments were conducted on two benchmark datasets (SYSU-MM01 and RegDB).

[0075] SYSU-MM01, one of the most challenging datasets for cross-modal person re-identification, was collected using four visible light cameras and two infrared cameras, covering both indoor and outdoor environments. The training set consists of 22,258 RGB images and 11,909 infrared images, encompassing 395 unique identities. The query set contains 3,803 infrared images, while the database contains 301 randomly selected RGB images.

[0076] RegDB uses a dual-camera setup for data collection, including one visible light camera and one corresponding infrared camera. It comprises 412 identities, half used for training and the other half for testing. Each identity is represented by 10 RGB images and 10 infrared images.

[0077] In this invention, the standard cumulative matching characteristic (CMC) curve and mean accuracy (mAP) are used as evaluation metrics, which are widely used in retrieval tasks.

[0078] Experimental Results: To fully verify the method of this invention, this invention was compared with some recent cross-modal pedestrian re-identification methods.

[0079] The SYSU-MM01 dataset. As shown in Table 1, the TMAL proposed in this invention achieves the best results in All-Search mode, with a Rank 1 of 70.58% and an mAP of 73.96%. Specifically, compared with the recently proposed FMCNet, DART, and PMT, the method of this invention exceeds the Rank 1 by at least 1.86% and the mAP by at least 7.67%. Furthermore, compared with the auxiliary modality-based methods X-Modality and SMCL, the method of this invention improves the Rank 1 by at least 3.19% and the mAP by 12.18%. This is because the text modality introduced by the CLIP model has strong discriminability and carries more semantic information. This helps prevent training oscillations and enhances the model's cross-modal retrieval capabilities.

[0080]

[0081]

[0082] Table 1

[0083] The RegDB dataset. Similarly, as shown in Table 2, the model of this invention achieved 82.65% Rank1 and 85.04% mAP on the RegDB dataset with visible light retrieval of infrared images. However, at the Rank1 metric, the method of this invention is slightly inferior to state-of-the-art methods. This may be partly due to the limited size of RegDB, i.e., only 10 infrared images and 10 visible light images per pedestrian, resulting in insufficient learning of textual modal semantic information.

[0084]

[0085] Table 2

[0086] Module ablation experiments: Table 3 shows the performance improvements achieved by the two proposed modules, MAM and IEM. MAM improves the Rank1 and mAP metrics by 5.11% and 3.80%, respectively, demonstrating its effectiveness in guiding and aligning infrared and visible modes. Similarly, IEM improves the Rank1 and mAP metrics by 4.86% and 2.50%, respectively. This may be attributed to the enhancement of identity semantics by the inherent structure of the proposed modules. When MAM and IEM are combined, the performance is further improved, demonstrating the effectiveness of the overall framework.

[0087] Baseline Modal alignment module Identity information enhancement module R1 R10 mAP √ 64.29 95.54 69.81 √ √ 69.40 96.17 73.61 √ √ 69.15 96.65 72.31 √ √ √ 70.56 96.22 73.90

[0088] Table 3

[0089] Modal Alignment Module (MAM) Analysis: Figure 3 (a) and (b) in the figure respectively show L VIA and L TA Performance improvements achieved in the All-search and Indoor Search settings of SYSU-MM01. For example, in the All-search setting, L VIA It improved by 3.27% and 1.54% in Rank1 and mAP metrics, respectively, while L TA The performance was improved by 1.84% and 2.24% on Rank1 and mAP, respectively. When the two are combined, the text alignment loss and visible-infrared alignment loss further improve the overall performance, experimentally verifying the intuition of the present invention to use the CLIP model to provide high-level semantic information and align modalities to text modalities.

[0090] Visual analysis: such as Figure 3As shown in (c), this invention simultaneously visualizes visible light and infrared image features, as well as text features learned in 3D coordinates. Specifically, this invention randomly selected eight pedestrians from SYSU-MM01, with 40 RGB images and 40 infrared images for each pedestrian. Features of the same pedestrian in different modalities were well clustered together, especially around the text modality. This clearly demonstrates the intuition of this invention that the text modality not only possesses strong semantic information but is also very effective in assisting the alignment of two modalities.

[0091] The above embodiments are provided merely for the purpose of describing the present invention and are not intended to limit the scope of the invention. The scope of the invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the invention should be covered within the scope of the invention.

Claims

1. A cross-modal person re-identification method based on text modality alignment, characterized in that, The method includes: Train a learnable vector to capture identity-related modal sharing information in infrared and visible light images; The text modal representation of each pedestrian ID is obtained by combining the learned vectors after training, and the infrared modal representation and visible light modal representation of the pedestrian ID are extracted from the infrared image and the visible light image using an image encoder; The image encoder is trained based on the alignment between the text modality representation, the infrared modality representation, and the visible light modality representation; The trained image encoder is used to encode pedestrians in the target infrared image or the target visible light image, and the cross-modal pedestrian re-identification results are obtained by classifying the encoding results. The training of a learnable vector to capture identity-related modal sharing information in infrared and visible light images includes: A learnable vector is randomly initialized, and the parameters of the image encoder and text encoder are fixed. Obtain the identity information description of each person's ID in the infrared image. and the description of identity information in visible light images ; Description of the identity information and the description of the identity information After vector mapping, the vectors are concatenated with the learnable vector to obtain infrared text vectors and visible light text vectors. Infrared text vectors and visible light text vectors are input into the text encoder, and infrared images and visible light images are input into the image encoder. Then, optimization is performed based on image-to-text contrast loss and text-to-image contrast loss to obtain the trained learnable vectors. The step of obtaining the text modal representation of each pedestrian ID by combining the trained learnable vectors includes: Retrieve the identity information description of a group of people in a visible light image. ; Description of the identity information After vector mapping, it is concatenated with the learnable vector to obtain the visible light text vector; Input the visible light text vector into the text encoder to obtain the text modal representation of the pedestrian ID; The structure of the image encoder includes: One ViT layer, A pedestrian identity information enhancement module and a block feature extractor; among them; And the following Each ViT layer and the pedestrian identity information enhancement module form a stacked structure. , For natural numbers, the processing flow of the pedestrian identity information enhancement module includes: Get the output of the ViT layer Local feature representation and a global feature representation ; based on Local feature representation Generate modally shared channel weights ; Computation of modal sharing features ;in, This indicates element-wise multiplication; Calculate modal-specific features ;in, Indicates instance normalization; Representing global features and characteristics This serves as the output feature of the pedestrian identity information enhancement module.

2. The method according to claim 1, characterized in that, The infrared modal representation of the pedestrian ID in the infrared image is extracted using an image encoder, including: For the infrared image, obtain the output features of the pedestrian identity information enhancement module of the last layer; Calculate prototype Q and features Similarity matrix between ; Using a similarity matrix These features are used as weights to aggregate block features and create latent semantic partial features. ;in, Indicates the first One potential semantic part feature; Based on the latent semantic features and the global feature representation The infrared mode representation is obtained.

3. The method according to claim 1, characterized in that, The loss used to train the image encoder includes: text alignment loss. Visible-infrared alignment loss Cross-entropy loss and triplet classification loss ; wherein, the text alignment loss Used to guide the alignment of infrared modal representation and visible light modal representation with text modal representation; the visible light-infrared alignment loss Used to guide the alignment of visible light mode representation with infrared mode representation.

4. The method according to claim 3, characterized in that, The text alignment loss ;in, Indicates the first The images include: infrared images and visible light images. This indicates the number of pedestrian IDs. Indicates the first A pedestrian, Indicates the first Image modality representation of a single image Indicates the first Text modal representation of a pedestrian Image modality representation and text modal representation cosine similarity, Indicates the first A pedestrian, This represents the value in the target distribution.

5. The method according to claim 3, characterized in that, Visible-infrared alignment loss ;in, Indicates the first Features of the image This indicates the total number of images, which include infrared images and visible light images. , It is the temperature coefficient. Indicates the first Image modality representation of a single image This represents the total number of pedestrians. Indicates the first A pedestrian.

6. A cross-modal person re-identification device based on text modality alignment, characterized in that, The device includes: A learnable vector acquisition module is used to train a learnable vector to capture identity-related modal sharing information in infrared and visible light images; The image encoder training module is used to obtain the text modal representation of each pedestrian ID by combining the trained learnable vectors, and to extract the infrared modal representation and visible light modal representation of the pedestrian ID in the infrared image and the visible light image using the image encoder; the image encoder is trained based on the alignment between the text modal representation, the infrared modal representation and the visible light modal representation; The pedestrian re-identification module is used to encode pedestrians in the target infrared image or target visible light image based on the trained image encoder, and obtain cross-modal pedestrian re-identification results by classifying the encoding results; The training of a learnable vector to capture identity-related modal sharing information in infrared and visible light images includes: A learnable vector is randomly initialized, and the parameters of the image encoder and text encoder are fixed. Obtain the identity information description of each person's ID in the infrared image. and the description of identity information in visible light images ; Description of the identity information and the description of the identity information After vector mapping, the vectors are concatenated with the learnable vector to obtain infrared text vectors and visible light text vectors. Infrared text vectors and visible light text vectors are input into the text encoder, and infrared images and visible light images are input into the image encoder. Then, optimization is performed based on image-to-text contrast loss and text-to-image contrast loss to obtain the trained learnable vectors. The step of obtaining the text modal representation of each pedestrian ID by combining the trained learnable vectors includes: Retrieve the identity information description of a group of people in a visible light image. ; Description of the identity information After vector mapping, it is concatenated with the learnable vector to obtain the visible light text vector; Input the visible light text vector into the text encoder to obtain the text modal representation of the pedestrian ID. The structure of the image encoder includes: One ViT layer, A pedestrian identity information enhancement module and a block feature extractor; among them; And the following Each ViT layer and the pedestrian identity information enhancement module form a stacked structure. , For natural numbers, the processing flow of the pedestrian identity information enhancement module includes: Get the output of the ViT layer Local feature representation and a global feature representation ; based on Local feature representation Generate modally shared channel weights ; Computation of modal sharing features ;in, This indicates element-wise multiplication; Calculate modal-specific features ;in, Indicates instance normalization; Representing global features and characteristics This serves as the output feature of the pedestrian identity information enhancement module.

7. A computer device, characterized in that, The computer device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the cross-modal pedestrian re-identification method based on text modality alignment as described in any one of claims 1-5.

Images