Image cross-modal person re-identification method and system based on text clue alignment

Abstract

The present invention discloses an image cross-modal pedestrian re-identification method and system based on text clue alignment, which relates to the technical field of pedestrian re-identification. Specifically, the method and system include: using the visual encoder in the trained pedestrian re-identification model to extract features from the to-be-retrieved image and the retrieval images in the retrieval library to obtain feature vectors, calculating the similarity between the feature vector of the to-be-retrieved image and the feature vectors of each retrieval image and arranging them in descending order, and taking the first K corresponding retrieval images as the recognition results of the to-be-retrieved image; the method and system for pedestrian re-identification effectively mine the high-level text semantic information hidden in the image data, thereby improving the accuracy of pedestrian re-identification.

Description

Technical Field

[0001] The present invention relates to the technical field of pedestrian re-identification, and in particular to a method and system for cross-modal pedestrian re-identification based on text clue alignment. Background Art

[0002] The person re-identification (Re-ID) task has important applications in public safety and security, as Re-ID can be used to track the location of suspects in urban surveillance cameras, thereby locating their location information. Previous Re-ID tasks have primarily focused on pedestrian retrieval between RGB images captured by daytime cameras, which is insufficient for cross-modal retrieval between RGB images captured by daytime cameras and infrared (IR) images captured by nighttime cameras. The visible-infrared person re-identification (VI-ReID) task involves retrieving all images containing the same pedestrian from a database containing images of that pedestrian, given a daytime (or nighttime) image. The image can be captured by a nighttime (or daytime) camera. In recent years, many related studies have attempted to address this problem by proposing methods to effectively eliminate the modality gap, believing that the modality gap is the main obstacle to achieving satisfactory retrieval performance between daytime RGB images and nighttime IR images.

[0003] Existing methods for eliminating the modality gap between daytime RGB images and nighttime infrared (IR) images can be categorized into two approaches. One approach is based on generative models, aiming to utilize generative techniques to learn the transformation relationship between RGB and IR images. This approach can convert RGB (IR) images into IR (RGB) images, unifying them into the same modality and eliminating the modality gap. However, images generated by generative model-based methods inherently exhibit distribution differences compared to real images, and the quality of the generated images may be unsatisfactory. These factors limit the performance of these approaches.

[0004] In contrast, methods based on modality invariance have shown significant advantages. The idea behind this approach is to identify common pedestrian features shared between RGB and infrared (IR) images as discriminative features, aiming to eliminate modal differences. However, visual content learned solely with image supervision lacks high-level semantic information, making modality alignment challenging. With the significant progress made by large models in various research fields, some literature has explored applying CLIP to the task of person re-identification (VI-ReID). It was found that by implicitly mining high-level textual semantic information from RGB and infrared (IR) images and using this information as new cues for image alignment, the gap between the two modalities can be effectively narrowed. However, using CoOP technology (a cue learning method) to obtain image-based text vectors has difficulty effectively mining the high-level textual semantic information hidden in the image data, thus failing to realize the potential of large models for VI-ReID. Summary of the Invention

[0005] Based on the technical problems existing in the background technology, the present invention proposes an image cross-modal pedestrian re-identification method and system based on text clue alignment, which effectively mines the high-level text semantic information hidden in the image data and improves the accuracy of cross-modal pedestrian re-identification.

[0006] The proposed cross-modal person re-identification method based on text clue alignment uses the visual encoder in the trained person re-identification model to extract features from the target image and the search images in the search library to obtain feature vectors. The similarity between the feature vector of the target image and the feature vectors of each search image is calculated and arranged in descending order. The top K corresponding search images are taken as the recognition results of the target image.

[0007] The training process of the person re-identification model is as follows:

[0008] Generate a pedestrian description vector based on the acquired pedestrian image, and use the frozen CLIP model, the pre-trained text fusion network and the pre-trained interaction network to generate a newly defined text feature vector;

[0009] The CLIP model's visual encoder is used as an image feature extractor, with the front part of the visual encoder serving as a modality-shared visual encoder and the back part as a modality-private visual encoder. The image feature vector obtained by passing the pedestrian image through the trainable visual encoder is used as the key and value in the cross-attention. The newly defined text feature vector is used as the query in the cross-attention. Feature exchange is performed to obtain a semantically guided feature vector, which is then input into the classifier.

[0010] Construct an overall loss function and adjust the trainable parameters in the visual encoder and classifier.

[0011] Furthermore, in generating a pedestrian description vector based on the acquired pedestrian image, the pedestrian description vector includes a pedestrian identification attribute-aware text description and a pedestrian contour-aware text feature vector. The pedestrian description vector generation process is as follows:

[0012] The pedestrian image and the questions corresponding to the predefined attribute description template are input into the BLIP model, and the description template is filled with information using the question-answering model to form a pedestrian identification attribute perception text description;

[0013] The SCHP technique is used to extract pedestrian contour information from pedestrian images and input it into the frozen CLIP model. The improved CoOP technique is used to obtain the pedestrian contour-aware text feature vector.

[0014] Furthermore, the improved CoOP technology is used to obtain the pedestrian contour perception text feature vector, specifically:

[0015] A set of learnable feature vectors is set for each category of pedestrian to represent the outline text description information, and the pedestrian outline information and the corresponding outline text description information are input into the frozen CLIP model respectively;

[0016] The contrast loss in the CLIP model is used to optimize the text description information, learn text features that can express pedestrian contour information, and thus obtain a pedestrian contour-aware text feature vector.

[0017] Furthermore, the generation process of the newly defined text feature vector is specifically as follows:

[0018] Input the pedestrian identification attribute-aware text description into the text encoder of the frozen CLIP model to obtain the pedestrian identification attribute-aware text feature vector;

[0019] Then, the pedestrian identification attribute perception text feature vector and the pedestrian contour perception text feature vector are concatenated and input into the pre-trained text fusion network to obtain the fused text features;

[0020] Input the pedestrian image corresponding to the pedestrian contour perception text feature vector into the visual encoder of the frozen CLIP model to obtain the image feature vector;

[0021] The fused text feature and image feature vector are input into the pre-trained interaction network to obtain a newly defined text feature vector.

[0022] Furthermore, the text fusion network includes three fully connected layers, and the interaction network includes a standard Transformer decoder with a depth of 3;

[0023] The two networks construct the following pre-training loss function in pre-training :

[0024] ;

[0025] in, is the loss function from image to text, is the loss function from text to image, is the identity loss function.

[0026] Furthermore, the total loss function is constructed as follows:

[0027] Generate a set of newly defined text feature vectors corresponding to all pedestrian images in the training set based on the pre-trained text fusion network and interaction network;

[0028] Counting the prototype features of different types of text features from the newly defined text feature vector set;

[0029] The first loss function is constructed by taking the prototype features of the class text as the alignment clue, so that different modal features are aligned with the prototype features of the class text;

[0030] The semantically guided feature vector is input into the classifier and constrained by a second loss function constructed using ID loss and ternary loss, so that the semantically guided feature vector has identity discrimination and the intra-class distance is smaller than the inter-class distance at the feature space level.

[0031] The total loss function is obtained by adding the first loss function and the second loss function.

[0032] Furthermore, the first loss function The formula is as follows:

[0033] ;

[0034] ;

[0035] ;

[0036] in, and is a hyperparameter that balances the two losses, and The daytime RGB image is aligned to the text-like prototype feature, and the nighttime IR image is aligned to the text-like prototype feature. and are the number of daytime RGB images and nighttime IR images in a batch of training data, respectively. Indicates the The one-hot vector of the category label corresponding to each pedestrian image, Indicates the When the pedestrian image is a daytime RGB image, the image feature vector obtained after the trainable visual encoder is Indicates the When the pedestrian image is a night IR image, the image feature vector obtained after the trainable visual encoder is For the Pedestrian images corresponding to category labels The text-like prototype features, Represents the total number of different types of text features, Represents the similarity calculation between two features.

[0037] Furthermore, the second loss function The formula is as follows:

[0038] ;

[0039] in, and is a hyperparameter that balances the two losses, is the ID loss function, It is a three-yuan loss.

[0040] Furthermore, the visual encoder of the CLIP model adopts a dual-branch structure, and the two branch structures operate independently, one of which is used to extract features from pedestrian images that are daytime RGB images, and the other branch is used to extract features from pedestrian images that are nighttime IR images;

[0041] The first four layers of each branch structure of the visual encoder are used as modality-shared visual encoders, and the remaining layers are used as modality-private visual encoders.

[0042] The image cross-modal person re-identification system based on text clue alignment uses the visual encoder in the trained person re-identification model to extract features from the search image and the search images in the search library to obtain feature vectors. The similarity between the feature vector of the search image and the feature vectors of each search image is then calculated and sorted in descending order. The top K corresponding search images are taken as the recognition results of the search image.

[0043] The training process of the person re-ID model includes a fusion and interactive training module, a visual encoder optimization module, and a loss construction module:

[0044] The fusion and interactive training module is used to generate pedestrian description vectors based on the acquired pedestrian images, and to generate newly defined text feature vectors using the frozen CLIP model, the pre-trained text fusion network, and the pre-trained interactive network.

[0045] In the visual encoder optimization module, the visual encoder of the CLIP model is used as an image feature extractor, where the front part of the visual encoder serves as a modality-shared visual encoder and the back part serves as a modality-private visual encoder. The image feature vector obtained by passing the pedestrian image through the trainable visual encoder is used as the key and value in the cross-attention. The newly defined text feature vector is used as the query in the cross-attention. Feature exchange is performed to obtain a semantically guided feature vector, which is then input into the classifier.

[0046] The loss building module is used to build the total loss function and adjust the trainable parameters in the visual encoder and classifier.

[0047] The advantages of the image cross-modal pedestrian re-identification method and system based on text clue alignment provided by the present invention are as follows: A large model technology is used in combination with a predefined description template to obtain a pedestrian identification attribute-aware text description in a question-and-answer manner. The contour information is fine-tuned using SCHP technology (CoOP technology) to obtain a pedestrian contour-aware text feature vector to compensate for the problem that the large model cannot describe pedestrian contour information. The newly defined text feature vector is used to constrain the modality shared visual encoder, allowing the visual encoder to pay more attention to salient areas with important identification features. The text-like prototype features statistically calculated from the set of newly defined text feature vectors are used as new alignment clues. That is, a text-like prototype is obtained from a large number of newly defined text feature vectors, and images of the same type but different modalities are aligned to the text-like prototype, thereby improving the final image recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the process of the present invention;

[0049] Figure 2 Schematic diagram of generating attribute-aware text description for pedestrian identification;

[0050] Figure 3 Schematic diagram of generating pedestrian contour-aware text feature vector;

[0051] Figure 4 Schematic diagram for generating newly defined text feature vectors;

[0052] Figure 5 Schematic diagram of the training process of the visual encoder. DETAILED DESCRIPTION

[0053] The technical solutions of the present invention are described in detail below through specific embodiments. Numerous specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0054] like Figures 1 to 5 As shown in the figure, the cross-modal person re-identification method based on text clue alignment proposed in the present invention uses the visual encoder in the trained person re-identification model to extract features from the search image and the search images in the search library to obtain feature vectors. Then, the similarity between the feature vector of the search image and the feature vectors of each search image is calculated and arranged in descending order. The top K corresponding search images are taken as the recognition results of the search image.

[0055] The training process of the pedestrian re-identification model TCCN is as follows:

[0056] Generate a pedestrian description vector based on the acquired pedestrian image, and generate a newly defined text feature vector using the frozen CLIP model, the pre-trained text fusion network, and the pre-trained interaction network. The pedestrian description vector includes a pedestrian identification attribute-aware text description and a pedestrian contour-aware text feature vector.

[0057] The CLIP model's visual encoder is used as an image feature extractor, with the front part of the visual encoder serving as a modality-shared visual encoder and the back part as a modality-private visual encoder. The image feature vector obtained by passing the pedestrian image through the trainable visual encoder is used as the key and value in the cross-attention. The newly defined text feature vector is used as the query in the cross-attention. Feature exchange is performed to obtain a semantically guided feature vector, which is then input into the classifier.

[0058] Construct an overall loss function and adjust the trainable parameters in the visual encoder and classifier.

[0059] Among them, the CLIP (Contrastive Language-Image Pre-training) model is a multimodal pre-training model developed by OpenAI, which achieves cross-modal alignment of images and text through contrastive learning.

[0060] This example focuses on how to generate rich text descriptions of pedestrians from pedestrian image data and how to use these rich text descriptions for semantic hints to enhance the feature extraction capabilities of large-scale visual encoders. The generated pedestrian identification attribute-aware text descriptions and pedestrian contour-aware text feature vectors are fused to form a newly defined text feature vector with rich semantics. This newly defined text feature vector is then used to constrain the visual encoder in the person re-identification model, allowing it to focus more on salient areas with important identification features. This addresses the problem of poor retrieval performance between daytime RGB images and nighttime IR images.

[0061] In one embodiment, a pedestrian description vector is generated based on the acquired pedestrian image, and a newly defined text feature vector is generated using the frozen CLIP model, the pre-trained text fusion network, and the pre-trained interaction network, specifically (a1) to (a3):

[0062] (a1) Inputting pedestrian images and questions corresponding to predefined attribute description templates into the BLIP model, using a question-answering model to fill in the description templates with information to form attribute-aware text descriptions for pedestrian identification. BLIP (Bootstrapping Language-Image Pre-training) is a multimodal vision-language pre-training model developed by Salesforce Research. It aims to unify vision and language tasks, improve data quality, and enhance generation capabilities through bootstrapping methods.

[0063] like Figure 2 As shown in the figure, in generating pedestrian identification attribute-aware text description based on pedestrian images, we first predefine the attribute description template, such as "A {age} {gender} is {action}, wearing {upper clothes} , {lower-body clothes} and {shoe type}, The {gender} has {short or long} hair, {yes or no} The person wearing {glasses or not}." At the same time, we set the questions corresponding to the description template, such as "question1=the person is male or female?", "question2=the person age is young or teenager or adult or old?" and "question3=the person is walking or standing?". Then, the pedestrian image is input into the large model, and the intelligent question mode in the BILP model is used to fill in the pedestrian attribute information in the description template to form a pedestrian identification attribute-aware text description.

[0064] (a2) Pedestrian contour information is extracted from pedestrian images using the SCHP technique and input into the frozen CLIP model. A pedestrian contour-aware text feature vector is obtained using the improved CoOp technique. The CoOp technique is a cue learning method applied to vision-language models that aims to improve image recognition performance by automatically optimizing cue words.

[0065] Pedestrian image Input into the human semantic segmentation model SCHP to obtain pedestrian contour information, namely:

[0066] ;

[0067] in, represents dot product, Indicates the pedestrian images, for The corresponding pedestrian profile information, express The contour mask of For A matrix of all ones of the same size used to compute the complement of the mask.

[0068] like Figure 3 As shown, pedestrian profile information is obtained Finally, in order to obtain the outline-related text prompts, the improved CoOP technology is used to use a set of learnable feature vectors for each category of pedestrians to represent their text description information. , It can be expressed as: "a photo of a ",here represents the learnable feature vector, It represents the number of learnable vectors, and then the pedestrian contour information And the corresponding text description information Input to the visual encoder of the frozen CLIP model respectively and text encoders Get visual features and text features ,Right now:

[0069] ;

[0070] Then, the contrast loss in the CLIP model is used to optimize the text description information, learn text features that can express pedestrian contour information, and obtain a pedestrian contour-aware text feature vector.

[0071] The contrastive loss in the CLIP model is as follows:

[0072] ;

[0073] ;

[0074] ;

[0075] in, is the contour-aware image-text loss function, is the contour-aware text-image loss function, with the superscript It is only used to distinguish the losses referred to and has no specific meaning. 、 It only refers to the interaction between images and text, and has no specific meaning. Indicates the number of images in a mini-batch of data, Indicates calculating the similarity of two features. is the pedestrian image index, For the tentative visual features, specifically pedestrian images Visual encoder after human semantic segmentation model SCHP and frozen CLIP model and text encoders , the visual features obtained and text features .

[0076] Based on the above steps, the pedestrian contour perception text feature vector can be obtained ,in, Represents the number of pedestrian category labels, For the The category labels corresponding to the pedestrian images.

[0077] This embodiment uses large model technology and combines it with predefined description templates to obtain pedestrian identification attribute-aware text descriptions in a question-and-answer manner. It first uses SCHP technology to extract pedestrian contour information from pedestrian images, then uses improved CoOP technology to take pedestrian contour information as input, and predefines a set of learnable vectors. Then, it optimizes based on contrast loss to learn this set of vectors. Because this set of vectors is related to pedestrian contour information, it obtains pedestrian contour-aware text feature vectors to make up for the problem that the large model cannot describe pedestrian contour information.

[0078] (a3) Generate pedestrian identification attribute-aware text description and pedestrian contour-aware text feature vector based on pedestrian image and fuse them to form a newly defined text feature vector;

[0079] like Figure 4 As shown, the pedestrian identification attribute-aware text description is input into the text encoder of the frozen CLIP model. In the above example, we get the pedestrian recognition attribute perception text feature vector ; Then the pedestrian recognition attribute perception text feature vector The corresponding pedestrian contour-aware text feature vector After splicing, input it into the pre-trained text fusion network to obtain the fused text features . Transform the pedestrian contour-aware text feature vector Corresponding pedestrian image Input to the visual encoder of the frozen CLIP model In the image feature vector ; The fused text features and image feature vector Input into the pre-trained interactive network to obtain the newly defined text feature vector .

[0080] In this embodiment, the text fusion network It is composed of three fully connected layers. The text fusion network includes three fully connected layers, the interactive network Including a standard Transformer decoder with a depth of 3; that is, in the interactive network In, that is,

[0081] ;

[0082] in, For interactive networks The instance text features obtained after .

[0083] Then use residual connection to instance text features To update:

[0084] ;

[0085] in, is the updated instance text feature, that is, the newly defined text feature vector, It is a scientific parameter used to control the scaling of the residual connection. is the number of channels (number of dimensions).

[0086] The text fusion network and the interaction network refer to the execution process of the above two networks during the pre-training process. In order to ensure that the redefined semantic information accurately reflects the image information, the image-to-text loss and the text-to-image loss are adopted. On the other hand, in order to ensure that the redefined text features have discriminative ability, the identity loss (Id loss) is introduced. The pre-training loss function of this part is It can be defined as:

[0087] ;

[0088] ;

[0089] ;

[0090] ;

[0091] in, is the loss function from image to text, is the loss function from text to image, is the identity loss function, For the The image feature vector obtained by freezing the visual encoder of the CLIP model is: For the The newly defined text feature vector corresponding to the pedestrian image, Indicates calculating the similarity of two features. Indicates the The unique heat vector of the category label corresponding to each pedestrian image also corresponds to the one-hot identity label. One-hot is an encoding method that converts categorical variables into binary vectors. Represents the classifier, in this part of the process, the pre-training loss function is used Update only the model text fusion network , interactive network and classifier , in fusing text into the network , interactive network and classifier After pre-training is completed, the network is fused based on the pre-trained text , interactive network Get the newly defined text feature vector set ; The number of images in the training set represented by is ,through the above fusion method, a rich new defined text feature vector is ,generated for each pedestrian image.

[0092] In one embodiment, the visual encoder of the CLIP model is constrained according to the newly defined text feature vector to obtain a feature vector with richer recognition, specifically:

[0093] like Figure 5 As shown, the visual encoder adopts a dual-branch structure, one of which is used to extract features from daytime RGB images of pedestrians, and the other is used to extract features from nighttime IR images of pedestrians. The two branches have the same division of the modality-shared and modality-private parts. Both use the first four layers of the visual encoder as the modality-shared visual encoder, and the remaining layers as the modality-private visual encoder. Specifically, one branch is the modality-shared visual encoder Es + the modality-private visual encoder Epa, which is used for feature extraction from daytime RGB images, and the other branch is the modality-shared visual encoder Es + the modality-private visual encoder Epi, which is used for feature extraction from nighttime IR images.

[0094] The daytime RGB image is passed through the modality shared visual encoder Es + modality private visual encoder Epa to obtain the daytime image feature vector , the night IR image is passed through the modality shared visual encoder Es + modality private visual encoder Epi to obtain the night image feature vector ; When the pedestrian image is a daytime RGB image, the daytime image feature vector is converted into As the key and value in the cross attention, the newly defined text feature vector is used as the query in the cross attention to obtain the feature vector after semantic guidance; when the pedestrian image is a night IR image, the night image feature vector is used in the semantic driving module. As the key and value in the cross attention, the newly defined text feature vector is used as the query in the cross attention to obtain the semantically guided feature vector.

[0095] Since the cross attention process of daytime RGB images and nighttime IR images is the same, it can be briefly described as follows: in the semantic driving module, the newly defined text feature vector is used as the query in the cross attention, and the pedestrian image is used as the query in the cross attention. Features extracted by a trainable visual encoder As the key and value of the cross attention mechanism, its formula is as follows:

[0096] ;

[0097] The above, is the cross attention mechanism in the semantic driving module, , , Represent the query matrix, value matrix, and key matrix, namely query, key, and value. After calculation by this formula, the feature vector set after semantic guidance constraints is obtained. , Represents the number of images in the training set, To train the index of the images in the set, is the feature vector after semantic guidance.

[0098] In one embodiment, in the frozen text encoder, the frozen text fusion network , freeze the interaction network Finally, during the training process of the visual encoder, the semantically guided feature vector is input into the classifier and constrained using identity loss and ternary loss. On the other hand, the text-like prototype features are counted from the newly defined text feature vector set, and the semantically guided feature vector is brought closer to the text-like prototype features.

[0099] Generate a set of newly defined text feature vectors corresponding to all pedestrian images in the training set based on the pre-trained text fusion network and interaction network ,from The prototype feature set of different text features is counted in , is the total number of different types of text features, For category labels The prototype features of the text class are used as alignment clues to construct the first loss function, so that different modal features are aligned to the prototype features of the text class.

[0100] In order to make the features identity discriminative, conventional ID loss is used. At the same time, in order to make the extracted features show that the intra-class distance is smaller than the inter-class distance at the feature space level, the semantically guided feature vector is input into the classifier and constrained by the second loss function constructed using ID loss and ternary loss.

[0101] Among them, the first loss function The details are as follows:

[0102] ;

[0103] ;

[0104] ;

[0105] in, and is a hyperparameter that balances the two losses, and The daytime RGB image is aligned to the text-like prototype feature, and the nighttime IR image is aligned to the text-like prototype feature. and are the number of daytime RGB images and nighttime IR images in a batch of training data, respectively. Indicates the The one-hot vector of the category label corresponding to each pedestrian image, Indicates the When the pedestrian image is a daytime RGB image, the image feature vector obtained after the trainable visual encoder is Indicates the When the pedestrian image is a night IR image, the image feature vector obtained after the trainable visual encoder is For the Pedestrian images corresponding to category labels The text-like prototype features, Represents the total number of different types of text features, Represents the similarity calculation between two features.

[0106] The second loss function The details are as follows:

[0107] ;

[0108] ;

[0109] ;

[0110] ;

[0111] ;

[0112] in, and is a hyperparameter that balances the two losses, is the ID loss function, is the ternary loss, It represents the number of pictures in a batch of training data. represents the parameters of the classifier, Represents a pedestrian image It is the feature extracted by the trainable visual encoder (because the training batch data contains both daytime RGB images and nighttime IR images, and use collectively referred to as), Indicates the One-hot vectors of class labels, is the number of pedestrian images in a batch of training data, It represents the distribution probability of positive sample pairs, It represents the probability of the distribution of negative samples. , represents the positive sample pair The Euclidean distance, For the pedestrian images is the feature extracted by the trainable visual encoder, represents the negative sample pair ( The Euclidean distance, For the pedestrian images is the feature extracted by the trainable visual encoder, Indicates the Positive sample domain / set of positive samples, Indicates the Negative sample domain / set of negative samples.

[0113] Therefore, the total loss function of training is It can be expressed as:

[0114] ;

[0115] The total loss function is only used for the parameters of the visual encoder and the classifier parameters , and the cross-attention mechanism parameters , , In the inference stage, only the visual encoder trained with image data obtains the feature vector of the image for similarity calculation of the retrieval part.

[0116] This embodiment uses newly defined text feature vectors to constrain the visual encoder, allowing it to focus more on salient regions with important recognition features. It also uses the text-like prototype features statistically derived from the newly defined text feature vectors as new alignment cues. Specifically, it derives a text-like prototype from a large number of newly defined text feature vectors and aligns similar but different modality images toward this prototype, thereby improving final image recognition accuracy.

[0117] In one embodiment, the visual encoder obtained through the above training process is used to perform image retrieval, specifically:

[0118] If the pedestrian image is a daytime RGB image, the daytime RGB image is passed through the modality-shared visual encoder Es + the modality-private visual encoder Epa to obtain the daytime image feature vector. At the same time, the retrieval image in the retrieval library is passed through the modality-shared visual encoder Es + the modality-private visual encoder Epa to obtain the retrieval image feature vector. The similarity between the feature vector of the image to be retrieved and the feature vector of each retrieval image is calculated and arranged in descending order. The first K corresponding retrieval images are taken as the recognition results of the image to be retrieved;

[0119] If the pedestrian image is a night IR image, the night IR image is passed through the modal shared visual encoder Es + the modal private visual encoder Epi to obtain the night image feature vector. At the same time, the retrieval image in the retrieval library is passed through the modal shared visual encoder Es + the modal private visual encoder Epa to obtain the feature vector of the retrieval image. The similarity between the feature vector of the image to be retrieved and the feature vector of each retrieval image is calculated and arranged in descending order. The first K corresponding retrieval images are taken as the recognition results of the image to be retrieved.

[0120] In summary, the main solutions of this embodiment are:

[0121] First, the BLIP model is used to generate pedestrian identification attribute-aware text descriptions based on predefined pedestrian attribute templates. Then, based on the pedestrian contour information and a set of additional learnable feature vectors, the frozen CLIP model is used to obtain visual features and text features according to the improved CoOP technology. At the same time, the contrast loss in the CLIP model is used to optimize the set of learnable feature vectors introduced to obtain the pedestrian contour-aware text feature vector.

[0122] Then freeze the CLIP model and use the obtained pedestrian recognition attribute perception text description and pedestrian contour perception text feature vector to fusion network. , interactive network Pre-training is performed, and then the pedestrian recognition attribute perception text description and pedestrian contour perception text feature vector are re-passed through the frozen CLIP model and the pre-trained text fusion network , interactive network Processing is performed to obtain a newly defined text feature vector.

[0123] Finally, freeze the text encoder, freeze the pre-trained text fusion network, freeze the pre-trained interaction network, and train the visual encoder based on the pedestrian image and the newly defined text feature vector. Specifically, the image feature vector obtained by the trainable visual encoder of the pedestrian image is used as the key and value in the cross-attention, and the newly defined text feature vector is used as the query in the cross-attention to perform feature exchange. In this process, the visual encoder is optimized by the total loss function to obtain the trained visual encoder. The trained visual encoder will be used in the pedestrian re-identification execution process to obtain the recognition result of the image to be retrieved.

[0124] In order to verify the advantages of the visual encoder obtained in this embodiment, the following experiments were performed:

[0125] Experimental data:

[0126] Experiments are conducted on two publicly available VI-ReID (visible-infrared) datasets:

[0127] SYSU-MM01: Currently one of the most challenging large-scale datasets designed for cross-modal person re-identification, it contains 287,628 visible (RGB) images and 15,792 infrared (IR) images from 491 individuals. Following a standard protocol, the training set includes 22,258 visible images and 11,909 IR images from 395 individuals, while the test set contains images of 96 individuals. The data was captured using four visible cameras and two IR cameras in indoor and outdoor environments. During testing, IR images serve as query images, and visible images serve as retrieval images in the retrieval gallery. The dataset includes two test modes: all-search, which utilizes all visible images from both indoor and outdoor environments; and indoor-search, which only considers images from two indoor cameras. The dataset offers single-shot and multi-shot gallery configurations, with either one or ten visible images selected for each identity. This study comprehensively evaluates performance under all these configurations.

[0128] RegDB: is a compact VI-ReID dataset containing 8,240 images of 412 pedestrians. Each individual has 10 visible images and 10 infrared images, all captured by the same camera, covering both modalities. In this experiment, 2,060 visible images (i.e., RGB images) and 2,060 infrared images (i.e., IR images) were randomly selected from 206 pedestrians as the training set, and the remaining images constituted the test set. This dataset provides two evaluation scenarios: retrieval from visible images to infrared images and retrieval from infrared images to visible images. It is important to note that the dataset was randomly split into training and test sets 10 times in the experiment, and the average results were used to report the final performance to ensure fairness and accuracy of the evaluation.

[0129] Evaluation Metrics: This experiment uses rank-K (K=1, 10, 20) and mean average precision (mAP) to evaluate the performance of the algorithm. The rank-K metric assesses the percentage of successful searches in which at least one relevant individual is retrieved among the top K results. The mAP metric analyzes the model's ability to accurately predict the order.

[0130] Benchmark model comparison and experimental analysis:

[0131] Table 1 shows the overall performance comparison on SYSU-MM01. Performance is expressed as Rank-k accuracy (%) and mAP (%). The best results are shown in bold.

[0132] Table 1

[0133]

[0134] Table 2 shows the overall performance comparison results on RegDB. Performance is expressed as Rank-k accuracy (%) and mAP (%). The best results are shown in bold.

[0135] Table 2

[0136]

[0137] In Tables 1 and 2, the literature sources are: TPAMI: IEEE Transactions on Pattern Analysis and Machine Intelligence; ICCV: IEEE International Conference on Computer Vision; CVPR: IEEE Conference on Computer Vision and Pattern Recognition; TIP: IEEE Transactions on Image Processing; TIFS: IEEE Transactions on Information Forensics and Security; ICMR: ACM International Conference on Multimedia Retrieval; TMM: IEEE Transactions on Multimedia (a top journal in the field of multimedia (non-conference)); ICME: IEEE International Conference on Multimedia and Expo.

[0138] Methods: AGW: Attention Generalized mean pooling with Weighted triplet loss; CAJ: Cross-Attention Joint Embedding for Person Re-Identification; MPANet: joint Modality and Pattern Alignment Network; SPOT: structure-aware positional transformer; DEEN: diverse embedding expansion network; CSDN: CLIP-driven Semantic Discovery Network; MIP; Modality-aware and Instance-aware Visual Prompts (MIP) network; J-ReID: join visible-infrared alignment (VIA) and visual-textual reasoning network. MUN: Modal Uncertainty-Constrained Network, SSRR: Structural Semantic Representation Reconstruction: Structural Semantic Representation,

[0139] Combining Table 1 and Table 2, the following is explained:

[0140] (1) First, several traditional modality-invariant methods, such as AGW, CAJ, MPANET, DEEN, MUN, and SSRR, were compared. These methods mainly rely on the relationship between images to extract modality-invariant features without incorporating text information in the images. Based on this finding, the present embodiment outperforms the best MUN method. For example, in the full search task of the SYSU-MM01 dataset, the pedestrian re-identification method of the present embodiment improved the mAP by approximately 2.41% and the Rank-1 score by approximately 2.32%. In order to verify the adaptability of the present embodiment in different scenarios, a comparison was conducted in the indoor search scenario of the same dataset. In this scenario, the present embodiment improved the mAP by approximately 3.57% and the Rank-1 score by approximately 4.93%. In order to further evaluate the effectiveness and robustness of the method, tests were conducted on the RegDB dataset, comparing the performance in visible to infrared and infrared to visible retrieval scenarios. In the visible to infrared and infrared to visible retrieval scenarios, compared with the best performing SSRR method, the present embodiment reduced the mAP by approximately 3.28% and 4.99%, respectively. Furthermore, in infrared-to-visible light scenarios, the Rank-1 score is approximately 0.52% lower than that of SSRR. However, in visible-to-infrared scenarios, this embodiment outperforms the SSRR method in Rank-1 score, improving by approximately 0.87%. These experimental results demonstrate that incorporating rich textual information can effectively guide the visual encoder of the CLIP model (large language model), thereby enhancing the extraction of modality-invariant features. Furthermore, this embodiment fails to outperform other algorithms in certain scenarios, suggesting that the introduced textual information may be affected by specific contextual factors, which may affect the final results.

[0141] (2) To further verify the effectiveness of the proposed method, this embodiment was compared with several language model-based methods, such as MIP, J-ReID, and CSDN. These methods use text information from images to mine modality-invariant features for identity recognition. Compared with the best-performing CSDN method, it was found that on the SYSU-MM01 dataset, for the ALL Search and Indoor Search tasks, the method of this embodiment improved mAP and Rank-1 by approximately 3.21% and 1.82%, and 0.07% and 0.73%, respectively. On the RegDB dataset, for the Visible to Infrared and Infrared to Visible retrieval tasks, the method of this embodiment improved mAP and Rank-1 by approximately 0.42% and 0.27%, and 0.22% and 1.28%, respectively. Overall, the method of this embodiment demonstrated superior performance in different scenarios and datasets, which was reflected in all evaluation indicators. Compared to the CSDN method, this example incorporates richer language cues and introduces a more efficient method for constructing textual cues, enabling it to better focus on pedestrian recognition information, thereby further improving performance. This demonstrates that richer semantic textual cues can effectively guide the visual encoder of a large model, helping it to better extract modality-invariant features of pedestrians, thereby improving performance metrics.

[0142] (3) In addition, it can be observed from Tables 1 and 2 that some traditional modality-invariant methods, such as MUN, outperform the language model-based method MIP that introduces textual cues, but still underperform the CSDN and J-ReID methods. This phenomenon is attributed to the fact that the design of textual cues in the MIP method is still imperfect, making it difficult to effectively guide the visual encoder to fully utilize its powerful feature extraction capabilities. Therefore, the key to successfully applying large language model-based visual encoders for cross-modal person re-ID tasks lies in how to generate effective textual cues and how to construct appropriate cues to effectively constrain the visual encoder.

[0143] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. The image cross-modal pedestrian re-identification method based on text clue alignment is characterized by: The visual encoder in the trained person re-identification model is used to extract features from the image to be retrieved and the retrieval images in the retrieval library to obtain feature vectors. The similarity between the feature vector of the image to be retrieved and the feature vectors of each retrieval image is calculated and arranged in descending order. The first K corresponding retrieval images are taken as the recognition results of the image to be retrieved. The training process of the person re-identification model is as follows: Generate a pedestrian description vector based on the acquired pedestrian image, and use the frozen CLIP model, the pre-trained text fusion network and the pre-trained interaction network to generate a newly defined text feature vector; The CLIP model's visual encoder is used as an image feature extractor, with the front part of the visual encoder serving as a modality-shared visual encoder and the back part as a modality-private visual encoder. The image feature vector obtained by passing the pedestrian image through the trainable visual encoder is used as the key and value in the cross-attention. The newly defined text feature vector is used as the query in the cross-attention. Feature exchange is performed to obtain a semantically guided feature vector, which is then input into the classifier. Construct an overall loss function and adjust the trainable parameters in the visual encoder and classifier.

2. The image cross-modal person re-identification method based on text clue alignment according to claim 1 is characterized in that In generating a pedestrian description vector based on the acquired pedestrian image, the pedestrian description vector includes a pedestrian identification attribute-aware text description and a pedestrian contour-aware text feature vector. The pedestrian description vector generation process is as follows: The pedestrian image and the questions corresponding to the predefined attribute description template are input into the BLIP model, and the description template is filled with information using the question-answering model to form a pedestrian identification attribute perception text description; The SCHP technique is used to extract pedestrian contour information from pedestrian images and input it into the frozen CLIP model. The improved CoOP technique is used to obtain the pedestrian contour-aware text feature vector.

3. The image cross-modal person re-identification method based on text clue alignment according to claim 2 is characterized in that The improved CoOP technology is used to obtain the pedestrian contour perception text feature vector, specifically: A set of learnable feature vectors is set for each category of pedestrian to represent the outline text description information, and the pedestrian outline information and the corresponding outline text description information are input into the frozen CLIP model respectively; The contrast loss in the CLIP model is used to optimize the contour text description information, and text features that can express pedestrian contour information are learned, thereby obtaining a pedestrian contour perception text feature vector.

4. The image cross-modal person re-identification method based on text clue alignment according to claim 2 is characterized in that The generation process of the newly defined text feature vector is specifically as follows: Input the pedestrian identification attribute-aware text description into the text encoder of the frozen CLIP model to obtain the pedestrian identification attribute-aware text feature vector; Then, the pedestrian identification attribute perception text feature vector and the pedestrian contour perception text feature vector are concatenated and input into the pre-trained text fusion network to obtain the fused text features; Input the pedestrian image corresponding to the pedestrian contour perception text feature vector into the visual encoder of the frozen CLIP model to obtain the image feature vector; The fused text feature and image feature vector are input into the pre-trained interaction network to obtain a newly defined text feature vector.

5. The image cross-modal person re-identification method based on text clue alignment according to claim 4 is characterized in that The text fusion network includes three fully connected layers, and the interaction network includes a standard Transformer decoder with a depth of 3; The two networks construct the following pre-training loss function in pre-training : ； in, is the loss function from image to text, is the loss function from text to image, is the identity loss function.

6. The image cross-modal person re-identification method based on text clue alignment according to claim 1 is characterized in that The total loss function is constructed as follows: Generate a set of newly defined text feature vectors corresponding to all pedestrian images in the training set based on the pre-trained text fusion network and interaction network; Counting the prototype features of different types of text features from the newly defined text feature vector set; The first loss function is constructed by taking the prototype features of the class text as the alignment clue, so that different modal features are aligned with the prototype features of the class text; The semantically guided feature vector is input into the classifier and constrained by a second loss function constructed using ID loss and ternary loss, so that the semantically guided feature vector has identity discrimination and the intra-class distance is smaller than the inter-class distance at the feature space level. The total loss function is obtained by adding the first loss function and the second loss function.

7. The method for cross-modal person re-identification based on text clue alignment according to claim 6, characterized in that: The first loss function The formula is as follows: ； ； ； in, and is a hyperparameter that balances the two losses, and The daytime RGB image is aligned to the text-like prototype feature, and the nighttime IR image is aligned to the text-like prototype feature. and are the number of daytime RGB images and nighttime IR images in a batch of training data, respectively. Indicates the The one-hot vector of the category label corresponding to each pedestrian image, Indicates the When the pedestrian image is a daytime RGB image, the image feature vector obtained after the trainable visual encoder is Indicates the When the pedestrian image is a night IR image, the image feature vector obtained after the trainable visual encoder is For the Pedestrian images corresponding to category labels The text-like prototype features, Represents the total number of different types of text features, Represents the similarity calculation between two features.

8. The method for cross-modal person re-identification based on text clue alignment according to claim 6, characterized in that: The second loss function The formula is as follows: ； in, and is a hyperparameter that balances the two losses, is the ID loss function, It is a three-yuan loss.

9. The method for cross-modal person re-identification based on text clue alignment according to claim 1, characterized in that: The visual encoder of the CLIP model adopts a dual-branch structure. The two branch structures operate independently. One branch structure is used to extract features from pedestrian images that are daytime RGB images, and the other branch structure is used to extract features from pedestrian images that are nighttime IR images. The first four layers of each branch structure of the visual encoder are used as modality-shared visual encoders, and the remaining layers are used as modality-private visual encoders.

10. Image cross-modal person re-identification system based on text clue alignment, characterized by: The visual encoder in the trained person re-identification model is used to extract features from the image to be retrieved and the retrieval images in the retrieval library to obtain feature vectors. The similarity between the feature vector of the image to be retrieved and the feature vectors of each retrieval image is then calculated and sorted in descending order. The top K corresponding retrieval images are taken as the recognition results of the image to be retrieved. The training process of the person re-ID model includes a fusion and interactive training module, a visual encoder optimization module, and a loss construction module: The fusion and interactive training module is used to generate pedestrian description vectors based on the acquired pedestrian images, and to generate newly defined text feature vectors using the frozen CLIP model, the pre-trained text fusion network, and the pre-trained interactive network. In the visual encoder optimization module, the visual encoder of the CLIP model is used as an image feature extractor, where the front part of the visual encoder serves as a modality-shared visual encoder and the back part serves as a modality-private visual encoder. The image feature vector obtained by passing the pedestrian image through the trainable visual encoder is used as the key and value in the cross-attention. The newly defined text feature vector is used as the query in the cross-attention. Feature exchange is performed to obtain a semantically guided feature vector, which is then input into the classifier. The loss building module is used to build the total loss function and adjust the trainable parameters in the visual encoder and classifier.

Images