An unsupervised pedestrian re-identification method and system from text to image

Abstract

The present application relates to the field of pedestrian re-identification, and discloses a kind of unsupervised text-to-image pedestrian re-identification method and system, comprising: constructing unsupervised dataset;Image is divided into regions, each image region and text are encoded respectively, and the text with high cosine similarity is selected for each image region to be merged as new text, to obtain new dataset;Using new dataset to train ReID model;Based on the pre-trained ReID model, reverse search is carried out, and the screened dataset is obtained;Using screened dataset to train ReID model, to obtain trained ReID model;The text to be identified is input into trained ReID model to obtain Top-k similar images.The present application avoids the dependence on large language model, the search text can focus on different body regions, form more accurate and comprehensive description, and through reverse search, the noise and redundancy of text are reduced, and the accuracy of text and image matching is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of pedestrian re-identification, and particularly relates to an unsupervised text-to-image pedestrian re-identification method and system. BACKGROUND

[0002] With the continuous development of computer vision, text-to-image pedestrian re-identification (TIReID) has become an important research topic and is widely used in security monitoring, smart cities, and personnel tracking. The goal of the TIReID task is to match pedestrian images through natural language descriptions, overcoming the problem of traditional pedestrian re-identification methods relying only on image features, and promoting the research of cross-modal information fusion. However, the practical application of TIReID technology faces many challenges, especially in data acquisition, labeling costs, and privacy protection.

[0003] Traditional TIReID methods rely on labeled datasets, with each image matched with corresponding text descriptions and unique pedestrian ID labels, relying on deep learning models such as convolutional neural networks (CNN) to extract image features and perform matching. Although these methods have shown high accuracy in experimental environments, in practical applications, there is a problem of difficulty in obtaining large-scale labeled datasets. Especially in large-scale video data such as surveillance videos, manual labeling is costly and prone to mislabeling and missing labeling, which makes it difficult for existing supervised learning methods to be directly applied to real-world scenarios. In addition, existing TIReID datasets have significant bias in the labeling process, resulting in significant inconsistencies in the descriptions of the same pedestrian between different annotators, which severely affects the generalization ability of the model. Label noise in the dataset also makes it prone to overfitting during training, reducing the model's effectiveness in real-world environments.

[0004] In order to solve these problems, unsupervised learning methods have gradually gained attention in recent years. Some unsupervised TIReID methods attempt to generate pseudo-labels to compensate for the lack of labeled data, such as generating image descriptions through large language models (LLMs) or visual question answering (VQA) techniques for training. However, these methods still rely on external models, and the generated pseudo-labels lack accuracy and context information, especially in scenarios with high privacy protection requirements, using these methods may pose a risk of privacy leakage. SUMMARY

[0005] The present application aims to overcome the deficiencies in the prior art, and provide a method and system for unsupervised text-to-image pedestrian re-identification, which is realized based on an unsupervised data set, avoids dependence on large language model annotation data, focuses the search text on different body regions by dividing the pedestrian image into regions, forms a more accurate and comprehensive description, reduces text noise and redundancy through reverse search, and improves the accuracy of text and image matching.

[0006] The present application provides the following technical solutions:

[0007] In a first aspect, a method for unsupervised text-to-image pedestrian re-identification is provided, comprising:

[0008] An unsupervised data set is constructed, which includes multiple groups of image-text pairs;

[0009] The images in the image-text pairs are divided into regions, and then each image region and text are encoded, the text with high cosine similarity is selected for each image region, and the selected texts are combined as new texts to obtain a new data set;

[0010] The new data set is used to train a ReID model to obtain a pre-trained ReID model;

[0011] Based on the pre-trained ReID model, the most similar regions of each text in the new data set in the corresponding image are selected from the image-text pairs with consistent results in the new data set to obtain a screened data set;

[0012] The screened data set is used to train the ReID model to obtain a trained ReID model;

[0013] The text to be identified is input into the trained ReID model to obtain Top-k similar images, thereby realizing unsupervised text-to-image pedestrian re-identification.

[0014] Further, the construction of the unsupervised data set comprises:

[0015] An unlabeled training set is obtained and a corpus , the unlabeled training set includes N pedestrian images, and the corpus includes M pedestrian description texts;

[0016] Suitable texts are mined for each image x from the corpus , and an image-text pair (x, T i ) is generated, wherein each text T i is composed of one or more texts in the corpus .

[0017] Further, the image in the image-text pair is divided into regions, comprising:

[0018] Divide the image into four image regions R according to a preset ratio k , k = 1, 2, 3, 4, respectively corresponding to the head region, the upper body region, the lower body region and the shoe region;

[0019] Wherein, R represents the real space, H represents the image height, W represents the image width, and C represents the RGB channel number.

[0020] Further, the image regions and the text are encoded respectively, comprising:

[0021] For each image region R k , the visual Transformer of CLIP is used to extract features, and each image region is divided into image blocks to generate a token sequence ; wherein, represents the CLS token feature vector corresponding to the kth image region, and v represents that the feature vector is a visual feature, respectively represent the feature vectors of the 1st,…, N k image blocks in the kth image region, and N k represents the maximum image block sequence number obtained after the kth image region is divided into blocks, , H k and W k respectively represent the height and width of the image region R k .

[0022] For the text t, the CLIP text encoder is used to encode into a sequence by Byte Pair Encoding, and input into the Transformer model to capture the context relationship; wherein, SOS represents the sequence start token, w1, w2,…, w m respectively represent the 1st, 2nd,…, mth text subword embedding vectors, and EOS represents the sequence end token.

[0023] Further, the text with high cosine similarity is selected for each image region, and the new text is merged, comprising:

[0024] Calculate the cosine similarity between the image region R k and the text t, and the formula is as follows:

[0025] ;

[0026] Wherein, EOS token feature vector representing text t, represents the CLS token feature vector corresponding to the k-th image region;

[0027] For each image region, select the top n texts with high cosine similarity and concatenate them into a comprehensive description text T u :

[0028] ;

[0029] in, They represent the u-th text corresponding to image regions R1, R2, R3, and R4 respectively, and n represents the number of texts retained in each image region sorted from high to low according to cosine similarity.

[0030] Furthermore, the use of a new dataset to train a ReID model includes:

[0031] Input the image-text pairs in the new dataset D1 into the ReID model, map the image and text features into the same embedding space through cross-modal contrastive learning, and output the InfoNCE loss function value;

[0032] When the InfoNCE loss function value decreases by less than 1e-4 for five consecutive training cycles on the validation set, it is considered to have converged and the pre-trained ReID model is obtained.

[0033] Furthermore, the pre-trained ReID model selects image-text pairs whose most similar regions of each text in the new dataset in the corresponding image are consistent with the new dataset results, including:

[0034] For the image region R k The corresponding u-th text , use the pre-trained ReID model to calculate the text and image regions R j The cosine similarity between , where j = 1, 2, 3, 4, R j For R k The same four physical regions are only distinguished by the index variable j during the traversal calculation process;

[0035] Normalize the similarity scores of each image region to obtain a new normalized score ;

[0036] if Only one image region is matched, and the calculation Corresponding to each area of ​​the image The similarity score of , the region index y with the highest similarity is:

[0037] ;

[0038] wherein, denotes the normalized score the value of j when the maximum value is taken; if the region index y corresponding to the highest similarity corresponds to the kth image region, then the value of y is retained , otherwise it is filtered as noise;

[0039] If appears in multiple image regions, a deduplication operation is performed.

[0040] Further, the deduplication operation includes:

[0041] Calculate the comprehensive score in each image region , the formula is as follows:

[0042] ;

[0043] wherein, is a weight coefficient, and n represents the number of texts retained in each image region according to the cosine similarity from high to low;

[0044] Keep in the image region with the highest comprehensive score.

[0045] Further, the method further includes testing the trained ReID model:

[0046] Input the text in the test set into the trained ReID model, calculate the cosine similarity of the text with all image regions in the test set, and output a candidate image list sorted by cosine similarity from high to low; adopt Rank-k0 index as an evaluation standard to evaluate the retrieval performance of the model.

[0047] In a second aspect, an unsupervised text-to-image pedestrian re-identification system is provided, comprising:

[0048] A data set construction module is configured to construct an unsupervised data set, wherein the unsupervised data set includes multiple groups of image-text pairs.

[0049] A data recombination module is configured to divide the images in the image-text pairs into regions, then encode each image region and text respectively, select texts with high cosine similarity for each image region, and combine them as new texts to obtain a new data set.

[0050] A first model training module is configured to train an ReID model using the new data set to obtain a pre-trained ReID model.

[0051] The data screening module is configured to select image-text pairs in which the most similar region of each text in the new data set in the corresponding image is consistent with the new data set result based on the pre-trained ReID model, and obtain a screened data set.

[0052] The second model training module is configured to train the ReID model using the screened data set, and obtain a trained ReID model.

[0053] The recognition module is configured to input the text to be recognized into the trained ReID model, and obtain a candidate image list, thereby realizing unsupervised text-to-image pedestrian re-identification.

[0054] Compared with the prior art, the present application has the following advantages:

[0055] (1) The present application is based on the construction of an unsupervised data set, which can generate accurate descriptions for unlabeled images without manual annotation or external models, thereby avoiding dependence on large language model annotation data and improving the adaptability and generalization ability of the model in actual scenarios. In addition, the present application uses the text description retrieved from the existing corpus as a supervision signal, and matches it with the image features, thereby realizing more flexible and efficient pedestrian re-identification.

[0056] (2) The present application divides the image in the image-text pair into regions, encodes each image region and text, selects texts with high cosine similarity for each image region, and merges them as new texts to obtain a new data set. The ReID model is trained using the new data set to obtain a pre-trained ReID model. By using a partition retrieval strategy based on human body parts, different regions of the pedestrian image are segmented and region-specific attributes are extracted, thereby alleviating the misalignment problem in direct image-to-text matching and improving retrieval performance through more detailed region retrieval.

[0057] (3) The present application selects image-text pairs in which the most similar region of each text in the new data set in the corresponding image is consistent with the new data set result based on the pre-trained ReID model, and obtains a screened data set. The ReID model is trained using the screened data set to obtain a trained ReID model. Through reverse retrieval, the noise and redundancy of the text are reduced, and the accuracy of text-image matching is improved. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 is a flowchart of the unsupervised text-to-image pedestrian re-identification method in the embodiments of the present application. DETAILED DESCRIPTION

[0059] The application will be further described below with reference to the drawings. The following examples are only used to more clearly illustrate the technical solutions of the application, and cannot be used to limit the protection scope of the application.

[0060] The term "and / or", only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " generally represents that the front and rear associated objects are in an "or" relationship.

[0061] Example 1

[0062] As Figure 1 shown, the embodiment provides an unsupervised text-to-image pedestrian re-identification method, and the steps are as follows:

[0063] Step 1, constructing an unsupervised data set, the unsupervised data set comprising a plurality of image-text pairs.

[0064] The constructing of the unsupervised data set comprises:

[0065] obtaining an unlabeled training set and a corpus , the unlabeled training set comprising N pedestrian images, and the corpus comprising M pieces of pedestrian description text;

[0066] mining appropriate text for each image x from the corpus and generating an image-text pair (x, T i ), wherein each text T i is composed of one or more texts in the corpus . For each pedestrian image, multiple description texts can be generated to form multiple image-text pairs.

[0067] Step 2, dividing the image in the image-text pair into regions, then encoding each image region and text respectively, selecting a text with high cosine similarity for each image region, merging as a new text, and obtaining a new data set D1.

[0068] Step 2.1, dividing the image in the image-text pair into regions, specifically comprising:

[0069] dividing the image into four image regions R k according to a preset ratio, k=1, 2, 3, 4, corresponding to a head region, an upper body region, a lower body region, and a shoe region respectively; wherein R represents a real number space, H represents image height, W represents image width, and C represents RGB channel number.

[0070] Step 2.2, encoding each image region, specifically including:

[0071] For each image region R k , a visual Transformer (ViT) of a contrastive language-image pre-training model (CLIP) is used to extract features, and each image region is divided into image blocks of size to generate a token sequence ; wherein represents the CLS token feature vector corresponding to the kth image region, v represents that the feature vector is a visual feature, represents the feature vector of the 1st,…, N k th image block in the kth image region, N k represents the maximum image block sequence number obtained after the kth image region is divided into blocks, , H k and W k represent the height and width of the image region R k , respectively. These tokens model the spatial relationship through the Transformer, and finally represent the feature of the CLS token , which is projected into a shared image-text embedding space.

[0072] Step 2.3, encoding the text, specifically including:

[0073] For text t, a CLIP text encoder is used to obtain a sequence through byte pair encoding (BPE), and input into a Transformer model to capture context relationships; wherein SOS represents a sequence start token, w1, w2,…, w m represent the 1st, 2nd,…, mth text subword embedding vectors, respectively, and EOS represents a sequence end token. Finally, the output corresponding to the EOS token is projected into a joint embedding space as a global text feature representation.

[0074] Step 2.4, selecting text with high cosine similarity for each image region, merging as a new text, specifically including:

[0075] Calculate the cosine similarity between image region R k and text t, the formula is as follows:

[0076] ;

[0077] wherein represents the EOS token feature vector of the text t, represents the CLS token feature vector corresponding to the kth image region;

[0078] For each image region, select the top n texts with high cosine similarity and concatenate them into a comprehensive description text T u :

[0079] ;

[0080] in, They represent the u-th text corresponding to image regions R1, R2, R3, and R4 respectively, and n represents the number of texts retained in each image region sorted from high to low according to cosine similarity.

[0081] Step 3: Use the new dataset to train the ReID model until the model converges to obtain the pre-trained ReID model.

[0082] The new dataset is used to train the ReID model, including:

[0083] The image-text pairs in the new dataset D1 are input into the ReID model, and the image and text features are mapped into the same embedding space through cross-modal contrastive learning, and the information noise contrast estimation (InfoNCE) loss function value is output;

[0084] When the InfoNCE loss function value decreases by less than 1e-4 for five consecutive training cycles on the validation set, it is considered to have converged and the pre-trained ReID model is obtained.

[0085] Step 4: Based on the pre-trained ReID model, select the image-text pairs whose most similar regions of each text in the corresponding image are consistent with the results of the new dataset to obtain the filtered dataset D2. The specific steps are as follows:

[0086] For the image region R k The corresponding u-th text , use the pre-trained ReID model to calculate the text and image regions R j The cosine similarity between , where j = 1, 2, 3, 4, R j For R k The same four physical regions are only distinguished by the index variable j during the traversal calculation process;

[0087] Since similarity scores tend to be concentrated in a smaller range when matching text descriptions in a large corpus, the similarity scores of each image region are normalized to obtain a new normalized score ;

[0088] if Only one image region is matched, and the calculation Corresponding to each area of ​​the image The similarity score of the region with the highest similarity score is y:

[0089]

[0090] wherein, the value of j when the normalized score is maximum; if the region index y with the highest similarity corresponds to the kth image region, then is retained, otherwise it is filtered as noise;

[0091] If appears in multiple image regions, a deduplication operation is performed. The deduplication operation is as follows:

[0092] Calculate the comprehensive score of each image region , the formula is as follows:

[0093]

[0094] wherein, is a weight coefficient, and n represents the number of texts retained in each image region according to the cosine similarity from high to low;

[0095] Keep in the image region with the highest comprehensive score.

[0096] Step 5, use the screened data set D2 to train the ReID model until the model converges, and obtain the trained ReID model.

[0097] Step 6, test the trained ReID model, specifically:

[0098] Input the texts in the test set into the trained ReID model, calculate the cosine similarity between the texts and all image regions in the test set, and output a candidate image list sorted by cosine similarity from high to low; use the Rank-k0 index as an evaluation standard to evaluate the retrieval performance of the model. The Rank-k0 index represents the percentage of correct matching results appearing in the top k0 (k0 = 1, 5, 10) retrieval results, and the higher the value, the better the model retrieval performance.

[0099] Step 7, input the text to be identified into the trained ReID model, and perform real-time comparison between the encoded text and the pedestrian image library to be searched, and output the Top-k similar images, realizing unsupervised text-to-image pedestrian re-identification.

[0100] Embodiment 2

[0101] ​​This embodiment adopts the method of Example 1 to perform unsupervised text-to-image pedestrian re-identification, and the steps are as follows:

[0102] Step 1: Take RSTPReid as the image source and CUHK-PEDES and ICFG-PEDES as the text source together to construct an unsupervised dataset without relying on pedestrian ID information.

[0103] Table 1 is the statistical data of the training set and test set of the above three mainstream ReID datasets. Specifically, the LLMs are used to divide each long text into 2-5 shorter descriptions, each of which focuses on a specific attribute (such as color or clothing type). Each piece of text contains 5-14 words. In order to minimize gender bias, the subject expression in the text is standardized, and gender-specific terms (such as “He” or “She”) are replaced with “The person”. At the same time, with the help of a deduplication algorithm, redundant text descriptions are removed. Finally, a more refined and diverse corpus is obtained.

[0104] Table 1 Dataset

[0105]

[0106] Step 2: Divide the image into four image regions according to the ratio of 2:3:3:2, which correspond to the head region, upper body region, lower body region and shoe region respectively. Take the ViT-B / 16 model of CLIP as the image encoder and the Transformer in CLIP as the text encoder. Use the method of step 2 in Example 1 to encode the image regions and the text. Then select the top n texts with high cosine similarity for each image region and merge them as new texts to obtain a new dataset D1. In this embodiment, n = 100, and after merging, each image matches 100 long text descriptions.

[0107] Step 3: Use the new dataset D1 to select the top q texts for each image to train the ReID model until the model converges, and obtain a pre-trained ReID model.

[0108] In this embodiment, the ReID model uses a robust dual embedding model (RDE), and its image-text encoder is consistent with the CLIP model. The number of paired texts for training each image is set to 3. To achieve unsupervised training and avoid relying on pedestrian ID, the InfoNCE loss is used instead of the TAL loss in RDE. The image is adjusted to 384x128 pixels, and the input text is limited to 77 tokens. The model uses the Adam optimizer with an initial learning rate of , and is trained for 60 cycles with cosine decay.

[0109] Step 4, based on the pre-trained ReID model, the most similar region of each text in the corresponding image in the new data set is selected as the image-text pair consistent with the new data set result, and a screened data set D2 is obtained.

[0110] In this embodiment, the weight coefficient = 0.5 is used to balance the similarity score and the ranking-based selection. For example, region two and region three of a certain picture match to the same text. According to the formula:

[0111] ;

[0112] The similarity score of this text with region two is 0.7, the ranking is 20, the corresponding ranking score is 0.8, and the comprehensive score is 0.7 + 0.5 * 0.8 = 1.1; the similarity score with region three is 0.67, the ranking is 2, the corresponding ranking score is 0.98, and the comprehensive score is 0.67 + 0.5 * 0.98 = 1.16 > 1.1. Therefore, this text will be kept in region three.

[0113] Step 5, the screened data set D2 is used to train the ReID model until the model converges, and a trained ReID model is obtained.

[0114] Step 6, test the trained ReID model: input the text in the test set into the trained ReID model, calculate the cosine similarity of the text with all image regions in the test set, and output a candidate image list sorted by cosine similarity from high to low; use the Rank-k0 (k0 = 1, 5, 10) index as the evaluation standard to evaluate the retrieval performance of the model.

[0115] This embodiment compares the method of the present application with 10 published supervised methods including Dual Path, SSAN, IRRA, etc. on the basis of the unsupervised data set in Table 1, and simultaneously, the CLIP is used as the baseline model to construct the unsupervised data generation path. The results are as shown in Table 2.

[0116] Table 2 Comparison of experimental results

[0117]

[0118] As can be seen from Table 2, although there is a performance gap between the method of the present application and the supervised method, the present method significantly reduces the dependence on data annotation, reduces the training cost and privacy risk. In addition, the present method is significantly better than the baseline model under the unsupervised setting, and the Rank-1 is increased by 7.65%, 11.52% and 13.25% respectively on the three data sets. These results show that even in the scene without relying on LLMs or external detection models, the present method still has significant retrieval accuracy.

[0119] This embodiment is implemented on an NVIDIA RTX 4090 GPU with 24GB of video memory. The implementation of commands and other operations are well-known Linux operating techniques and will not be described in detail here.

[0120] Example 3

[0121] Based on the same inventive concept as Example 1, this embodiment provides an unsupervised text-to-image person re-identification system, including:

[0122] A dataset construction module, configured to construct an unsupervised dataset, wherein the unsupervised dataset includes multiple sets of image-text pairs;

[0123] The data reorganization module is used to divide the image in the image-text pair into regions, then encode each image region and text separately, select the text with high cosine similarity for each image region, merge them as the new text, and obtain a new dataset;

[0124] The first model training module is used to train the ReID model using a new dataset to obtain a pre-trained ReID model;

[0125] The data screening module is used to select the image-text pairs whose most similar regions of each text in the corresponding image in the new dataset are consistent with the results of the new dataset based on the pre-trained ReID model, thereby obtaining the filtered dataset;

[0126] The second model training module is used to train the ReID model using the filtered data set to obtain a trained ReID model;

[0127] The recognition module is used to input the text to be recognized into the trained ReID model to obtain a list of candidate images and realize unsupervised text-to-image pedestrian re-identification.

[0128] The specific functional implementation of each of the above modules can be found in the relevant content of the method in Example 1 and will not be elaborated on here.

[0129] The unsupervised text-to-image pedestrian re-identification method and system provided by the application are realized based on the construction of an unsupervised data set, can generate accurate descriptions for unlabeled images without manual annotation or external models, can avoid dependence on large language model annotation data, and can improve the adaptability and generalization ability of the model in actual scenes. In addition, the application uses the text description retrieved from the existing corpus as a supervision signal, and matches the image features to realize more flexible and efficient pedestrian re-identification. The application divides the image in the image-text pair into regions, then encodes each image region and text, selects the text with high cosine similarity for each image region, merges the selected texts as new texts, obtains a new data set, trains a ReID model using the new data set to obtain a pre-trained ReID model, uses a human body part-based partition retrieval strategy to segment different regions of the pedestrian image and extract region-specific attributes, thereby relieving the misalignment problem in direct image-to-text matching, and improving the retrieval performance through more detailed region retrieval. The application selects the image-text pairs in which the most similar region of each text in the new data set in the corresponding image is consistent with the result of the new data set based on the pre-trained ReID model, obtains a screened data set, trains a ReID model using the screened data set to obtain a trained ReID model, reduces the noise and redundancy of the text through reverse retrieval, and improves the accuracy of text-image matching.

[0130] Those skilled in the art will appreciate that embodiments of the application can be supplied as a method, a system, or a computer program product. Therefore, the application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer usable program code.

[0131] The application is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in a flow or multiple flows and / or blocks Figure 1 The functions specified in a flow or multiple flows and / or blocks

[0132] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 The functions of a flow or multiple flows and / or a block or multiple blocks in accordance with the present application can be implemented with software, hardware or with a combination of software and / or firmware. Figure 1

[0133] The software can be stored on one or more computer readable media, however, the computer readable media presented herein are only exemplary. For example, computer readable media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired computer program code in the form of computer readable instructions or data structures and that can be accessed by a computer or other programmable data processing apparatus. Figure 1 The functions of a flow or multiple flows and / or a block or multiple blocks in accordance with the present application can be implemented with software, hardware or with a combination of software and / or firmware. Figure 1

[0134] The above description is merely the preferred embodiment of the present application, and it should be pointed out that those skilled in the art can make several improvements and modifications without departing from the technical principles of the present application, and these improvements and modifications should also be considered as falling within the protection scope of the present application.​​

Claims

1. An unsupervised text-to-image person re-identification method, characterized in that: include: Constructing an unsupervised dataset, wherein the unsupervised dataset includes multiple sets of image-text pairs; The image in the image-text pair is divided into regions, and then each image region and text are encoded separately. For each image region, the text with high cosine similarity is selected and merged as the new text to obtain a new dataset. Use the new dataset to train the ReID model to obtain a pre-trained ReID model; Based on the pre-trained ReID model, the image-text pairs whose most similar regions of each text in the corresponding image in the new dataset are consistent with the results of the new dataset are selected to obtain the filtered dataset; The filtered data set is used to train the ReID model to obtain a trained ReID model; The text to be identified is input into the trained ReID model to obtain the Top-k similar images, realizing unsupervised text-to-image person re-identification.

2. The unsupervised text-to-image person re-identification method according to claim 1, characterized in that The construction of the unsupervised dataset includes: Obtaining an unlabeled training set and corpus , the unlabeled training set , including N pedestrian images, the corpus , including M pedestrian description texts; From the corpus Mining appropriate text for each image x and generating image-text pairs (x, T i ), where each text T i By corpus It consists of one or more texts in .

3. The unsupervised text-to-image person re-identification method according to claim 1, characterized in that The step of dividing the image in the image-text pair into regions includes: The image Divide into four image regions R according to the preset ratio k , k=1,2,3,4, corresponding to the head area, upper body area, lower body area and shoe area respectively; Among them, R represents the real space, H represents the image height, W represents the image width, and C represents the number of RGB channels.

4. The unsupervised text-to-image person re-identification method according to claim 3, characterized in that The encoding of each image region and text separately includes: For each image region R k , CLIP's visual Transformer is used to extract features and divide each image region into Image blocks of size, generating token sequences ;in, represents the CLS token feature vector corresponding to the k-th image region, v indicates that the feature vector is a visual feature, Represents the 1st,…, Nth in the kth image region respectively k The feature vector of the image block, N k Indicates the maximum image block number obtained after the k-th image area is divided into blocks. , H k and W k Represents the image area R k height and width; For the text t, use CLIP text encoder and encode it into a sequence through Byte Pair Encoding , and input the Transformer model to capture the context relationship; where SOS represents the start of sequence marker, w1, w2,…, w m The 1st, 2nd, ..., mth text subword embedding vectors respectively, and EOS represents the end-of-sequence marker.

5. The unsupervised text-to-image person re-identification method according to claim 3, characterized in that The method of selecting text with high cosine similarity for each image region and merging them as new text includes: Calculate the image area R k Cosine similarity with text t , the formula is as follows: ； in, EOS token feature vector representing text t, represents the CLS token feature vector corresponding to the k-th image region; For each image region, retain the top n texts with the highest cosine similarity and concatenate them into a comprehensive description text T u : ； in, They represent the u-th text corresponding to image regions R1, R2, R3, and R4 respectively, and n represents the number of texts retained in each image region sorted from high to low according to cosine similarity.

6. The unsupervised text-to-image person re-identification method according to claim 1, characterized in that The new dataset is used to train the ReID model, including: Input the image-text pairs in the new dataset D1 into the ReID model, map the image and text features into the same embedding space through cross-modal contrastive learning, and output the InfoNCE loss function value; When the InfoNCE loss function value decreases by less than 1e-4 for five consecutive training cycles on the validation set, it is considered to have converged and the pre-trained ReID model is obtained.

7. The unsupervised text-to-image person re-identification method according to claim 3, characterized in that The pre-trained ReID model selects the image-text pairs whose most similar regions of each text in the new dataset in the corresponding image are consistent with the new dataset results, including: For the image region R k The corresponding u-th text , use the pre-trained ReID model to calculate the text and image regions R j The cosine similarity between , where j = 1, 2, 3, 4, R j For R k The same four physical regions are only distinguished by the index variable j during the traversal calculation process; Normalize the similarity scores of each image region to obtain a new normalized score ; if Only one image region is matched, and the calculation Corresponding to each area of ​​the image The similarity score of , the region index y with the highest similarity is: ； in, Represents the normalized score The value of j when the maximum value is taken; if the region index y with the highest similarity corresponds to the kth image region, then retain , otherwise it is filtered as noise; if If it appears in multiple image areas, deduplication is performed.

8. The unsupervised text-to-image person re-identification method according to claim 7, characterized in that The deduplication operation includes: calculate Comprehensive score in each image area , the formula is as follows: ； in, , is the weight coefficient, n represents the number of texts retained in each image region sorted from high to low according to cosine similarity; Will Retain the image regions with the highest overall scores.

9. The unsupervised text-to-image person re-identification method according to claim 1, characterized in that The method also includes testing the trained ReID model: The text in the test set is input into the trained ReID model, and the cosine similarity between the text and all image regions in the test set is calculated. A list of candidate images is output, sorted from high to low by cosine similarity. The Rank-k0 indicator is used as the evaluation criterion to evaluate the retrieval performance of the model.

10. An unsupervised text-to-image person re-identification system, characterized in that include: A dataset construction module, configured to construct an unsupervised dataset, wherein the unsupervised dataset includes multiple sets of image-text pairs; The data reorganization module is used to divide the image in the image-text pair into regions, then encode each image region and text separately, select the text with high cosine similarity for each image region, merge them as the new text, and obtain a new dataset; The first model training module is used to train the ReID model using a new dataset to obtain a pre-trained ReID model; The data screening module is used to select the image-text pairs whose most similar regions of each text in the corresponding image in the new dataset are consistent with the results of the new dataset based on the pre-trained ReID model, thereby obtaining the filtered dataset; The second model training module is used to train the ReID model using the filtered data set to obtain a trained ReID model; The recognition module is used to input the text to be recognized into the trained ReID model to obtain a list of candidate images and realize unsupervised text-to-image pedestrian re-identification.

Images