Text-to-image pedestrian re-identification method based on word level enhancement and region selection

Abstract

The invention relates to the technical field of pedestrian re-recognition, in particular to a text-to-image pedestrian re-recognition method based on word level enhancement and region selection, which comprises the following steps: acquiring a pedestrian image to be recognized and a corresponding text; constructing a WIRS model: inputting a text into a text feature extraction module, and aligning global significant features of the image text; inputting the pedestrian image into an image feature extraction module, endowing local features of different body parts with weights, and fusing the local features with original features according to a learned proportion to obtain the overall similarity of the local image and the text; and mining unmatched negative correlation words in the text by using a false positive mining branch module. According to the method, the problems that text features and image features are not efficiently extracted and discrimination is insufficient in an existing method are solved.

Description

Technical Field

[0001] This invention relates to the field of pedestrian re-identification technology, and more particularly to a text-to-image pedestrian re-identification method that combines word-level enhancement and region selection. Background Technology

[0002] Text-to-image person re-identification (TI-Reid) aims to find corresponding pedestrian images from multiple cameras based on a given text description, and has irreplaceable application value in fields such as public safety and missing person services.

[0003] However, TI-Reid faces two major challenges: First, images and text are heterogeneous modalities, making it difficult to directly align cross-modal features; second, the task generally suffers from large intra-class differences and small inter-class differences, meaning that different images or descriptions of the same person may differ significantly, while the features of different people may be highly similar, which further increases the difficulty of retrieval.

[0004] The GNA-RNN model suffers from core problems such as rigid attention mechanism constraints, one-sided modality fusion, and limited generalization ability, which restricts its applicability in complex scenes. The block-word matching framework proposed by Chen et al. is insufficient because complex background noise or occlusion often exists in images, causing noisy local blocks to have a significant negative impact on the model. The CFine model proposed by Yan et al. is insufficient in extracting fine-grained features, which limits the model's performance. Although the above methods all use separate pre-trained visual encoders and text encoders, this results in a lack of cross-modal interaction in the encoder during pre-training, which makes the learned semantic information insufficient in cross-modal alignment.

[0005] The SSAN model proposed by Ding et al. failed to notice the different importance of different local images. Some local images do not correspond to any text keywords, while some local images correspond to multiple keywords. Therefore, different local images should be distinguished during feature alignment. However, the model is not efficient enough in extracting text features and image features, and its discriminative ability is insufficient. Summary of the Invention

[0006] To address the shortcomings of existing methods, this invention solves the problems of insufficient efficiency and discriminative power in the extraction of text and image features.

[0007] The technical solution adopted in this invention is: a text-to-image person re-identification method with word-level enhancement and region selection, comprising the following steps: Step 1: Collect images of pedestrians to be identified and their corresponding text; Step 2: Construct the WIRS model, including: inputting text into the text feature extraction module to align the global salient features of the image and text; inputting pedestrian images into the image feature extraction module to assign weights to the local features of different body parts, and fusing them with the original features according to the learned proportions to obtain the overall similarity between the local image and the text; and using the false positive mining branch module to mine mismatched negative related words in the text. In a preferred embodiment of the present invention, the text feature extraction module includes: Text The word embeddings are encoded into word embedding vectors with text description tags and then input into the CLIP text encoder to obtain the text embeddings. and text global features T g ; Will Inputting the TWFAU module yields text embeddings. ;Will copy After the text is divided into segments, different scores are added to each word embedding using the WAM module, and then the results are input into the RMP module to obtain a local text representation. .

[0008] In a preferred embodiment of the present invention, the TWFAU module includes: Will Text embeddings are obtained by performing Reshape operations, DConv, BN, and Prelu functions. Then input the Conv, BN, and Prelu functions to obtain the text embedding. Then input Conv to get the text embedding. .

[0009] In a preferred embodiment of the present invention, the image feature extraction module includes: pedestrian images Input the CLIP image encoder to obtain global image features. I g and local features of the image ;right The image local representation is obtained by stitching. ;Will Input RRM module outputs local image representation .

[0010] In a preferred embodiment of the present invention, the RRM module includes: Will Input the fully connected layer to obtain the weight values ​​for each local region; then combine the weight values ​​with... Multiplying the local features together yields the weighted image local features. Finally, through learnable parameters ,Will and by and Proportional fusion, output image local visual representation ;in, , .

[0011] In a preferred embodiment of the present invention, the false positive detection branch module includes: and After inputting Conv 1x1 respectively, calculate the sum of each word and the expression. Similarity score of each region in, ; calculate The maximum of the similarity scores ,when Calculate negative similarity when it is negative. ;Will s neg With local similarity Summation yields local negative similarity. .

[0012] As a preferred embodiment of the present invention, when When it is positive, for Balanced sampling, then calculating the positive image text pairs. and negative image text pairs , Then calculate the total cross-ReLU loss. .

[0013] As a preferred embodiment of the present invention, utilizing T g and I g Calculate global similarity reuse s g , s l , s local-neg Calculate the total ranking loss ; , , This is a hyperparameter.

[0014] As a preferred embodiment of the present invention, the total loss function , For the loss of identity.

[0015] As a preferred embodiment of the present invention ; It's a hyperparameter.

[0016] The beneficial effects of this invention are: 1. In order to better align fine-grained features and further improve the accuracy of word-region alignment and the overall performance of the model, this invention proposes the word-level augmentation and region reselection framework WIRS; 2. By using CLIP to obtain the classification embedding of the image and the EOS embedding of the text, the global salient features of the image and text can be effectively aligned. 3. In terms of fine-grained features, the word embeddings are enhanced by the text word-level feature auxiliary unit TWFAU to obtain more discriminative word embeddings; 4. By using the Region Reselection (RRM) module to assign weights to local features of different body parts and fusing them with the original features based on the proportions learned by the model, the overall similarity between local images and text is influenced. 5. Use the False Positive Mining (FPM) branch to mine non-matching negatively related words in the text, to prevent these negatively related words from being ignored because the image and text have too high overall similarity. 6. Extensive experiments were conducted on three public benchmark datasets (CUHK-PEDES, ICFG-PEDES, and RSTPReid), and Rank1, Rank5, Rank10, and mAP metrics were calculated to demonstrate the effectiveness of the proposed model. Attached Figure Description

[0017] Figure 1 This is a block diagram of the text-to-image pedestrian re-identification method with word-level enhancement and region selection according to the present invention; Figure 2 This is a schematic diagram of the structure of the TWFAU of the present invention; Figure 3 This is a schematic diagram of the RRM structure of the present invention; Figure 4 This is a schematic diagram of the behavior recognition of the present invention. Figure 1 ; Figure 5 This is a schematic diagram of the behavior recognition of the present invention. Figure 2 ; Figure 6 This is a schematic diagram of the behavior recognition of the present invention. Figure 3 . Detailed Implementation

[0018] The present invention will be further described below with reference to the accompanying drawings and embodiments. The drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0019] like Figure 1 As shown, a text-to-image person re-identification method with word-level enhancement and region selection includes the following steps: The structure of WIRS is as follows Figure 1 As shown, it consists of a text feature extraction branch, an image feature extraction branch, and a false positive mining branch (FPM). Inspired by the partial successful transfer of CLIP in text-to-image person re-identification, CLIP is pre-trained and used as the backbone network of WIRS.

[0020] The text feature extraction branch includes modules such as a text encoder and TWFAU; the image feature extraction branch uses modules such as an image encoder and RRM; both branches are responsible for obtaining the corresponding global and local features, as well as the input features for the false positive mining branch; after obtaining the input features, the false positive mining branch further mines false positive samples, and the specific process is as follows: In the text feature extraction branch, the text is first... Encoded as a tag and used and To improve computational efficiency, the maximum length of the encoded sequence is limited to 77. This token sequence is then used as input to the CLIP text encoder and processed through a 12-layer self-attention and feedforward network to obtain the output text embedding, denoted as... , token1 to token N Each word embedding vector is a padded vector (if the text description is less than 77 characters, a padding vector is added). SOS and EOS are the start and end markers of a text description. To further enhance the text word embedding, The text is input into the TWFAU module and undergoes operations such as depthwise separable convolution to obtain enhanced text embeddings. Then, copy Corresponding image A local area (in this article) Set to 6); it is then passed through a Word Attention Module (WAM) to add different scores to each word embedding, highlighting those words that are more discriminative and locally relevant to the image; finally, it is subjected to row-wise max pooling (RMP) to obtain the local text representation, denoted as . Furthermore, TWFAU was enhanced. After convolution to change the channels, the features are used as input features for the false positive detection branch; because the CLIP text encoder is an improved Transformer and uses an upper triangular mask in multi-head attention, the text encoder output... Embeddings not only contain the contextual information of the entire text but also capture complex semantic relationships. Therefore, they are treated as global features of the text and denoted as... T g .

[0021] However, the word embeddings obtained by the CLIP text encoder also suffer from information compression in semantic representation, which leads to the loss of fine-grained semantics (such as polysemous ambiguity and grammatical structure dependence) in cross-modal alignment. Therefore, TWFAU enhances the semantic features of word embeddings, which is crucial for subsequent fine-grained word-region matching.

[0022] Furthermore, considering that overly deep module structures often lead to later extracted features forgetting earlier features, and that residual connections do not always effectively alleviate this problem, it is best to be lightweight while ensuring performance; specifically, such as... Figure 2 As shown, TWFAU has a simple and lightweight structure, mainly including operations such as depthwise separable convolution, batch normalization, and PreLU. Among them, depthwise separable convolution is an efficient convolution operation that decomposes standard convolution into depthwise convolution and pointwise convolution. Depthwise convolution can obtain the statistical features of each word embedding, while pointwise convolution can further transform these features. In addition, the PReLU activation function it uses has advantages over the traditional ReLU activation function, as it can adaptively learn the slope of the negative half-axis and increase the non-linear expressive power of the model, enabling the model to learn more complex patterns. The specific calculation method is as follows: If the input is Then when The output is ,when The output is , These are learnable parameters.

[0023] Finally, TWFAU is implemented in the following way: (1) in, ; ; This represents the text features extracted by the CLIP text encoder; This represents depthwise separable convolution; Indicates batch normalization; This is the Prelude activation function; Indicates the convolution operation; This is the text representation of the final output of TWFAU.

[0024] like Figure 1 In the image feature extraction branch, on the one hand, the image... The input is fed into the CLIP image encoder to obtain the block embedding token and the classification label CLS. Since the classification label contains information about the entire image, it is used as a global feature of the image, denoted as . I g On the other hand, the image Divided into The data is then input into the image encoder separately to obtain... There are 1 classification label; where each classification label is a local feature of the image, denoted as . After splicing them together, we get This is the local representation of the image; then, ... The input is fed into the RRM for region-weighted selection and enhancement, thereby obtaining a local representation of the image. Furthermore, it is changed by one-dimensional convolution. After determining the number of channels, it is used as a local feature of the image for the false positive mining branch.

[0025] Depend on Figure 1 It can be seen that the local features of the image Will be related to local text features Word-region matching aims to enable the model to focus more on the fine-grained features of images and text and to calculate their correlation at a fine-grained level. Since local image features only contain partial information about pedestrians, these local features are only correlated with certain attribute words in the text. This results in a different number of attribute words corresponding to each local image feature. Therefore, to improve the accuracy of fine-grained feature cross-modal matching, and to assign higher weights to local images with more corresponding attribute words and lower weights to local images with fewer or no related attribute words, a Region Reselection Module (RRM) is proposed. The RRM first... Local image features are enhanced or weakened to a certain extent, giving greater weight to local features with more corresponding attribute words during matching. Then, the enhanced or weakened local image features are processed through learnable parameters. The features are fused with the original input features to prevent local image features from being forgotten after weighting.

[0026] The structure of RRM is as follows: Figure 3 As shown, local image features The input to the fully connected layer yields the weights for each local feature. Since weights that are too large or too small hinder rapid model fitting during training, they are scaled according to the dimensionality of the local features, and a sigmoid activation function is used to control the weights between 0 and 1. Then, the weights are compared with... Multiplying the local features together yields the weighted image local features. Finally, through learnable parameters The weighted local features are compared with by and The proportions are blended. The RRM implementation is as follows: (2) in, ; Local features of the input RRM; hyperparameters The value is 512. It is a fully connected layer; As a learnable parameter, it was initially set to 0.5 after experimental comparison; This represents the local visual representation after weighting. This represents the local visual representation of the RRM output.

[0027] Finally, to enable cross-modal matching of text and image features, it is also necessary to calculate the similarity between text and image features; and to represent the local aspects of the image. and text local representation The images and text are concatenated into a unified representation, and then local similarity is calculated. The formulas for calculating global and local similarity between images and text are as follows: (3) (4) in, s g This represents the global feature similarity between image and text pairs. s l This represents the local feature similarity between image and text pairs.

[0028] When calculating the similarity of local features between images and text, there is sometimes an over-reliance on matching word-region pairs, while non-matching word-region pairs are ignored due to their small proportion, which may lead to incorrect matching between images and text. To address this issue, a false positive mining branch is introduced into the model; false positive samples are distinguished by mining negative word-region pairs; false positive samples represent image-text pairs that match most attribute words in the text description, but do not match only a few words, resulting in incorrect matching.

[0029] like Figure 1 As shown, the local features of the image and local features of text , respectively denoted as and ( n (equal to 77); will and The input to this branch is reduced by convolution; then, the input of each word is calculated. The similarity score for each region is calculated using the following formula: (5) in,

[0030] Each word will correspond to The similarity score for each region; calculate this. The maximum similarity score is denoted as . ,if A negative similarity indicates that the word has no correlation with the image, i.e., it is a negatively correlated word; the negative similarity is obtained by summing the similarities of all negatively correlated words. s neg It emphasizes words in the text that do not match the image; s neg and s l Summation yields the corrected local similarity. s local-neg The calculation formula is as follows: (6) (7) (8) in, Mask mining (.) express s i Set to zero when positive. s i Retain if negative.

[0031] During the reasoning phase, s g , s l and s local-neg The summation is used as the final similarity score for the image-text pair.

[0032] Identity loss is a loss function frequently used in person re-identification. During model training, it predicts person identity through feature classification and calculates a loss value based on the different probabilities of the predictions. Its purpose is to improve the discriminative power of the features themselves. To ensure that both global and local features have a certain degree of discriminative power, identity loss is calculated separately for them during training. The formula for identity loss is: (9) in, Features representing images or text It is a fully connected layer.

[0033] global image features Global text features Local image features and local text features Calculate the identity loss separately; then sum them to obtain the total identity loss. The calculation formula is as follows: (10) in, It is a hyperparameter, and after experimental comparison, we chose to set it to 1.

[0034] Furthermore, in text-to-image person re-identification tasks, ranking loss is a commonly used loss function; it measures the difference in ranking between the model's predictions and the actual results. To ensure correct matching and ranking of text and image features in the feature space, with positive image-text pairs having higher similarity and negative image-text pairs having lower similarity, ranking loss will be used to optimize the model parameters. The formula for ranking loss is: (11) in, and For mismatched image-text pairs; For matching image-text pairs; hyperparameters It is 0.2; The resulting ranking loss.

[0035] The total ranking loss during model training L ranking The formula is as follows: (12) in, L r (s g ) This represents the ranking loss calculated based on global similarity. L r (s l ) This represents the ranking loss calculated based on local similarity. L r (s local-neg ) This represents the ranking loss in the local similarity calculation after false positive detection; , , These are hyperparameters because global features of images and text have the greatest impact on model performance. Choose 1; local fine-grained features can focus on more image details and text keywords, supplementing areas where global features are insufficient in discriminative power, therefore Take 0.5, Take 0.25; , , After conducting comparative experiments with other different values, it was found that when =1、 =0.5、 When the value is 0.25, the model's evaluation index is also optimal.

[0036] A custom cross-ReLU loss is used in the false positive detection branch. This loss function aims to: 1) maximize the word region similarity in positive sample pairs to reduce intra-class discrepancies; and 2) identify the least likely matching word region pairs in negative sample pairs and minimize their similarity. A balanced sampling strategy is employed here; this loss function aims to ensure that as many words in the text as possible are correlated with local elements of the image.

[0037] The formula for calculating cross-ReLU loss for positive image-text pairs; (13) in, This is a hyperparameter; n is the batch size. The maximum similarity between a word and a local image element, if A positive value indicates a positive correlation between the word and a certain part of the image, but the degree of this positive correlation is not important. It can be set to a smaller value, 0.001; if... Scaling will affect The size. After experimental comparison, Set it to 1.

[0038] For negative image-text pairs, the cross-Relu loss is calculated as shown in Equation (14), which aims to extract the words with the strongest negative correlation to the local image as much as possible.

[0039] (14)

[0040] in, This represents the maximum negative similarity between a word in the text and a local area of ​​the image. Represents the cross-ReLU loss for negative image-text pairs; in order to As small as possible It can be larger; after experimental comparison, it was set to 0.15. This is a hyperparameter, set to 1.

[0041] Finally, the total cross-ReLU loss is obtained by summing: (15) The total loss used for backpropagation can be expressed as: (16) Experimental procedure: To comprehensively evaluate the performance of the WIRS model, a systematic comparative experiment was conducted with current mainstream methods on three benchmark datasets: CUHK-PEDES, ICFG-PEDES, and RSTPReid.

[0042] CUHK-PEDES is the first publicly available benchmark dataset in the field of text-to-image person re-identification. Its data scale and annotation system have laid an important foundation for research in this field. The training set contains 34,054 pedestrian images and 68,108 text descriptions, covering 11,003 pedestrian identity categories. The validation set and test set contain 3,078 and 3,074 images, and 6,158 and 6,156 text descriptions, respectively, each covering 1,000 pedestrian identity categories.

[0043] RSTPReid is built on the MSMT17 dataset; it contains 20,505 pedestrian images taken by 15 cameras; each image corresponds to two text descriptions; these images cover complex indoor and outdoor scene changes and various backgrounds, making the dataset more challenging; according to the official partitioning strategy, the training set contains 3,701 identity categories; the validation set and test set each contain 200 identity categories. ICFG-PEDES is also built on the MSMT17 dataset; it contains the most images than the previous two, including 54,522 pedestrian images; the training set contains 34,674 image-text pairs and 3,102 identity categories; the test set contains 19,848 image-text pairs and 1,000 identity categories.

[0044] Evaluation indicator system: For evaluation, Rank-K metric and mean precision (mAP) were used. Rank-K metric measures the model's retrieval performance and intuitively reflects the model's ability to quickly locate and correctly match images. It means that for a given pedestrian text description, the proportion of the top K images retrieved from the image library that match the text description is rank-K. Rank-1, Rank-5, and Rank-10 were selected from Rank-K as reference metrics for model evaluation. Mean precision (mAP) is a key indicator for measuring the quality of the retrieval model, and it can comprehensively reflect the model's ability to recommend positive matches in the top positions.

[0045] Rank-1 is considered a core representation of model performance; therefore, during training, the optimal model parameters are saved based on the Rank-1 metric.

[0046] Experimental parameters and hardware environment: The experiments involved in this invention were trained and tested on an Nvidia GeForce RTX 3090Ti using the deep learning framework PyTorch 2.3.0. The backbone network of the model consists of a pre-trained image encoder (CLIP-ViT-B / 16) and a pre-trained CLIP text encoder. The image was augmented with scaling, flipping, cropping, normalization, and erasing operations to resize it to 384x128. The maximum length of the text tag sequence was 77, and the model used the Adam optimizer. For the pre-trained CLIP backbone network, its initial learning rate was set to 1x10. -6 The remaining parameters are set to an initial learning rate of 0.001; the learning rate scheduler is MultiStepLR, which can dynamically reduce the learning rate according to the preset, thereby helping the model converge more stably; the model adopts mixed precision training, which on the one hand reduces the memory usage and accelerates the training process; on the other hand, it can maintain the stability of the model values.

[0047] Comparative experiment: To verify the performance of WIRS in the text-to-image person re-identification task, comparative experiments were conducted with current mainstream models on three benchmark datasets. In Tables 1-3, Pre(CLIP) represents the performance of the pre-trained CLIP on the three datasets. As the baseline model of WIRS, it can enhance the cross-modal retrieval and generalization capabilities of WIRS.

[0048] As shown in Table 1, comparative experiments were conducted on the CUHL-PEDES dataset. The selected comparative experimental models covered early basic models (such as ViTAA, DSSL, etc.), new backbone network models (IVT, etc.), and the current cutting-edge models in this field (such as CPCL). Different technical routes were involved, and mainstream encoder combinations were used to fully verify the competitiveness of WIRS.

[0049] In the table, WIRS's Rank-1 score is 70.68%, which is higher than the other methods compared, indicating that it is superior to other methods in accurately matching the target pedestrian at the first position. In terms of retrieval coverage (Rank-10), WIRS reaches 91.93%, which is also higher than the compared methods, making it easier to capture correct results in large-scale retrieval scenarios. Finally, for overall retrieval quality (mAP), WIRS is 61.32%, which also reaches a relatively advanced performance, only slightly lower than CPCL's 63.19%.

[0050] Table 1. Performance evaluation results of different models on the CUHK-PEDES dataset.

[0051] As shown in Table 2, on the ICFG-PEDES dataset, WIRS achieved Rank-1 and mAP of 62.64% and 35.82% respectively, surpassing other comparative methods. Compared with the recent CFine, it improved Rank-1 by 1.81%, Rank-5 by 2.2%, and Rank-10 by 1.92%, showing a significant overall performance improvement.

[0052] Table 2. Evaluation results of different models on the ICFG-PEDES dataset.

[0053] Table 3 compares WIRS on the RSTPReid dataset. Its Rank-1 score reached 61.5%, higher than other models (such as IRRA's 60.2%), and its Rank-10 score also reached 88.6%, leading other methods. This shows that even with few samples and occlusion, WIRS can still accurately match target pedestrians and has a higher probability of including target pedestrians in large-scale retrieval. However, in terms of mAP, IRRA reached 47.17%, which is better than WIRS's 43.81%.

[0054] Table 3. Evaluation results of different models on the RSTPReid dataset.

[0055] In summary, the analysis shows that WIRS, with its CLIP generalization ability and the fine-grained enhancements of TWFAU and RRM, outperforms other models on various datasets in multiple metrics, especially the most important Rank-1 metric, which shows a significant improvement, surpassing other methods on all three datasets. This provides a new, fast, and accurate solution for matching target pedestrians in this field. Furthermore, comparative analysis on the CUHK-PEDES and RSTPReid datasets reveals that the mAP metric is not optimal. This is mainly because the model sometimes places certain positive image samples in the middle position during retrieval and ranking. Further research will be conducted to address this issue. ablation experiment The impact of TWFAU and RRM on model performance was tested through ablation experiments; the network model after removing TWFAU and RRM was regarded as the baseline.

[0056] Table 4 shows the ablation experiment results on three datasets. The evaluation metrics are Rank-1, Rank-5, and Rank-10. When TWFAU is added to the baseline, Rank-1 improves by 4.77%, 3.12%, and 5.75% respectively on the three datasets compared to the baseline, significantly enhancing the ability to retrieve the first target pedestrian. In addition, single-layer bidirectional LSTM is used as a text encoder in some models, such as MFPE and ViTAA. This is mainly due to its unique advantages in processing sequence data, especially its suitability for the core needs of text semantic understanding. Therefore, to further verify the enhancement effect of the TWFAU module on word embedding, a single-layer bidirectional LSTM was used instead of TWFAU in the baseline. It was found that, compared to the baseline, Rank-1 improved by 0.26% and 3.65% on the CUHK-PEDES and RSTPReid datasets, respectively, but decreased by 2.56% on the ICFG-PEDES dataset, which is significantly worse than the effect of adding TWFAU to the baseline. This further illustrates the effectiveness and irreplaceability of TWFAU.

[0057] Table 4 shows the ablation experiments on three datasets, with evaluation metrics of rank-1, rank-5, and rank-10.

[0058] When RRM is added to the baseline, its Rank-1 performance improves by 0.60%, 1.17%, and 1.38% on the three datasets compared to the baseline, respectively. The improvement is more significant on the latter two datasets, primarily because the images in these datasets exhibit more occlusion, and RRM's local weighting strategy effectively reduces the interference of occlusion noise, resulting in a more pronounced effect on these datasets. Furthermore, RRM's performance improvement on the baseline is not as significant as TWFAU's, and its improvement on the CUHK-PEDES dataset is even more limited. This is mainly due to two reasons: First, without TWFAU enhancement, the key attribute words / attribute phrase features obtained by the input word attention module may be biased, leading to... During training, local image enhancement strategies struggle to learn the true feature distribution relationships of local regions, resulting in limited performance improvement of RRM on the baseline. Secondly, the poses of human images in the CUHK-PEDES dataset are relatively simple, leading to insufficient discriminative learning of local images by RRM. Consequently, the ablation experiment results of RRM on the CUHK-PEDES dataset are inferior to those trained on two other datasets with richer human poses. Finally, by adding TWFAU and RRM to the baseline, i.e., WIRS, the Rank-1 performance improved by 4.94%, 3.63%, and 6.15% respectively compared to the baseline. Significant improvements were also achieved in Rank-5 and Rank-10, fully demonstrating the effectiveness of their collaborative work.

[0059] To further evaluate the impact of the modules on retrieval quality, as shown in Table 5, the evaluation metric was changed to mAP, and ablation experiments were conducted again. Compared to the baseline, adding TWFAU improved mAP by 4.77%, 2.54%, and 4.18% on the three datasets, respectively, resulting in a significant improvement in retrieval quality. Adding RRM to the baseline also improved the mAP metric. If both TWFAU and RRM were added to the baseline, mAP improved by 5.12%, 3.50%, and 3.97% on the three datasets, respectively, demonstrating a significant improvement.

[0060] Table 5 shows the ablation experiments on three datasets, with mAP as the evaluation metric.

[0061] Visualization Analysis

[0062] Figure 4 , Figure 5 , Figure 6Three sets of search results for Baseline and WIRS are presented, denoted as A, B, and C respectively. All three searches were performed on the CUHK-PEDES dataset. In each set of search results, the first row contains the target image and the top 10 search results for Baseline, and the second row contains the target image and the top 10 search results for WIRS. In addition, dashed boxes mark the correct searched images, and square boxes indicate the information corresponding to the text. In A, the underlined text describes a white and gray striped shirt and green pants. The search results show that both Baseline and WIRS effectively identified the white and gray striped shirt feature, but Baseline failed to identify the green pants as well, while WIRS identified this feature more accurately, as indicated by the box in A. In B, for the text descriptions "woman, white shirt, black jacket," both Baseline and WIRS retrieved corresponding pedestrian features. However, for the feature "a bag on her right arm," WIRS was more precise than Baseline, as indicated by the box in B, where the bag appeared on the left arm or right hand. In C, Baseline made an error in gender identification; its 10th result was a man, and the text description "black pants" resulted in a skirt. WIRS avoided these issues in its search results. However, both Baseline and WIRS made one error in identifying a blue-green T-shirt, as indicated by the box in C.

[0063] Visual analysis shows that, compared to the baseline, WIRS can focus on more details in text descriptions and images, thereby improving the overall matching accuracy. Therefore, it further verifies the enhancement effect of TWFAU and RRM on fine-grained features. However, the shortcoming is that the understanding of color-related words is not very good (such as green pants, blue-green T-shirt, etc.). This is mainly caused by insufficient pre-training of CLIP, which makes CLIP lack a lot of general image and text information related to color.

[0064] To address the challenges of insufficient fine-grained feature extraction and difficulties in cross-modal alignment in TI-Reid, this invention proposes a network architecture based on word-level enhancement and region selection (WIRS). This network uses CLIP as its backbone and incorporates two modules: TWFAU and RRM. Experiments show that TWFAU significantly enhances text word embeddings, solving the problem of fine-grained cross-modal alignment. The RRM module can flexibly enhance local image data, making the alignment of fine-grained features more accurate. Furthermore, a false positive mining branch proposed by Xu et al. is used in WIRS to mine negatively correlated word-region pairs while performing fine-grained feature matching, finding hard negative samples with only subtle differences, further improving the model's generalization ability. Experimental results on three public datasets show that WIRS performs excellently in the TI-Reid task.

[0065] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A text-to-image person re-identification method with word-level enhancement and region selection, characterized in that, Includes the following steps: Step 1: Collect images of pedestrians to be identified and their corresponding text; Step 2: Construct the WIRS model, including: inputting text into the text feature extraction module and aligning the global salient features of the image text; The pedestrian image is input into the image feature extraction module, which assigns weights to the local features of different body parts and fuses them with the original features based on the learned proportions to obtain the overall similarity between the local image and the text. The false positive mining branch module is used to mine the mismatched negative related words in the text. The WIRS model is backpropagated and trained using the total loss function.

2. The text-to-image person re-identification method with word-level enhancement and region selection according to claim 1, characterized in that, The text feature extraction module includes: Text The word embeddings are encoded into word embedding vectors with text description tags and then input into the CLIP text encoder to obtain the text embeddings. and text global features T g ; Will Inputting the TWFAU module yields text embeddings. ;Will copy After the text is divided into segments, different scores are added to each word embedding using the WAM module, and then the results are input into the RMP module to obtain a local text representation. .

3. The text-to-image person re-identification method with word-level enhancement and region selection according to claim 2, characterized in that, The TWFAU module includes: Will Text embeddings are obtained by performing Reshape operations, DConv, BN, and Prelu functions. Then input the Conv, BN, and Prelu functions to obtain the text embedding. Then input Conv to get the text embedding. .

4. The text-to-image person re-identification method with word-level enhancement and region selection according to claim 1, characterized in that, The image feature extraction module includes: pedestrian images Input the CLIP image encoder to obtain global image features. I g and local features of the image ;right The image local representation is obtained by stitching. ;Will Input RRM module outputs local image representation .

5. The text-to-image person re-identification method with word-level enhancement and region selection according to claim 4, characterized in that, The RRM module includes: Will Input the fully connected layer to obtain the weight values ​​for each local region; then combine the weight values ​​with... Multiplying the local features together yields the weighted image local features. Finally, through learnable parameters ,Will and by and Proportional fusion, output image local visual representation ;in, , .

6. The text-to-image person re-identification method with word-level enhancement and region selection according to claim 1, characterized in that, The false positive detection branch module includes: Will and After inputting Conv 1x1 respectively, calculate the sum of each word and the expression. Similarity score of each region ;in, , calculate The maximum of the similarity scores ;when When the value is negative, calculate the negative similarity. ;Will s neg With local similarity Summation yields local negative similarity. .

7. The text-to-image person re-identification method with word-level enhancement and region selection according to claim 6, characterized in that, when When it is positive, for Balanced sampling, then calculating the positive image text pairs. and negative image text pairs , ; Obtain the total cross-ReLU loss .

8. The text-to-image person re-identification method with word-level enhancement and region selection according to claim 6, characterized in that, use T g and I g Calculate global similarity reuse s g , s l , s local-neg Calculate the total ranking loss ; , , This is a hyperparameter.

9. The text-to-image person re-identification method with word-level enhancement and region selection according to any one of claims 7 and 8, characterized in that, Total loss function , For the loss of identity.

10. The text-to-image person re-identification method with word-level enhancement and region selection according to claim 9, characterized in that, ; It's a hyperparameter.

Images