Text-image pedestrian re-identification method for noise correlation

Abstract

The invention discloses a text-image pedestrian re-identification method for noise correlation. The method comprises the following steps: respectively carrying out global feature representation and local feature representation on an image and a text, and carrying out global feature marking and local feature marking; dividing the image and text data by adopting a dynamic selection and correction method specific to an instance; constructing a multi-granularity global feature learning module, and inputting the local features of the image and the text into the multi-granularity global feature learning module to obtain multi-granularity global features; and training an optimization network model through ID classification loss and similarity distribution matching loss, carrying out loss calculation on the identity ID, and completing similarity distribution matching loss calculation on the multi-granularity global features. Through the technical scheme of the invention, the matching precision of pedestrian re-identification based on text description is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of cross-modal person re-identification, and in particular relates to a text-image person re-identification method for noise correlation. Background Art

[0002] Person re-identification based on text description (TPReID) aims to retrieve target person images with high semantic relevance to a given text description from an image database. Text description provides a more natural and comprehensive way to describe person attributes, which is usually easier to obtain than real-time images. Therefore, TPReID has received increasing attention in recent years, especially for its potential applications in finding suspects, missing children or the elderly.

[0003] Compared with image-based person re-identification, TPReID uses more accessible open text queries, providing a more friendly search process. However, due to the inherent semantic gap and appearance attribute redundancy between different modalities, cross-modal alignment becomes a corresponding challenge. In addition, compared with traditional text-image retrieval, TPReID focuses on cross-modal retrieval of people with finer granularity, which requires not only the model to understand the text description, but also to accurately identify and match pedestrians. Therefore, the TPReID task places higher requirements on feature extraction, similarity measurement, and optimization strategies, making it a more challenging and complex task.

[0004] To address these challenges, most existing technologies rely heavily on accurately matching image-text pairs for learning. In reality, due to human posture, camera angle, lighting, and other unavoidable factors in the image, there are some low-correlation or mismatched text descriptions in the image, namely, noise associations. The noise association problem in cross-modal retrieval refers to the false correspondence of image-text pairs in TPReID, namely, false positive pairs: some mismatched image-text pairs are mistakenly used as matching pairs for cross-modal learning, which will overfit the noise data and lead to a decrease in the performance of the method. Summary of the invention

[0005] The embodiment of the present invention provides a text-image person re-identification method for noise correlation, which solves the problem of incorrect matching of text-image pairs in a data set and improves the matching accuracy of text-image pairs.

[0006] The embodiment of the present invention provides a method for text-image person re-identification with respect to noise association, comprising: Perform global feature representation and local feature representation on images and texts respectively, and perform global feature labeling and local feature labeling; A dynamic instance-specific selection and correction method is used to partition image and text data; Construct a multi-granularity global feature learning module, input the local features of the image and text into the multi-granularity global feature learning module, and obtain multi-granularity global features; The network model is optimized through training of ID classification loss and similarity distribution matching loss, the loss of identity ID is calculated, and the similarity distribution matching loss is calculated for multi-granularity global features.

[0007] Furthermore, the global feature representation and local feature representation of the image and text respectively include: The image is segmented into a series of equal-sized and non-overlapping image blocks, and the segmented image blocks are converted into a one-dimensional tag sequence through a trainable linear projector. An additional CLS tag is added to the one-dimensional tag sequence, and the CLS position information is introduced. The processed CLS tag is sent to the CLIP-ViT image encoder for processing.

[0008] Furthermore, the global feature representation and local feature representation of the image and text respectively also include: using lowercase byte pair encoding to tokenize the input text, and adding special tags SOS and EOS to the tokenized text sequence to indicate the beginning and end of the sequence respectively, introducing a special tag CLS position information, and merging the position information into the tokenized text sequence.

[0009] Furthermore, the dynamic instance-specific selection and correction method is used to divide the image and text data, including: Obtain global and local feature representations of images and texts through the backbone network; Use the global features of the image and text to predict the identity ID and obtain the corresponding confidence level; The consistency and difference of identity ID prediction based on the global features of images and texts are used to divide image and text data into clean subsets and noisy subsets.

[0010] Furthermore, the method of using the consistency and difference of the global features of the image and text to predict the identity ID to divide the image and text data into a clean subset and a noise subset includes: If the confidence values ​​of the corresponding image and text data are both higher than the corresponding threshold, they are classified as clean data; If only one of the confidence values ​​of the corresponding image and text data is higher than the corresponding threshold, it is classified as noise data; If the confidence values ​​of the corresponding image and text data are both lower than the corresponding thresholds, they are classified as abnormal data and directly eliminated.

[0011] Furthermore, the multi-granularity global feature learning module is constructed, and the local features of the image and the text are input into the multi-granularity global feature learning module to obtain the multi-granularity global features, including: Calculate the correlation weights between local features and global features, and select the top 2K tags with the highest scores from the local feature tags of the image and text according to the size of the correlation weights to form a new tag sequence; The first K markers are used to construct a high-level discriminant sequence, the last K markers are used to construct a mid-level discriminant sequence, and all local feature markers are used to construct a low-level discriminant sequence, thus obtaining a set of multi-granularity global features.

[0012] Furthermore, the network model is optimized by training the ID classification loss and the similarity distribution matching loss, including the calculation of the loss of the identity ID: Using traditional cross entropy to calculate the loss of clean data ; Computing the loss for noisy data using generalized cross entropy ; The sum of the loss of clean data and noisy data is taken as the identity loss : .

[0013] Furthermore, the training and optimization of the network model through ID classification loss and similarity distribution matching loss also includes the calculation of similarity distribution matching loss for multi-granularity global features, specifically: Calculate the similarity distribution matching loss when matching from image to text in multi-granular global features ; Calculate the similarity distribution matching loss when matching from text to image in multi-granular global features ; The bidirectional similarity distribution matching loss is: .

[0014] Furthermore, the network model is optimized through ID classification loss and similarity distribution matching loss training, and the overall optimization goal is: ; in, L For the overall optimization goal, For identity ID loss, is the bidirectional similarity distribution matching loss.

[0015] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention proposes a dynamic instance-specific selection and correction method (DISDR) to reduce the adverse effects of noise correlation through precise data partitioning, thereby improving the robustness of the model to noisy data and the matching accuracy.

[0016] (2) The present invention designs a multi-granularity global feature learning module, which eliminates useless information labels through a label selection mechanism, improves the global feature generation method in the CLIP model, promotes the model to focus on meaningful parts, and further improves the matching effect of the pedestrian re-identification model.

[0017] (3) The proposed method can effectively divide noise sample pairs as the noise level increases, whether in the case of well-matched or synthetic noise. It is superior to the baseline model and significantly improves the robustness to noise correlation problems.

[0018] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 The present invention provides a flowchart of a method for text-image person re-identification with respect to noise association.

[0021] Figure 2 It is a structural schematic diagram of an optimized network model for a noise-correlated text-image person re-identification method provided by an embodiment of the present invention.

[0022] Figure 3 It is a schematic diagram of a model of subset division and label correction provided by an embodiment of the present invention.

[0023] Figure 4 This is a performance comparison table of the method provided in an embodiment of the present invention and the SOTA method on the RSTPReid dataset.

[0024] Figure 5 This is an ablation experiment result table provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0026] The present invention proposes a multi-granularity image-text feature person re-identification network based on noise association processing. The network divides the data set into a clean subset and a noisy subset through a dynamic instance-specific partitioning method, and uses the clean subset for learning multi-granularity global features, and the multi-granularity feature set for loss calculation to complete model training optimization.

[0027] It should be noted that the multi-granularity image text feature pedestrian re-identification network based on noise association processing in this embodiment includes four parts: a feature extraction dual encoder, a dynamic instance-specific division and correction branch, a multi-granularity global feature learning branch, and an optimization strategy.

[0028] Among them, the purpose of the dynamic instance-specific selection and correction branch (DISDR) is to divide the noisy dataset into clean subsets and noisy subsets through the consistency and difference of image and text for identity ID prediction, and further correct the matching labels.

[0029] The multi-granularity global feature learning branch (MGGFL) generates a set of multi-granularity global image text features by mining local information at different levels, strengthening the influence of important local information and enhancing fine-grained clues.

[0030] Figure 1 A flowchart of a method for text-image person re-identification with noise correlation provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the present embodiment provides a method for text-image pedestrian re-identification with noise association, which is applicable to scenarios such as public security and video surveillance of text-image dual recognition, and specifically includes the following steps: S101: Performing global feature representation and local feature representation on the image and text respectively, and performing global feature labeling and local feature labeling.

[0031] Specifically, before feature extraction, image data needs to be preprocessed.

[0032] The preprocessing process includes: resizing all images to 384×128 pixels and normalizing them with predefined mean and standard deviation. During the training phase, data augmentation techniques such as random horizontal flipping, image padding and cropping, and random erasing are used to increase the diversity of training data. During the validation and testing phases, only image resizing, tensor conversion, and normalization are performed to ensure data consistency.

[0033] The purpose of preprocessing is to simulate the situation in which images and texts are not aligned in real scenes by randomly shuffling images of different identity IDs so that the corresponding text contents generate noise data.

[0034] In the process of image and text feature extraction, preferably, CLIP-ViT image encoder is used as the image backbone network to obtain the image The CLIP-Xformer text encoder is used to extract the given text characteristics.

[0035] The image extraction process includes: The image is segmented into a series of equal-sized and non-overlapping image blocks, and the segmented image blocks are converted into a one-dimensional tag sequence through a trainable linear projector. An additional CLS tag is added to the tag sequence, and the CLS position information is introduced. The processed tags are sent to the CLIP-ViT image encoder for processing.

[0036] Figure 2 A schematic diagram of the structure of an optimized network model for a noise-correlated text-image person re-identification method provided by an embodiment of the present invention, combined with Figure 2 , the specific process of image extraction is: First, the input pedestrian image Split the image into a series of equal-sized, non-overlapping blocks, The size of is H×W, and the number of image blocks obtained is indivual, Indicates the size of each image block. Then, the segmented image blocks are labeled and converted into a one-dimensional label sequence through a trainable linear projector. In order to capture the overall information of these labels and consider the mutual correlation between image blocks, an additional CLS label is added to the label sequence and the position information is introduced. The purpose of the CLS label block is to clarify the positional relationship between each label block.

[0037] Finally, the processed tags are sent to the CLIP-ViT image encoder for processing to simulate the interaction and correlation between different image blocks. The final extracted image features can be expressed as: , where N is the number of image blocks, represents the global image features, Represents local image features.

[0038] Combination Figure 2 , the text extraction process is as follows: First, the input text is tokenized using lowercase byte pair encoding, and special tags SOS and EOS are added to the tokenized text sequence to indicate the start and end of the sequence, respectively. In addition, a special tag CLS position information is introduced and the position information is merged into the tokenized text sequence.

[0039] Then, after the text tag sequence is input into the CLIP-Xformer text encoder, the CLIP-Xformer text encoder includes a self-attention mechanism (Self Attention) and a fully connected layer (Feed Forward), using the self-attention mechanism to capture the correlation between each part of the sequence.

[0040] Finally, the text representation is obtained: ,in represents the global text feature, Represents local text features.

[0041] S102: Using a dynamic instance-specific selection and correction method to divide image and text data.

[0042] It should be noted that the construction of the dynamic instance-specific selection and correction module (DISDR) includes: predicting the identity ID through the global features of the image and text to obtain the corresponding confidence. Then, the confidence values ​​of different modalities are used to select reliable sample pair instances, and the data is divided into two subsets: a clean subset and a noisy subset, and the associated labels are corrected. These subsets are divided using the confidence of the warm-up training phase to better mitigate confirmation bias. Finally, different optimization strategies are used to supervise each subset.

[0043] Specifically, they include: S1021: Obtain global feature representation and local feature representation of images and texts through the backbone network.

[0044] The global feature representation and local feature representation of the image are generated by the CLIP-ViT image encoder, and the global feature representation and local feature representation of the text are generated by the CLIP-Xformer text encoder.

[0045] S1022: Use the global features of the image and text to predict the identity ID and obtain the corresponding confidence level.

[0046] Specifically, the global image features extracted by the CLIP-ViT image encoder are: ,make represents the weight of a single-layer linear classifier, and K represents the total number of pedestrian categories. The corresponding identity ID is , Image Instance The category confidence of can be expressed as:

[0047] The above formula also expresses the deep neural network for image instances The memory strength of is the output of the linear classifier, For image instances The category confidence.

[0048] Similarly, the global text features extracted by the CLIP-Xformer text encoder are ,make represents the weight of a single-layer linear classifier, and K represents the total number of pedestrian categories. The corresponding identity ID is , Text Instance The category confidence of can be expressed as:

[0049] For text instance The category confidence.

[0050] When calculating confidence, dynamic thresholds need to be set. If a global threshold or class threshold is used when predicting identity, the result is still suboptimal. This embodiment adopts an instance-specific threshold strategy, that is, the threshold setting is only related to the sample itself. At the same time, since the memory strength of a deep neural network for a single instance of different labels increases with the increase of training time, its partitioning threshold should also be increased accordingly, so as to achieve accurate dynamic data partitioning. Therefore, the present invention divides each instance into / The dynamic threshold is defined as :

[0051] in, t represents the training round, is a parameter that controls the stability of the threshold, Refers to selecting the maximum confidence value among the image / text confidence values. Indicates the type of pedestrian ID. .

[0052] S1023: Using the consistency and difference of the global features of the image and text to predict the identity ID, the image and text data are divided into a clean subset and a noise subset, specifically: If the confidence values ​​of the corresponding image and text data are both higher than the corresponding threshold, they are classified as clean data; If only one of the confidence values ​​of the corresponding image and text data is higher than the corresponding threshold, it is classified as noise data; If the confidence values ​​of the corresponding image and text data are both lower than the threshold, they are classified as abnormal data and directly eliminated.

[0053] Specifically, in order to solve the noise correlation problem existing in the image-text based pedestrian re-identification task dataset, the embodiment of the present invention uses the consistency and difference of the two different modalities of image and text for identity ID prediction to divide the noise data into two subsets.

[0054] Figure 3 A schematic diagram of a model for subset division and correction labeling provided by an embodiment of the present invention, combined with Figure 3 , assuming the identity ID is Image examples The category confidence is , text instance The category confidence is For instances of two different modalities, if the corresponding confidence values ​​are both higher than their thresholds, they are classified as clean data. C . Mathematically expressed as:

[0055] That is, when the image instance The category confidence Above the image dynamic threshold , and the text instance The category confidence Above text dynamic threshold When , the image-text pair is divided into clean data C .

[0056] If only one of the two instances has a confidence value higher than the threshold, it is classified as noise data. N . Mathematically expressed as:

[0057] That is, when the image instance The category confidence Above the image dynamic threshold , or, a text instance The category confidence Above text dynamic threshold , and remove clean data that satisfies both C When , the image-text pair is divided into noise data N .

[0058] If the confidence values ​​corresponding to instances of two different modes are both lower than the threshold, they are classified as abnormal data and directly eliminated.

[0059] After the subset division, the corresponding matching labels are further recalibrated using the above division. For the i-th group of sample points, the correction label Defined as:

[0060] S103: construct a multi-granularity global feature learning module, input the local features of the image and text into the multi-granularity global feature learning module, and obtain multi-granularity global features.

[0061] Specifically, we construct a multi-granular global feature learning module (MGGFL). Since there are subtle visual differences between different pedestrians, it is crucial to fully mine the fine-grained information of images and texts to improve performance.

[0062] Different from the existing methods that mine fine-grained information by learning a set of local features, the embodiments of the present invention mine local information at different levels to learn multi-granular global features. The influence of important local tags is strengthened, and important tag information is selected to generate global features by weighting, rather than selecting all tags to generate global features.

[0063] Specifically include: S1031: Calculate the correlation weights between local features and global features, and select the top 2K tags with the highest scores from the local feature tags of the image and text according to the size of the correlation weights, respectively, to form a new tag sequence.

[0064] S1032: The first K tags are used to construct a high-level discriminant sequence, the last K tags are used to construct a middle-level discriminant sequence, and all local feature tags are used to construct a low-level discriminant sequence, thereby obtaining a set of multi-granularity global features.

[0065] The embodiment of the present invention selects local information according to the correlation between the local label and the global label to synthesize more representative global features.

[0066] The correlation between local and global labels can be obtained by the self-attention map of the last encoding block. Given an image The output self-attention map of , the correlation weight between global labeling and local labeling can be expressed as:

[0067] Similarly, for a given text , and its self-attention map is :

[0068] According to the size of the relevant weights, the top 2K tags with the highest scores are selected from the local tags of each modality to form a new tag sequence. Specifically, for the image Will be from Select 2K markers from the , the corresponding relevant weight is For text Will be from Select 2K markers from the , the corresponding relevant weight is .

[0069] For images and text , the embodiment of the present invention performs embedding transformation on the selected marker features to generate a set of multi-granularity global features. The selected marker sequence only contains key information. In order to fully mine the discriminative fine-grained information, the selected marker sequence is divided into two subsequences, corresponding to different discriminative granularities. The former is a high-level discriminative sequence with the first K information markers. , and the latter is a middle-level discriminant sequence with the remaining K information tags The specific conversion is as follows:

[0070] The global image features aggregated from all local markers are recorded as low-level discriminant sequences In summary, we get a set of multi-granularity global image feature sets .

[0071] Similarly, the multi-granularity global text feature set is expressed as .

[0072] S104: The network model is optimized through training of ID classification loss and similarity distribution matching loss, the loss of identity ID is calculated, and the similarity distribution matching loss is calculated for multi-granularity global features.

[0073] The embodiment of the present invention uses ID classification loss and similarity distribution matching loss to train and optimize the network model, effectively improving the matching accuracy of pedestrian re-identification based on text descriptions and providing the model with robustness to noise. Specifically: S1041: Using traditional cross entropy to calculate the loss of clean data .

[0074] S1042: Computing Losses on Noisy Data Using Generalized Cross Entropy .

[0075] S1043: The sum of the loss of clean data and noise data is taken as the identity loss : .

[0076] Calculation of identity ID loss. By introducing ID classification loss, the feature representations of the same identity in the same modality can be made closer, thereby promoting feature alignment between images and texts. For the data segmentation task, the data is divided into clean data and noisy data based on the consistency and difference of image and text data for identity ID prediction, and different regularization techniques are used for optimization.

[0077] For clean set C , using the traditional cross entropy (CE) loss :

[0078] in, represents the number of clean samples, represents the traditional cross entropy loss, is the pedestrian identification tag, and This is the output of the linear classifier.

[0079] The mean absolute error (MAE) loss is theoretically proven to be a noise-robust loss under certain assumptions. However, using the MAE loss may lead to difficult optimization. While the CE loss is easy to optimize, it is sensitive to label noise. Therefore, for noisy datasets, N , we choose to use a more robust regularization technique, namely the generalized cross entropy (GCE) loss, which combines the advantages of MAE and CE to deal with label noise.

[0080]

[0081] in, represents the number of noise samples, It is the pedestrian identification tag.

[0082] Confidence: , .

[0083] Finally, the ID loss is:

[0084] Calculation of similarity distribution matching loss. Similarity distribution matching loss (SDM loss) is a new cross-modal matching loss that embeds the cosine similarity distribution of N×N image-text pairs into the KL divergence, aiming to reduce the distance between positive pairs in the joint embedding space and increase the distance between negative pairs, thereby associating representations of different modalities.

[0085] The obtained multi-granularity global features are used to calculate the similarity distribution matching loss, specifically: Given a mini-batch of N image-text pairs, construct a set of image-text representation pairs ,in, and Represents the global features of images and texts, is the corrected matching label, when the image and text pair is a matching pair, , ,otherwise, .

[0086] Probability of matching image with text for:

[0087] in, is the temperature hyperparameter that controls the peak of the probability distribution, Represents the probability that the sample pair is a true match.

[0088] Further, calculate the SDM loss from image to text in the mini-batch:

[0089] in, is a minimum value to avoid numerical overflow problems. Represents the true matching probability of the sample pair. The specific calculation process is as follows:

[0090] Further, we get the bidirectional SDM loss:

[0091] in, Represents the matching from image to text direction, Represents the matching of text to image. SDM loss is achieved by calculating the matching probability of sample pairs and the true matching probability of sample points and using KL divergence to implement constraints.

[0092] It should be noted that and The calculation method of varies depending on the direction. Specifically, When calculating, the formula Need to be replaced with .

[0093] The embodiment of the present invention adopts an end-to-end approach for training, and the overall optimization goal is defined as: , where L is the overall optimization goal, For identity ID loss, is the bidirectional similarity distribution matching loss.

[0094] Furthermore, Including: SDM loss of low-level discriminant sequence, SDM loss of middle-level discriminant sequence, SDM loss of high-level discriminant sequence, the overall optimization goal is defined as:

[0095] in, Loss of ID identity, is the corrected matching label, is the SDM loss of the low-level discriminant sequence, is the SDM loss of the middle-level discriminant sequence, is the SDM loss for high-level discriminative sequences.

[0096] Furthermore, the rationality and effectiveness of the embodiments of the present invention can be verified through experiments. The specific experimental settings and result analysis are as follows: The present invention selects to evaluate the method on the RSTPReid dataset. For the convenience of explanation, the dataset contains 4101 identities, 20505 pictures and 41010 text contents. The experiment divides the dataset into a training set, a validation set and a test set, which contain 3701, 200 and 200 identities respectively.

[0097] The performance is measured using two widely used evaluation metrics for retrieval tasks, namely Rank-k and mAP. Meanwhile, R-Sum is used to measure the overall quality, which is defined as the sum of R-1, R-5, and R-10. The Rank-k metric measures the probability of finding at least one matching person image among the top k candidates given a text query. A higher Rank-k score indicates that the system is more effective in returning relevant results.

[0098] In order to make a comprehensive comparison, this embodiment selects two methods with different strategies: (1) Unimodal pre-training methods use image and text encoders pre-trained separately on unimodal datasets, including SSAN: a local matching method and IVT: a global matching method.

[0099] (2) Visual-language pre-training methods, which leverage prior knowledge from image-text corpora to directly transfer the original aligned dual encoder to text-image retrieval, including CFine and IRRA. They transfer a lot of knowledge from CLIP.

[0100] The performance comparison between the method proposed in the embodiment of the present invention and the SOTA method on the RSTPReid dataset is shown in Figure 4 Among them, R-1, 5, 10 is the abbreviation of Rank-1, 5, 10. By comparing the experimental results, we can see that: (1) Under the condition of no noise introduction, the method provided by the present invention is significantly better than the general TPReID method on the RSTPReid dataset. Specifically, the method provided by the embodiment of the present invention improves R-1, R-5, R-10 and mAP by 1.2%, 1.45%, 0.95% and 0.07% respectively compared with the baseline model IRRA, which proves the effectiveness of the method.

[0101] (2) When noise is introduced into the data, the performance of ordinary methods will drop significantly or become poorer as the noise rate increases.

[0102] In contrast, the method provided by the embodiment of the present invention also shows obvious advantages over ordinary methods, highlighting the necessity of designing a model that can effectively withstand noise. Compared with the baseline IRRA, the method provided by the present invention improves R-1 and mAP by 3.35%, 2.95%, 6.25%, and 6.93% respectively at different noise rates. In addition, especially when the noise rate is 50%, the method provided by the present invention shows more obvious advantages, emphasizing its ability to improve overall performance, robustness, and stability. This phenomenon strongly demonstrates the effectiveness of the method provided by the present invention for noise association learning, greatly improving the noise resistance of the baseline model.

[0103] In addition, the effectiveness of each module can be proved by designing ablation experiments. Specifically, the two key modules DISDR and MGGFL are eliminated. Figure 5 From the experimental results shown, the following conclusions can be drawn: (1) Comparing the results of No.1 and No.2, after introducing the MGGFL module on the basis of the baseline, the Rank-1 and mAP increased by 3% and 2.39% respectively. This phenomenon verifies the effectiveness of the MGGFL module, which can eliminate the markers of useless information (such as occlusion and background in the image) and minimize the negative impact of useless information to perform robust feature learning, improve the effectiveness of the model, and thus promote the model to focus on the most distinctive parts.

[0104] (2) Comparing the results of No.1 and No.3, after introducing the DISDR module on the basis of the Baseline, the Rank-1 and mAP increased by 2.5% and 2.15% respectively, indicating that the DISDR module can significantly improve the robustness of the model to noise correlation, proving that the DISDR module can effectively identify the erroneous corresponding sample data in the data.

[0105] (3) Comparing the four sets of experimental results, the complete method proposed in this embodiment can achieve the best performance, indicating that these two components play their respective roles in improving the robustness and effectiveness of the model.

[0106] The embodiment of the present invention provides a method for text-image pedestrian re-identification for noise association. First, dynamic instance-specific data selection and matching label correction are performed, and the consistency and difference of identity ID prediction between image and text are comprehensively considered, and the dynamic instance-specific threshold strategy is used to achieve accurate division of noise data. Secondly, the divided data is label-corrected for more effective supervised learning. In addition, in order to fully mine the fine-grained information of images and texts to distinguish different pedestrians, the embodiment of the present invention also designs a multi-granularity global feature learning module, which learns multi-granularity global features by exploring local information at different levels, selects the most representative TOP2K local tags through the correlation weight map of the coding block, generates more discriminative global features, and further refines the selected tag sequence to classify it into high-level and intermediate subsequences, and combines it with the original low-level global feature subsequence to obtain a set of multi-granularity global feature discrimination sequences. The embodiment of the present invention realizes effective recognition of alignment errors in image and text data, and significantly improves the performance of the model when processing noisy data, and improves the accuracy of matching.

[0107] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this embodiment is not limited here.

[0108] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for person re-identification from text to image with noise correlation, characterized in that: include: Perform global feature representation and local feature representation on images and texts respectively, and perform global feature labeling and local feature labeling; A dynamic instance-specific selection and correction method is used to partition image and text data; Construct a multi-granularity global feature learning module, input the local features of the image and text into the multi-granularity global feature learning module, and obtain multi-granularity global features; The network model is optimized through training of ID classification loss and similarity distribution matching loss, the loss of identity ID is calculated, and the similarity distribution matching loss is calculated for multi-granularity global features.

2. The method according to claim 1, characterized in that The global feature representation and local feature representation of the image and text respectively include: The image is segmented into a series of equal-sized and non-overlapping image blocks, and the segmented image blocks are converted into a one-dimensional tag sequence through a trainable linear projector. An additional CLS tag is added to the one-dimensional tag sequence, and the CLS position information is introduced. The processed CLS tag is sent to the CLIP-ViT image encoder for processing.

3. The method according to claim 2, characterized in that The method of performing global feature representation and local feature representation on images and texts respectively also includes: using lowercase byte pairs to encode, tokenizing the input text, adding special tags SOS and EOS to the tokenized text sequence to indicate the beginning and end of the sequence respectively, introducing a special tag CLS position information, and merging the position information into the tokenized text sequence.

4. The method according to claim 1, characterized in that: The method of using dynamic instance-specific selection and correction to partition image and text data includes: Obtain global and local feature representations of images and texts through the backbone network; Use the global features of the image and text to predict the identity ID and obtain the corresponding confidence level; The consistency and difference of identity ID prediction based on the global features of images and texts are used to divide image and text data into clean subsets and noisy subsets.

5. The method according to claim 4, characterized in that The method uses the consistency and difference of the global features of the image and text to predict the identity ID, and divides the image and text data into a clean subset and a noise subset, including: If the confidence values ​​of the corresponding image and text data are both higher than the corresponding threshold, they are classified as clean data; If only one of the confidence values ​​of the corresponding image and text data is higher than the corresponding threshold, it is classified as noise data; If the confidence values ​​of the corresponding image and text data are both lower than the corresponding thresholds, they are classified as abnormal data and directly eliminated.

6. The method according to claim 5, characterized in that The multi-granularity global feature learning module is constructed, and the local features of the image and the text are input into the multi-granularity global feature learning module to obtain the multi-granularity global features, including: Calculate the correlation weights between local features and global features, and select the top 2K tags with the highest scores from the local feature tags of the image and text according to the size of the correlation weights to form a new tag sequence; The first K markers are used to construct a high-level discriminant sequence, the last K markers are used to construct a mid-level discriminant sequence, and all local feature markers are used to construct a low-level discriminant sequence, thus obtaining a set of multi-granularity global features.

7. The method according to claim 6, characterized in that The network model is optimized by training the ID classification loss and the similarity distribution matching loss, including the calculation of the loss of the identity ID: Using traditional cross entropy to calculate the loss of clean data ; Computing the loss for noisy data using generalized cross entropy ; The sum of the loss of clean data and noisy data is taken as the identity loss : .

8. The method according to claim 7, characterized in that The network model is optimized by training the ID classification loss and the similarity distribution matching loss, and also includes calculating the similarity distribution matching loss for multi-granularity global features, specifically: Calculate the similarity distribution matching loss when matching from image to text in multi-granular global features ; Calculate the similarity distribution matching loss when matching from text to image in multi-granular global features ; Bidirectional Similarity Distribution Matching Loss for: .

9. The method according to claim 8, characterized in that The network model is optimized by training the ID classification loss and the similarity distribution matching loss. The overall optimization goal is: ; Among them, L is the overall optimization goal, For identity ID loss, is the bidirectional similarity distribution matching loss.

Images