Text pedestrian re-identification method based on attention screening mask strategy
By employing an attention-based masking strategy, this method addresses the issues of insufficient modal interaction and coarse text description in text-to-image person re-identification. It achieves global feature alignment and local feature interaction, thereby improving recognition accuracy and computational efficiency.
Patent Information
- Application Number
- CN202410935990.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-12
- Publication Date
- 2024-11-12
AI Technical Summary
Existing text-to-image pedestrian re-identification methods suffer from insufficient modal interaction and coarse text descriptions, resulting in high false recognition rates when images are heavily occluded or have low resolution, and they are unable to effectively identify pedestrians when visual images are lacking.
We adopt an attention-based masking strategy, which uses an attention-based filtering mechanism to filter features from image and text modalities. We then use the Transformer attention mechanism to filter out important features for masking interaction. By combining global feature alignment and local feature interaction, we construct a multi-granular text-to-image person re-identification network.
It improves the accuracy of text-to-image person re-identification, reduces computational costs, achieves optimal performance on multiple evaluation metrics, and enhances recognition capabilities in the absence of visual images.
Smart Images

Figure CN118942109A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of pedestrian re-identification, and in particular relates to a text pedestrian re-identification method based on an attention screening mask strategy. Background Art
[0002] Person Re-identification (ReID) usually takes people of the same identity as input, and then retrieves people of the same identity from a set of person images captured by different cameras.
[0003] Traditional single-modal person re-identification usually extracts image features, then uses the image features as input, and calculates the distance between the input image features and the image features in the image library to achieve the retrieval effect. However, single-modal person re-identification will have a greater impact on the prediction effect of the model when the image is severely occluded or the image resolution is low, which will greatly increase the misrecognition rate. In addition, single-modal person re-identification requires at least one image as a query, which is difficult to obtain in some scenarios.
[0004] In contrast, text-to-image person re-identification solves the problems of traditional single-modal person re-identification. When visual images of the queried pedestrian are unavailable and only text descriptions are available, a text-to-image person re-identification algorithm must be used. Text and image person re-identification can help security personnel identify specific people based on eyewitness descriptions or text information in reports. In search and rescue operations, text-to-image person re-identification can help quickly identify and locate missing persons through descriptions provided by eyewitnesses, improving search efficiency.
[0005] At present, person re-identification from text to image can be divided into two categories: non-cross-modal interaction methods and cross-modal interaction methods. Non-cross-modal interaction methods mainly project the extracted text modality and image modality into the same space, and match the two different modalities through feature alignment. This matching method narrows the gap between the two modalities by designing a clever loss function, but ignores the noteworthy details in the two modalities. Cross-modal interaction methods mainly pass the image features and text features through feature extractors, and then interact the features through various attention mechanisms to achieve the effect of modal alignment. This method pays attention to the local detail features between the two modalities, but at the same time increases the computational cost. Summary of the invention
[0006] The purpose of the present invention is to address the problems of insufficient modal interaction and rough text description in text-to-image pedestrian re-identification methods based on local matching, and to provide a text-to-image multimodal pedestrian re-identification method based on image and text bidirectional masking under an attention screening mechanism. By performing attention screening on the text modality and the image modality, the higher the attention score, the greater the impact of the corresponding mark on the model. The features of the two screened modalities are then masked by the attention score, and the masked features are fully interacted with the unmasked features, thereby improving the accuracy of the text-to-image pedestrian re-identification algorithm.
[0007] The technical solution adopted by the present invention is:
[0008] The present invention implements a text pedestrian re-identification method based on an attention screening mask strategy, which mainly includes the following steps:
[0009] Step 1: Collect and preprocess the text-to-image person re-identification dataset:
[0010] Load widely used text-to-image person re-identification datasets, mainly including the CUHK-PEDES dataset and the ICFG-PEDES dataset, and then perform data enhancement by random horizontal flipping, image grayscale value changes, etc. to obtain a batch of pedestrian image training sets where x i represents the image of the i-th pedestrian, and B represents the total number of images;
[0011] Load the description text for the pedestrian where t i represents the i-th pedestrian image x i A text description of
[0012] Step 2: Construct a text-to-image person re-identification network and loss function based on the attention screening mask strategy. The network mainly includes an image encoder, a text encoder, an attention module, and an image-text mask module.
[0013] Step 2.1: Since the Contrastive Language-Image Pre-training (CLIP) model has excellent results in cross-modal image-text contrast learning, the image encoder and text encoder pre-trained by the CLIP model are used to extract features from the input image and text;
[0014] Step 2.1.1: For an input image I∈R H×W×C, where H, W, and C represent the height, width, and number of channels of the image, respectively. The image features are extracted using the ViT (Vision Transformer) pre-trained by CLIP. First, the image I is divided into N = H × W / P by the pre-trained (ViT) 2 patches, where P represents the length of a patch, and then adds a learnable marker [CLS] at the beginning of the patch. v To represent the global features of the image, the image features obtained after ViT are V = {v cls ,v1,...,v N}, where v cls Represents the global features of the image, {v1,...,v N} represents the characteristics of each patch;
[0015] Step 2.1.2: For the text description T of the input image, use the BERT model pre-trained by CLIP to extract features. First, pad the text descriptions of different lengths to make the text padding a fixed length. The text features after BERT output are T = {t cls ,t1,...,t M}, where M represents the length of the text, t cls Represents the global feature embedding of the sentence, {t1,...,t M} is the feature of local words in the sentence;
[0016] Step 2.2: After obtaining the feature embedding of the image and text, the global features of the image and the global features of the text are further aligned in the same latent space, i.e. global alignment. In a batch, the global features F of each image can be i v and text global feature F i t Constructing an image-text pair where Y i,j Represents the real matching label. If Y i,j =1 means Belong to the same matching pair. Through the description, we can get the formula:
[0017]
[0018] Where t represents the temperature hyperparameter, which is used to control the peak value of the probability distribution, sim(u,v)=u T v / ||u||||v|| represents the cosine similarity between vectors u and v, that is, the normalized dot product of u and v, P i,j Represents image-text pairs The ratio of the cosine similarity score between them to the total score;
[0019] Get score P i,j Afterwards, calculate P i,j With the true label Q i,j The divergence between them can be obtained, and the loss function from image to text is:
[0020]
[0021] ε is a decimal to avoid numerical problems; similarly, exchanging images and texts gives the text-to-image loss function L t2i , and finally the final global loss function is:
[0022] L gls =L i2t +L t2i
[0023] Step 2.3: After the image and text are sent to the encoder respectively, they will pass through a series of self-attention layers, and finally the encoder will output the attention score of each tag. The larger the attention score, the greater the contribution of the tag and the more important it is to the model task. Therefore, the first 2K tags with the highest scores are selected according to the attention score. The first 2K tags are masked and then local feature interaction is performed. Specifically, for an input image I, 2K v Local feature labels are selected from image features V = {v cls ,v1,...,v N}, for a piece of text T, 2K t Local features are selected from text features T = {t cls ,t1,...,t M}, and finally get the first 2K after screening v The image features are First 2K t The text features are
[0024] Step 2.4: After obtaining the filtered local modal features, perform a random mask operation on the first 2K filtered modal features;
[0025] Step 2.4.1: For text features, first mask the first 2K tokens after screening. The masking rate and parameters are the same as those set by BERT. The masked text features are Then the unmasked image and the masked text features are sent to a multimodal interaction layer. In order to allow the masked text and the unmasked image to fully interact, the masked text representation As query (Q), image features As key (K) and value (V), the two modalities are cross-attention interacted to obtain the following formula:
[0026]
[0027] in represents the feature representation after the interaction between the unmasked image and the masked text, LN represents layer normalization, MCA represents multi-head cross attention, d represents the embedding dimension of the masked tag, and for each masked feature Use a multi-layer perceptron to predict the probability of the original tag, and finally get the loss function of the text mask:
[0028]
[0029] Among them, M t represents the length of the masked text, V represents the text vocabulary size, and Y i,j represents the true label, is the prediction probability of the multi-layer perceptron, MLP is the multi-layer perceptron;
[0030] Step 2.4.2: For image masking, the operation is similar to text masking. The first 2K tags are filtered and then the MAE strategy is used to mask them. The masked image features are then As query (Q), unmasked text features As key (K) and value (V), the two modalities are multi-headed cross attention to obtain the unmasked text and masked image features Then the obtained masked image features Use a multi-layer perceptron to predict the probability of the original tag, and finally get the loss function of the image mask:
[0031]
[0032]
[0033] in is the predicted value of the multi-layer perceptron, M V is the masked image label, is the original image label, Y i,j Represents the true label, MLP is a multi-layer perceptron;
[0034] Step 2.5: For the local feature multimodal interaction module, the final loss function is the mask loss function of the text plus the image loss function. The two functions are added together to perform feature interaction on the two mask modules.
[0035] Finally, a multi-granular text-to-image person re-identification loss function based on global feature alignment and local feature interaction is constructed:
[0036] L=L id +L gls +αL mim +βL mlm
[0037] Where L id is the common identity loss function in person re-identification, L gls is the loss function of the global modality alignment designed above, L mim Image mask loss function, L mlm is the text mask loss function, α and β are the hyperparameters controlling the two mask modules respectively;
[0038] Step 3: Use the training data to train the constructed person re-identification network. During the training process, the CLIP-ViT-B / 16 model pre-trained by CLIP is used as the image encoder, and three data enhancement methods are used: random horizontal flipping, random cropping with padding, and random erasing. The size of all input images is 384x 128, the maximum length of the text sequence is 77, the selection rate in the first 2K feature selection is 0.6, the batch size is selected as 64, and the training is 60 epochs. The temperature hyperparameter is set to 0.015; finally, the trained person re-identification network is obtained;
[0039] Step 4: Use the trained person re-identification network to perform person re-identification.
[0040] Beneficial effects of the present invention:
[0041] Existing text-to-image person re-identification models either pay too much attention to local feature information or only select global interaction and ignore local information. The present invention provides a multi-granularity text-to-image person re-identification algorithm based on global feature alignment and local feature interaction. By introducing an improved global feature loss function, a rough global feature interaction is performed on the image modality and the text modality, thereby achieving the effect of roughly aligning the two modalities; a mask module is introduced to perform bidirectional interaction on the text and image modalities, which can fully utilize the local feature information of the two modalities without introducing a huge amount of calculation; at the same time, the Transformer attention mechanism is used to screen the features of the two modalities. The larger the attention score, the greater the influence of the tag on the feature. The first 2K tags are screened for mask interaction, thereby increasing the mask interaction capability to a greater extent; in the benchmark data set of text-to-image person re-identification, the present invention achieves the best in multiple evaluation indicators. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1It is the overall flow chart of the method of the present invention; DETAILED DESCRIPTION
[0043] The technical solution of the present invention is described in detail below with reference to the accompanying drawings.
[0044] The technical solution adopted by the present invention is:
[0045] The present invention implements a text pedestrian re-identification method based on an attention screening mask strategy, which mainly includes the following steps:
[0046] Step 1: Collect and preprocess the text-to-image person re-identification dataset:
[0047] Load widely used text-to-image person re-identification datasets, mainly including the CUHK-PEDES dataset and the ICFG-PEDES dataset, and then perform data enhancement by random horizontal flipping, image grayscale value changes, etc. to obtain a batch of pedestrian image training sets where x i represents the image of the i-th pedestrian, and B represents the total number of images;
[0048] Load the description text for the pedestrian where t i represents the i-th pedestrian image x i A text description of
[0049] Step 2: Construct a text-to-image person re-identification network based on the attention screening mask strategy, which mainly includes an image encoder, a text encoder, an attention module, and an image-text mask module;
[0050] Step 2.1: Since the Contrastive Language-Image Pre-training (CLIP) model has excellent results in cross-modal image-text contrast learning, the image encoder and text encoder pre-trained by the CLIP model are used to extract features from the input image and text;
[0051] Step 2.1.1: For an input image I∈R H×W×C , where H, W, and C represent the height, width, and number of channels of the image, respectively. The image features are extracted using the Vision Transformer (ViT) pre-trained by CLIP. First, the image I is divided into N = H × W / P by the pre-trained (ViT) 2 patches, where P represents the length of a patch, and then adds a learnable marker [CLS] at the beginning of the patch. v To represent the global features of the image, the image features obtained after ViT are V = {v cls,v1,...,v N}, where v cls Represents the global features of the image, {v1,...,v N} represents the characteristics of each patch;
[0052] Step 2.1.2: For the text description T of the input image, use the BERT model pre-trained by CLIP to extract features. First, pad the text descriptions of different lengths to make the text padding a fixed length. The text features after BERT output are T = {t cls ,t1,...,t M}, where M represents the length of the text, t cls Represents the global feature embedding of the sentence, {t1,...,t M} is the feature of local words in the sentence;
[0053] Step 2.2: After obtaining the feature embedding of the image and text, the global features of the image and the global features of the text are further aligned in the same latent space, i.e. global alignment. In a batch, the global features F of each image can be i v and text global feature F i t Constructing an image-text pair where Y i,j Represents the real matching label. If Y i,j =1 means Belong to the same matching pair. Through the description, we can get the formula:
[0054]
[0055] Where t represents the temperature hyperparameter, which is used to control the peak value of the probability distribution, sim(u,v)=u T v / ||u|||v|| represents the cosine similarity between vectors u and v, that is, the normalized dot product of u and v, P i,j Represents image-text pairs The ratio of the cosine similarity score between them to the total score;
[0056] Get score P i,j Afterwards, calculate P i,j With the true label Q i,j The divergence between them can be obtained, and the loss function from image to text is:
[0057]
[0058] Similarly, exchanging the image and text can obtain the text-to-image loss function L t2i , and finally the final global loss function is:
[0059] L gls =L i2t +L t2i
[0060] Step 2.3: After the image and text are sent to the encoder respectively, they will pass through a series of self-attention layers, and finally the encoder will output the attention score of each tag. The larger the attention score, the greater the contribution of the tag and the more important it is to the model task. Therefore, the first 2K tags with the highest scores are selected according to the attention score. The first 2K tags are masked and then local feature interaction is performed. Specifically, for an input image I, 2K v Local feature labels are selected from image features V = {v cls ,v1,...,v N}, for a piece of text T, 2K t Local features are selected from text features T = {t cls ,t1,...,t M}, and finally get the first 2K after screening v The image features are First 2K t The text features are
[0061] Step 2.4: After obtaining the filtered local modal features, perform a random mask operation on the first 2K filtered modal features;
[0062] Step 2.4.1: For text features, first mask the first 2K tokens after screening. The masking rate and parameters are the same as those set by BERT. The masked text features are Then the unmasked image and the masked text features are sent to a multimodal interaction layer. In order to allow the masked text and the unmasked image to fully interact, the masked text representation As query (Q), image features As key (K) and value (V), the two modalities are cross-attention interacted to obtain the following formula:
[0063]
[0064] in represents the feature representation after the interaction between the unmasked image and the masked text, LN represents layer normalization, MCA represents multi-head cross attention, d represents the embedding dimension of the masked tag, and for each masked feature Use a multi-layer perceptron to predict the probability of the original tag, and finally get the loss function of the text mask:
[0065]
[0066] Among them, M t represents the length of the masked text, V represents the text vocabulary size, and Y i,j represents the true label, is the prediction probability of the multi-layer perceptron, MLP is the multi-layer perceptron;
[0067] Step 2.4.2: For image masking, the operation is similar to text masking. The first 2K tags are filtered and then the MAE strategy is used to mask them. The masked image features are then As query (Q), unmasked text features As key (K) and value (V), the two modalities are multi-headed cross attention to obtain the unmasked text and masked image features Then the obtained masked image features Use a multi-layer perceptron to predict the probability of the original tag, and finally get the loss function of the image mask:
[0068]
[0069] in is the predicted value of the multi-layer perceptron, M V is the masked image label, is the original image label, Y i,j Represents the true label, MLP is a multi-layer perceptron;
[0070] Step 2.5: For the local feature multimodal interaction module, the final loss function is the mask loss function of the text plus the image loss function. The two functions are added together to perform feature interaction on the two mask modules.
[0071] Finally, a multi-granular text-to-image person re-identification loss function based on global feature alignment and local feature interaction is constructed:
[0072] L=L id +L gls +αL mim +βL mlm
[0073] Where L id is the common identity loss function in person re-identification, L gls is the loss function of the global modality alignment designed above, L mim Image mask loss function, L mlm is the text mask loss function, α and β are the hyperparameters controlling the two mask modules respectively;
[0074] Step 3: Use the training data to train the constructed person re-identification network. During the training process, the CLIP-ViT-B / 16 model pre-trained by CLIP is used as the image encoder, and three data enhancement methods are used: random horizontal flipping, random cropping with padding, and random erasing. The size of all input images is 384x 128, the maximum length of the text sequence is 77, the selection rate in the first 2K feature selection is 0.6, the batch size is selected as 64, and the training is 60 epochs. The temperature hyperparameter is set to 0.015; finally, the trained person re-identification network is obtained;
[0075] Step 4: Use the trained person re-identification network to perform person re-identification.
[0076] The beneficial effects of the present invention are:
[0077] Existing text-to-image person re-identification models either pay too much attention to local feature information or only select global interaction and ignore local information. The present invention provides a multi-granularity text-to-image person re-identification algorithm based on global feature alignment and local feature interaction. By introducing an improved global feature loss function, a rough global feature interaction is performed on the image modality and the text modality, thereby achieving the effect of roughly aligning the two modalities; a mask module is introduced to perform bidirectional interaction on the text and image modalities, which can fully utilize the local feature information of the two modalities without introducing a huge amount of calculation; at the same time, the Transformer attention mechanism is used to screen the features of the two modalities. The larger the attention score, the greater the influence of the tag on the feature. The first 2K tags are screened for mask interaction, thereby increasing the mask interaction capability to a greater extent; in the benchmark data set of text-to-image person re-identification, the present invention achieves the best in multiple evaluation indicators.
[0078] According to the above implementation scheme, the proposed text-to-image person re-identification algorithm based on attention screening mask is evaluated, trained and tested on the public dataset CUHK-PEDES, and compared with other text-to-image person re-identification models, and evaluated using three evaluation indicators: Rank-1, Rank-5, Rank-10 and mAP;
[0079] Table 1 Comparison of the present invention with other text-to-image person re-identification algorithms, with bold indicating the first.
[0080]
[0081] Table 1 shows the comparison results between the present invention and other methods. From the results, it can be seen that the probability of the algorithm of the present invention to retrieve the correct pedestrian in the first time is the highest, which is 73.87, which is 0.49 higher than IRRA. At the same time, the mAP value is also increased by 0.21, which shows that the effect of the algorithm is greatly improved, which is attributed to the effect of the image-text mask module under the introduced attention mechanism screening mechanism.
Claims
1. A text person re-identification method based on attention screening mask strategy, characterized in that: The following steps are involved: Step 1: Load the known text-to-image person re-identification dataset, then perform data enhancement on the image by randomly flipping the image horizontally and changing the image grayscale value to obtain the pedestrian image training set. where x i represents the image of the i-th pedestrian, and B represents the total number of images; Load the description text for the pedestrian where t i represents the i-th pedestrian image x i A text description of Step 2: Construct a text-to-image person re-identification network and loss function based on the attention screening mask strategy. The network includes an image encoder, a text encoder, an attention module, and an image-text mask module. The network processes the input data as follows: Step 2.1: Use the image encoder and text encoder pre-trained by the CLIP model to extract features from the input image and text. The specific method is as follows: Step 2.1.1: For an input image I∈R H×W×C , where H, W, and C represent the height, width, and number of channels of the image, respectively. The CLIP pre-trained visual transformer ViT is used to extract image features. First, the image I is divided into N = H × W / P by the pre-trained ViT. 2 patches, where P represents the length of a patch, and then adds a learnable marker [CLS] at the beginning of the patch. v To represent the global features of the image, the image features obtained after ViT are V = {v cls ,v1,...,v N }, where v cls Represents the global features of the image, {v1,...,v N } represents the characteristics of each patch; Step 2.1.2: For the text description T of the input image, the BERT model pre-trained by CLIP is used for feature extraction. First, the text descriptions of different lengths are padded to make the text padded to a fixed length. The text feature after output by BERT is defined as T = {t cls ,t1,...,t M }, where M represents the length of the text, t cls Represents the global feature embedding of the sentence, {t1,...,t M } is the feature of local words in the sentence; Step 2.2: After the image and text feature embedding is obtained, the global features of the image and the global features of the text are aligned in the same latent space, i.e., global alignment. In one batch, the global features of each image are aligned. and global features of the text Constructed as an image-text pair where Y i,j Represents the true matching label. If Y i,j =1 means Belong to the same matching pair; the formula is obtained by description: Where t represents the temperature hyperparameter, which is used to control the peak value of the probability distribution, sim(u,v)=u T v / ||u||||v|| represents the cosine similarity between vectors u and v, that is, the normalized dot product of u and v, P i,j Represents image-text pairs The ratio of the cosine similarity score between them to the total score; Get score P i,j Afterwards, calculate P i,j With the true label Q i,j The divergence between them gives the loss function from image to text: Similarly, swap the image and text to get the text-to-image loss function L t2i , and finally the final global loss function is: L gls =L i2t +L t2i Step 2.3: After the image and text are sent to the encoder respectively, they will pass through a series of self-attention layers, and finally the encoder will output the attention score of each tag. The larger the attention score, the greater the contribution of the tag and the more important it is to the model task. Therefore, the first 2K tags with the highest scores are selected according to the attention score, and the first 2K tags are masked and then local feature interaction is performed. Specifically, for an input image I, 2K v Local feature labels are selected from image features V = {v cls ,v1,...,v N }, for a piece of text T, 2K t Local features are selected from text features T = {t cls ,t1,...,t M }, and finally get the first 2K after screening v The image features are First 2K t The text features are Step 2.4: After obtaining the filtered local modal features, perform a random mask operation on the first 2K filtered modal features: Step 2.4.1: For text features, first mask the first 2K tokens after screening. The masking rate and parameters are the same as those set by BERT. The text features after masking are Then the unmasked image and the masked text features are sent to a multimodal interaction layer together; in order to allow the masked text and the unmasked image to fully interact, the masked text representation As query (Q), image features As key (K) and value (V), the two modalities are cross-attention interacted to obtain: in represents the feature representation after the interaction between the unmasked image and the masked text, LN represents layer normalization, MCA represents multi-head cross attention, d represents the embedding dimension of the masked tag, and for each masked feature Use a multi-layer perceptron to predict the probability of the original tag, and finally get the loss function of the text mask: Among them, M t represents the length of the masked text, V represents the text vocabulary size, and Y i,j represents the true label, is the prediction probability of the multi-layer perceptron, MLP is the multi-layer perceptron; Step 2.4.2: For image masking, filter the first 2K tags and then use the MAE strategy to mask them, and then use the masked image features As query (Q), unmasked text features As key (K) and value (V), the two modalities are multi-headed cross attention to obtain the unmasked text and masked image features Then the obtained masked image features Use a multi-layer perceptron to predict the probability of the original tag, and finally get the loss function of the image mask: in is the predicted value of the multi-layer perceptron, M V is the masked image label, l is the original image label, Y i,j Represents the true label, MLP is a multi-layer perceptron; Step 2.5: For the local feature multimodal interaction module, the final loss function is the mask loss function of the text plus the loss function of the image. The two functions are added together to perform feature interaction on the two mask modules. Finally, the multi-granularity text-to-image person re-identification loss function based on global feature alignment and local feature interaction is constructed as: L=L id +L gls +αL mim +βL mlm Where L id is the common identity loss function in person re-identification, L gls is the loss function of the global modality alignment designed above, L mim Image mask loss function, L mlm is the text mask loss function, α and β are the hyperparameters controlling the two mask modules respectively; Step 3: Use the training data to train the constructed person re-identification network, and finally obtain a trained person re-identification network; Step 4: Use the trained person re-identification network to perform person re-identification.
Citation Information
Patent Citations
Text-image cross-modal pedestrian re-identification method based on mask matching
CN117789248A
Cited By
Image restoration method and system based on dynamic decomposition and fusion
CN119540100A
Image restoration method and system based on dynamic decomposition and fusion
CN119540100B
Multi-modal target re-identification method based on text inversion and collaborative deformable aggregation
CN120318760A
