Pedestrian re-identification method based on similarity guidance and mismatching feature enhancement

Abstract

The invention discloses a pedestrian re-identification method based on similarity guidance and mismatching feature enhancement, and relates to the technical field of intelligent identification. The invention provides a similarity guide masking strategy, the strategy enhances the use of image information in a mask language modeling process so as to promote stronger cross-modal interaction, and different from random masking, the method guides the attention of the model to a more relevant image text corresponding relation so as to realize better fine-grained alignment; and a novel mismatching feature emphasis module is introduced, and the module innovatively utilizes mismatching features to improve the retrieval accuracy. The previous work mainly focuses on matched image text features, our mismatched feature emphasis module explores the importance of mismatched features, and these features play a key role in distinguishing visually similar individuals.

Description

Technical Field

[0001] The present invention relates to the field of intelligent recognition technology, and in particular to a pedestrian re-recognition method based on similarity guidance and mismatch feature enhancement. Background Art

[0002] Text-to-Image Person Re-identification (TI-ReID) is a technology that aims to retrieve target pedestrians from image databases using text descriptions. The technology was first proposed by Li et al. [1] in 2017, and the first benchmark dataset CUHK-PEDES was released. The core challenge of TI-ReID is how to achieve cross-modal alignment between images and texts and overcome modality heterogeneity.

[0003] The following references are cited:

[0004] [1] Li, S., Xiao, T., Li, H., Zhou, B., Yue, D., Wang, X., 2017. Person search with natural language description, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1970–1979.

[0005] [2]Chen, Y., Huang, R., Chang, H., Tan, C., Xue, T., Ma, B., 2021. Cross-modal knowledge adaptation for language-based person search. IEEE Transactions onImage Processing 30, 4057–4069.

[0006] [3] Zheng, Z., Zheng, L., Garrett, M., Yang, Y., Xu, M., Shen, YD, 2020b. Dual-path convolutional image-text embeddings with instance loss. ACM Transactionson Multimedia Computing, Communications, and Applications (TOMM) 16, 1–23.

[0007] [4]Zhang,Y.,Lu,H.,2018.Deep cross-modal projection learning forimage-text matching,in:Proceedings of the European conference on computervision(ECCV),pp.686–701.

[0008] [5]Li,S.,Xiao,T.,Li,H.,Zhou,B.,Yue,D.,Wang,X.,2017.Person search withnatural language description,in:Proceedings of the IEEE conference oncomputer vision and pattern recognition,pp.1970–1979.

[0009] [6]Chen,T.,Xu,C.,Luo,J.,2018b.Improving text-based person search byspatial matching andadaptive threshold,in:2018 IEEE Winter Conference onApplications of Computer Vision(WACV),IEEE.pp.1879–1887.

[0010] [7]Wei,D.,Zhang,S.,Yang,T.,Liu,J.,2023.Calibrating cross-modalfeature for text-based person searching.arXiv preprint arXiv:2304.02278.

[0011] [8]Zha,Z.J.,Liu,J.,Chen,D.,Wu,F.,2020.Adversarial attribute-textembedding for person search with natural language query.IEEE Transactions onMultimedia 22,1836–1846.

[0012] [9]Sarafianos,N.,Xu,X.,Kakadiaris,I.A.,2019.Adversarialrepresentation learning for text-to-image matching,in:Proceedings of theIEEE / CVF international conference on computer vision,pp.5814–5824.

[0013]

[10] Gao,C.,Cai,G.,Jiang,X.,Zheng,F.,Zhang,J.,Gong,Y.,Peng,P.,Guo,X.,Sun,X.,2021.Contextual non-local alignment over full-scale representation fortext-based person search.arXiv preprint arXiv:2101.03036.

[0014]

[11] Ding,Z.,Ding,C.,Shao,Z.,Tao,D.,2021.Semantically self-alignednetwork for text-to-image part-aware person re-identification.arXiv preprintarXiv:2107.12666.

[0015]

[12] Wu,Y.,Yan,Z.,Han,X.,Li,G.,Zou,C.,Cui,S.,2021.Lapscore:language-guided person search via color reasoning,in:Proceedings of the IEEE / CVFInternational Conference on Computer Vision,pp.1624–1633.

[0016]

[13] Wang,Z.,Fang,Z.,Wang,J.,Yang,Y.,2020.Vitaa:Visual-textualattributes alignment in person search by natural language,in:Computer Vision–ECCV 2020:16th European Conference,Glasgow,UK,August 23–28,2020,Proceedings,Part XII 16,Springer.pp.402–420.

[0017]

[14] Aggarwal,S.,Radhakrishnan,V.B.,Chakraborty,A.,2020.Text-basedperson search via attribute-aided matching,in:Proceedings of the IEEE / CVFwinter conference on applications of computer vision,pp.2617–2625.

[0018]

[15] Zhu,A.,Wang,Z.,Li,Y.,Wan,X.,Jin,J.,Wang,T.,Hu,F.,Hua,G.,2021.Dssl:Deep surroundings-person separation learning for text-based personretrieval,in:Proceedings of the 29th ACM International Conference onMultimedia,pp.209–217.

[0019]

[16] Niu,K.,Huang,Y.,Ouyang,W.,Wang,L.,2020.Improving description-based person re-identification by multi-granularity image-textalignments.IEEE Transactions on Image Processing 29,5542–5556.

[0020]

[17] Jing,Y.,Si,C.,Wang,J.,Wang,W.,Wang,L.,Tan,T.,2020.Pose-guidedmulti-granularity attention network for text-based person search,in:Proceedings of the AAAI Conference on Artificial Intelligence,pp.11189–11196.

[0021]

[18] Zheng,K.,Liu,W.,Liu,J.,Zha,Z.J.,Mei,T.,2020a.Hierarchical gumbelattention network for text-basedperson search,in:Proceedings of the 28th ACMInternational Conference on Multimedia,pp.3441–3449.

[0022]

[19] Ji,Z.,Hu,J.,Liu,D.,Wu,L.Y.,Zhao,Y.,2022.Asymmetric cross-scalealignment for text-based person search.IEEE Transactions on Multimedia.

[0023]

[20] Jiang,D.,Ye,M.,2023.Cross-modal implicit relation reasoning andaligning for text-to-image person retrieval,in:Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition,pp.2787–2797.

[0024]

[21] Taylor,W.L.,1953.“cloze procedure”:A new tool for measuringreadability.Journalism quarterly 30,415–433.

[0025]

[22] Shi, D., 2024. Transnext: Robust foveal visual perception for vision transformers, in: Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 17773–17783.

[0026] Existing TI-ReID technologies mainly face the following challenges and limitations:

[0027] 1. Early methods [2-9] mainly focused on the global matching of images and text. For example, Zhang et al. [4] proposed cross-modal projection classification loss and cross-modal projection matching loss to align global image and text features. Although these methods are intuitive and efficient, they lack the ability to model more fine-grained and complex image-text interactions and may not be able to fully capture the detailed correspondence between visual and text information.

[0028] 2. To address the limitations of global matching, subsequent studies [10-19] turned to local matching strategies, aiming to capture fine-grained image-text correspondences. For example, Niu et al.

[16] proposed a multi-granularity alignment model to achieve image-text alignment at three different granularities. However, these methods usually rely on region-based comparisons, which are computationally expensive.

[0029] 3. Recently, Jiang et al.

[20] introduced the Masked Language Modeling (MLM)

[21] task to implicitly model image-text correspondence, reducing computational overhead and avoiding additional supervision. However, the random mask strategy in MLM lacks deep involvement of image information, resulting in weak interaction between image and text.

[0030] 4. Existing methods mainly focus on matching image-text features, ignoring the important role of mismatching features in distinguishing visually similar individuals.

[0031] Therefore, it is necessary to propose a new solution to the problem between Question 3 and Question 4. Summary of the invention

[0032] The purpose of the present invention is to provide a pedestrian re-identification method based on similarity guidance and mismatch feature enhancement, through the similarity-guided multimodal interaction module: guide the model to focus on more relevant image-text correspondences, improve the fine-grained image-text alignment effect, and through the mismatch feature emphasis module: utilize mismatch features in TI-ReID tasks to improve the ability to distinguish between similar individuals, so as to solve the technical problems raised in the background technology.

[0033] To achieve the above object, the present invention provides the following technical solution: a pedestrian re-identification method based on similarity guidance and mismatch feature enhancement, comprising at least the following steps:

[0034] S1: Image and text feature extraction, using the pre-trained encoder in the contrastive language-image pre-trained model to extract image and text feature maps, that is, using the 12-layer Vision Transformer as the image encoder and the 12-layer Transformer architecture as the text encoder. The contrastive language-image pre-trained model is CLIP and the Vision Transformer is ViT.

[0035] S2: Similarity guides multimodal interaction, using the correlation between image and text features to select text features to be masked. Text features that are highly correlated with image features are more likely to be masked. This strategy encourages the use of image information in the MLM process, thereby enhancing cross-modal interaction between images and texts.

[0036] S3: mismatch feature enhancement is performed through a mismatch feature emphasis module, aiming to reduce the false positive rate and improve the overall retrieval accuracy. The mismatch feature emphasis module is MFE;

[0037] S4: Perform training and inference, using a multi-task learning approach to optimize four loss functions in four parallel components, including an SGMI module, an MFE module, an ID loss, and an SDM loss. The SGMI module calculates the MLM loss, and the MFE module contains the ITC loss. At the same time, the ID loss and the SDM loss are calculated independently. During the inference process, a similarity metric S is calculated for each text query and image candidate pair. This calculation uses the cosine similarity method to quantify the correlation between text and image features. Subsequently, S is used to establish a ranked list of individuals in the image library, and the highest-ranked image matches that are closest to the given text description are retrieved and presented.

[0038] Furthermore, the S1 at least comprises the following steps:

[0039] S1.1: Image feature extraction, using the CLIP pre-trained VIT from the input image I∈R H×W×CGet image features from , where H, W, and C represent the height, width, and number of channels of the image, respectively;

[0040] I is divided into N = H × W / P 2 A sequence of non-overlapping patches, P represents the patch size;

[0041] Map these patches to 1D markers using a trainable linear projection

[0042] F v is the feature sequence representing the entire image;

[0043] f j v represents the feature vector of the jth patch;

[0044] N represents the number of feature sequences;

[0045] After injecting position embedding and additional category tags, the tag sequence Input into the transformer block to model the correlation of each patch;

[0046] Used as a global image representation, denoted by g v ;

[0047] S1.2: Text feature extraction, for text input T, use CLIP's text encoder to obtain text representation;

[0048] Processing begins by tokenizing the lowercase input text using byte pair encoding, using a vocabulary of 49,152 tokens;

[0049] The text description is enclosed in [SOS] and [EOS] markers to indicate the start and end of the sequence;

[0050] Then, the segmented text It is fed into the transformer to analyze the relationship between the tags;

[0051] [EOS] mark on transformer top Treated as a global text representation g t .

[0052] Furthermore, the S2 at least includes the following steps:

[0053] First, calculate the text feature F t The corresponding global image feature g v The similarity between them is then normalized using the softmax function, see formula (1):

[0054]

[0055] Where L represents the length of the text tag, <f i t ,g v > represents f i t and g v The cosine similarity between them, ρ is the temperature hyperparameter, r i The sampling probability of a predetermined proportion of text tags used as a mask; exp is the abbreviation of the exponential function, which calculates the exponential value of the input. Here, exp maps each similarity score to a positive value; k is the summation symbol The index variable in represents the sum of the exponential similarity scores between all feature pairs from 1 to L; L represents the number of text features;

[0056] Using the probability mask text feature in formula (1), we can get the partially masked text feature, denoted as F mt ;

[0057] F mt and image features F v are jointly input into a multimodal interaction encoder, which consists of a multi-head criss-cross attention layer and four transformer blocks;

[0058] In order to effectively integrate image and text information, the partially masked text feature F mt As query Q, and image features F v As key K and value V in multi-head cross attention layer;

[0059] This interaction is implemented as follows:

[0060]

[0061] in represents the masked text representation that incorporates image information, is the set of masked text tag positions, and d represents the dimension of embedding;

[0062] K T The T in the matrix usually refers to the transpose operation of the matrix. After transposing K, K is matched with the dimension of the query matrix Q, so as to calculate the dot product of the two and generate the attention weight matrix.

[0063] For each mask position The multi-layer perceptron classifier is the probability distribution of the MLP prediction corresponding to the original label, calculated as follows (3):

[0064]

[0065] in represents the predicted probability of the jth token in the vocabulary at the i-th mask position; f i m represents the masked text feature vector after fusion of image information, which refers to the text feature at the mask position i after being processed by the multimodal interactive encoder;

[0066] The vocabulary used here is denoted as O, which is the same as that used in the text encoding process and consists of 49,152 tokens;

[0067] At the final stage of the SGMI module, the MLM loss is calculated based on the prediction of the masked text tokens. This loss aims to optimize the model to align text and visual features at a fine-grained level.

[0068] The MLM loss is the sum of the cross entropy between the masked text token and its label, defined as follows:

[0069]

[0070] in is the true label, the correct label is 1, and the others are 0.

[0071] Furthermore, the MFE operates in two different modes: unmatched image feature extraction and unmatched text feature extraction, each of which is equipped with a customized information aggregation strategy;

[0072] The unmatched image feature extraction is UIE, the unmatched text feature extraction is UTE, and the information aggregation strategy is IG strategy;

[0073] The IG strategy is a key component of UIE and UTE modes to address the challenge of identifying mismatched features in high-dimensional space;

[0074] The process for UTE is similar to UIE. The difference between UTE and UIE is that it operates from the text aspect, focusing on text features that do not match any image features, and the IG strategy in UTE aggregates image tags in a similar way to UIE.

[0075] Further, the MFE in the UIE mode identifies image features that do not match any text features in a given pair;

[0076] The MFE application in UIE mode includes at least the following steps:

[0077] After aggregating L text tags into K text tags using the IG strategy, the similarity between each image tag and the K text tags is calculated;

[0078] UIE’s IG strategy applies parameter-free adaptive average pooling to transform the text mark Ft Aggregate into a more compact representation Where K <L;

[0079] That is: the text tags are divided into K groups, and the index of the text features in the jth group is Indicates that multiple text features within each group are processed to form a single aggregate feature. Realizing that average pooling brings significant information loss, a single-layer neural network is used for projection and GELU activation before pooling. After pooling, layer normalization is applied to the output to ensure consistent feature scale;

[0080] The IG strategy is expressed as formula (5):

[0081]

[0082] in represents the set of text features in the jth group, is the index set of the jth group, t j is the aggregated feature;

[0083] Then, the text features and visual features F v Projected to the common feature space through 1x1 convolution, the similarity is then calculated, expressed as formula (6):

[0084] s i,j = <W v f i v ,W t t j >,i∈[1,N],j∈[1,K] (6)

[0085] Where W v , W t ∈R M×C , <W v f i v ,W t t j > is W v f i v and W t t j The cosine similarity between i,j represents the similarity between the i-th image tag and the j-th text tag;

[0086] When the similarity between an image feature and all K text features is less than 0, it is identified as an image feature that does not match any text feature;

[0087] To highlight the degree of mismatch, the negative similarity scores of this image feature and all text features are added together, expressed as formula (7):

[0088]

[0089] where s i represents the cumulative mismatch score of the i-th image feature. Only negative similarity values ​​are included in the sum to emphasize the mismatch. The scalar s i is copied M times to form s i ∈R M , so that it is consistent with f i v ∈R M Dimension alignment;

[0090] Then, this mismatch score s i Used with f i v Perform element-by-element multiplication of the weight factor to obtain the visual feature v i ∈R M , which emphasizes the degree of mismatch with all text tags, expressed as formula (8):

[0091] v i =s i ⊙f i v (8)

[0092] Using the same approach, we identify all image tags that do not match any text features, forming Where G represents the number of such recognized image features;

[0093] For each image-text pair, these image features are calculated to emphasize the impact of mismatched features. With the text feature F t The similarity between them, the average result, is expressed as formula (9):

[0094]

[0095] s m represents the average similarity score of image and text features in the case of mismatch; G represents the number of image features identified as mismatching with the text; L is the number of text features; <v i ,f j t > represents the i-th image feature v that is identified as not matching the text i and the jth text feature f j t The cosine similarity of

[0096] This is then combined with the global image-text similarity;

[0097] For B image-text pairs, the intra-pair similarity is calculated by the global feature cosine similarity and s m The similarity between the i-th image and the j-th text, i≠j, is determined by the cosine similarity of the global features, which can be expressed as formula (10) and formula (11):

[0098]

[0099] in and Represent the global features of the i-th image and the j-th text respectively, and the similarity from the i-th image to the j-th text is recorded as Measure the correspondence between visual elements and text descriptions, and obtain That is, the similarity between the i-th text and the j-th image;

[0100] The resulting hybrid similarity is used to calculate the image-text contrast loss, i.e., ITC loss, which is a method for training multimodal models to optimize the alignment between matching image-text pairs by maximizing the similarity between matching image-text pairs while minimizing the similarity between unmatched pairs;

[0101] In the last step of the MFE module, the similarity scores calculated in the previous step are used to calculate the ITC loss to enhance the effect of mismatched features;

[0102] By emphasizing these contrastive relationships, ITC loss significantly enhances the model's ability to distinguish visually similar individuals, as calculated in detail in the following formula group (12):

[0103]

[0104] where γ is the temperature hyperparameter.

[0105] Further, the application of the ID loss is: using the ID loss to group individuals based on their identities to ensure matching at the identity level;

[0106] For an image-text pair (I,T), the cross entropy loss is applied to the global feature (g v ,g t );

[0107] ID Loss The definition is expressed as formula (13):

[0108]

[0109] Where Wid represents the parameters of the fully connected layer used for classification, y id One-hot encoded vector representing the true label.

[0110] Furthermore, the application of the SDM loss is: using the SDM loss to minimize the image-text similarity μ i,j The distribution of matching labels ξ i,j The KL divergence between the normalized distributions of is shown in the following formulas (14) and (15):

[0111]

[0112]

[0113] The peak value of the probability distribution is controlled by the temperature hyperparameter τ, y i,j is the true matching label, y i,j =1 means are positive pairs from the same identity, while 0 represents negative pairs;

[0114] The SDM loss is expressed as formula (16):

[0115]

[0116] where ∈ is a decimal number to avoid numerical problems;

[0117] The total loss of training is the weighted sum of these losses:

[0118]

[0119] Where λ1 and λ2 are weight factors used to balance the contribution of ITC loss and MLM loss.

[0120] Compared with the prior art, the present invention has the following beneficial effects:

[0121] 1. This paper proposes a similarity-guided masking strategy, which enhances the use of image information in the masked language modeling process to promote stronger cross-modal interaction. Unlike random masking, this method guides the model's attention to more relevant image-text correspondences, thereby achieving better fine-grained alignment;

[0122] 2. This paper introduces a novel mismatching feature emphasis module, which innovatively utilizes mismatching features to improve retrieval accuracy. Previous work mainly focuses on matching image-text features, and our mismatching feature emphasis module explores the importance of mismatching features, which play a key role in distinguishing visually similar individuals. BRIEF DESCRIPTION OF THE DRAWINGS

[0123] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0124] Figure 1 It is a schematic diagram of the overall process of the present invention. DETAILED DESCRIPTION

[0125] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0126] Embodiment 1:

[0127] Given an image-text pair, we first extract image and text feature maps using the pre-trained encoders in the Contrastive Language-Image Pre-training (CLIP) model. Specifically, we use a 12-layer Vision Transformer (ViT) as the image encoder and a 12-layer Transformer architecture as the text encoder. Next, the extracted features are computed through four parallel components: Similarity-Guided Multimodal Interaction (SGMI), Mismatched Feature Emphasis (MFE), and identity identification (ID) loss and similarity distribution matching (SDM) loss.

[0128] The SGMI module deeply integrates image information into the masked language modeling (MLM) task, implicitly models fine-grained image-text correspondences and calculates the MLM loss. The MFE module enhances retrieval performance by emphasizing the role of mismatched features and includes image-text contrast (ITC) loss calculation. At the same time, the ID loss is used to optimize the identity discriminant representation, while the SDM loss is used to align the similarity distribution between modalities and enhance the cross-modal consistency of the learned features.

[0129] This parallel architecture enables each component to focus on its designated task while processing the same initial features. During back-propagation, the outputs and gradients of these components are aggregated, allowing the model to learn robust and nuanced representations. This approach effectively captures subtle fine-grained correspondences and key differences between image and text modalities.

[0130] Embodiment 2:

[0131] Based on the above embodiment 1, the following technical contents are specifically disclosed:

[0132] See also Figure 1 , a pedestrian re-identification method based on similarity guidance and mismatch feature enhancement at least includes the following steps:

[0133] S1: Image and text feature extraction, using the pre-trained encoder in the contrastive language-image pre-trained model to extract image and text feature maps, that is, using the 12-layer Vision Transformer as the image encoder and the 12-layer Transformer architecture as the text encoder. The contrastive language-image pre-trained model is CLIP and the Vision Transformer is ViT.

[0134] S2: Similarity guides multimodal interaction, using the correlation between image and text features to select text features to be masked. Text features that are highly correlated with image features are more likely to be masked. This strategy encourages the use of image information in the MLM process, thereby enhancing cross-modal interaction between images and texts.

[0135] It should be noted that the method of implicitly simulating the fine-grained correspondence between images and text using MLM has excellent retrieval performance in the TI-ReID task. However, the strategy of random masked text labeling lacks deep involvement of image information, resulting in insufficient cross-modal interaction. Therefore, it is proposed to use similarity to guide multimodal interaction.

[0136] S3: Mismatch feature enhancement is performed through the mismatch feature emphasis module, aiming to reduce the false positive rate and improve the overall retrieval accuracy. The mismatch feature emphasis module is MFE;

[0137] It should be noted that most methods focus on amplifying the beneficial effects of matched image-text features on retrieval, ignoring the effects of mismatched features that may significantly affect retrieval results. Highlighting the effects of mismatched features can help improve retrieval performance. To this end, we propose a mismatched feature emphasis (MFE) module. While traditional methods focus on using matched features to improve retrieval, MFE stands in a complementary position and emphasizes the discriminative ability of mismatches.

[0138] S4: Perform training and inference, using a multi-task learning approach to optimize four loss functions in four parallel components. The four parallel components include the SGMI module, the MFE module, the ID loss, and the SDM loss. The SGMI module calculates the MLM loss, and the MFE module contains the ITC loss. At the same time, the ID loss and the SDM loss are calculated independently. During the inference process, a similarity metric S is calculated for each text query and image candidate pair. This calculation uses the cosine similarity method to quantify the correlation between text and image features. Subsequently, S is used to establish a ranked list of individuals in the image library, and the highest-ranked image matches that are closest to the given text description are retrieved and presented.

[0139] S1 includes at least the following steps:

[0140] S1.1: Image feature extraction, using the CLIP pre-trained VIT from the input image I∈R H×W×C Get image features from , where H, W, and C represent the height, width, and number of channels of the image, respectively;

[0141] I is divided into N = H × W / P 2 A sequence of non-overlapping patches, P represents the patch size;

[0142] Map these patches to 1D markers using a trainable linear projection

[0143] F v Represents the feature sequence of the entire image, f j v represents the feature vector of the jth patch, and N represents the number of feature sequences;

[0144] After injecting position embedding and additional category (CLS) tags, the tag sequence Input into the transformer block to model the correlation of each patch;

[0145] Used as a global image representation, denoted by g v ;

[0146] S1.2: Text feature extraction, for text input T, use CLIP's text encoder to obtain text representation;

[0147] This encoder is a modified version of the Transformer architecture. Processing begins by tokenizing the lowercase input text using byte pair encoding (BPE) using a vocabulary of 49,152 tokens;

[0148] The text description is enclosed in [SOS] and [EOS] markers to indicate the start and end of the sequence;

[0149] Then, the segmented text It is fed into the transformer to analyze the relationship between the tags;

[0150] , the [EOS] mark at the top level of the transformer Treated as a global text representation g t .

[0151] S2 at least includes the following steps:

[0152] First, calculate the text feature F t The corresponding global image feature g v The similarity between them is then normalized using the softmax function, see formula (1):

[0153]

[0154] Where L represents the length of the text tag, <f i t ,g v > represents f i t and g v The cosine similarity between them, ρ is the temperature hyperparameter, r i The sampling probability of the predetermined proportion of text tokens used as masks;

[0155] exp is the abbreviation of exponential function, which calculates the exponential value of the input. Here exp maps each similarity score to a positive value. k is the summation symbol The index variable in represents the sum of the exponential similarity scores between all feature pairs from 1 to L. L represents the number of text features.

[0156] Using the probability mask text feature in formula (1), we can get the partially masked text feature, denoted as F mt ;

[0157] F mt and image features F v are jointly input into a multimodal interaction encoder, which consists of a multi-head criss-cross attention layer and four transformer blocks;

[0158] In order to effectively integrate image and text information, the partially masked text feature F mt As query Q, and image features F v As key K and value V in multi-head cross attention layer;

[0159] This interaction is implemented as follows:

[0160]

[0161] in represents the masked text representation that incorporates image information, is the set of masked text tag positions, and d represents the dimension of embedding;

[0162] K T The T in the matrix usually refers to the transpose operation of the matrix. After transposing K, the dimension of K can be matched with the query matrix Q, so as to calculate the dot product of the two and generate the attention weight matrix.

[0163] For each mask position The multi-layer perceptron classifier is the probability distribution of the MLP prediction corresponding to the original label, calculated as follows (3):

[0164]

[0165] in represents the predicted probability of the jth token in the vocabulary at the i-th mask position;

[0166] f i m It represents the masked text feature vector after the image information is integrated, and refers to the text feature at the mask position i after being processed by the multimodal interactive encoder.

[0167] The vocabulary used here is denoted as O, which is the same as that used in the text encoding process and consists of 49,152 tokens;

[0168] At the final stage of the SGMI module, the MLM loss is calculated based on the prediction of the masked text tokens. This loss aims to optimize the model to align text and visual features at a fine-grained level.

[0169] The MLM loss is the sum of the cross entropy between the masked text token and its label, defined as follows:

[0170]

[0171] in is the true label, the correct label is 1, and the others are 0;

[0172] By incorporating image information into the MLM task, the model gains a deeper understanding of the complex relationship between the two modalities. The MLM loss is crucial to enhance cross-modal feature alignment, thereby improving the model's ability to understand fine-grained cross-modal information.

[0173] MFE operates in two different modes: unmatched image feature extraction and unmatched text feature extraction, each with a customized information aggregation strategy;

[0174] Unmatched image feature extraction is UIE, unmatched text feature extraction is UTE, and information aggregation strategy is IG strategy;

[0175] The IG strategy is a key component of the UIE and UTE modes, which is used to address the challenge of identifying mismatched features in high-dimensional space;

[0176] The process for UTE is similar to that for UIE. The difference between UTE and UIE is that UTE operates from the textual aspect, focusing on text features that do not match any image features, and the IG strategy in UTE aggregates image tags in a similar way to UIE.

[0177] MFE in UIE mode identifies image features that do not match any text features in a given pair;

[0178] MFE application in UIE mode includes at least the following steps:

[0179] After aggregating L text tags into K text tags using the IG strategy, the similarity between each image tag and the K text tags is calculated;

[0180] The probability that an image tag has a similarity less than zero with all L text tags is extremely low, making it challenging to identify truly mismatched features. To address this problem, UIE’s IG strategy applies parameter-free adaptive average pooling to pool the text tags F t Aggregate into a more compact representation Where K <L;

[0181] That is: the text tags are divided into K groups, and the index of the text features in the jth group is Indicates that multiple text features within each group are processed to form a single aggregate feature. Aware that average pooling may result in significant information loss, a single-layer neural network (Linear) is used for projection and GELU activation before pooling. After pooling, layer normalization (LN) is applied to the output to ensure consistent feature scale.

[0182] The IG strategy is expressed as formula (5):

[0183]

[0184] in represents the set of text features in the jth group, is the index set of the jth group, t j is the aggregated feature;

[0185] Then, the text features and visual features F v Projected to the common feature space through 1x1 convolution, the similarity is then calculated, expressed as formula (6):

[0186] s i,j = <W v f i v ,W t t j >,i∈[1,N],j∈[1,K] (6)

[0187] Where W v , W t ∈R M×C , <W v f i v ,W t t j > is W v f i v and W t t j The cosine similarity between i,j represents the similarity between the i-th image tag and the j-th text tag;

[0188] When the similarity between an image feature and all K text features is less than 0, it is identified as an image feature that does not match any text feature;

[0189] To highlight the degree of mismatch, the negative similarity scores of this image feature and all text features are added together, expressed as formula (7):

[0190]

[0191] where s i represents the cumulative mismatch score of the i-th image feature. Only negative similarity values ​​are included in the sum to emphasize the mismatch. The scalar s i is copied M times to form s i ∈R M , so that it is consistent with f i v ∈R M Dimension alignment;

[0192] Then, this mismatch score s i Used with f i v Perform element-by-element multiplication of the weight factor to obtain the visual feature v i ∈R M , which emphasizes the degree of mismatch with all text tags, expressed as formula (8):

[0193] v i =s i ⊙f i v (8)

[0194] Using the same approach, we identify all image tags that do not match any text features, forming Where G represents the number of such recognized image features;

[0195] For each image-text pair, these image features are calculated to emphasize the impact of mismatched features. With the text feature F t The similarity between them, the average result, is expressed as formula (9):

[0196]

[0197] s m represents the average similarity score of image and text features in the case of mismatch; G represents the number of image features identified as mismatching with the text; L is the number of text features; <v i ,f j t > represents the i-th image feature v that is identified as not matching the text i and the jth text feature f j t The cosine similarity of

[0198] This is then combined with the global image-text similarity;

[0199] For B image-text pairs, the intra-pair similarity is calculated by the global feature cosine similarity and s m The similarity between the i-th image and the j-th text, i≠j, is determined by the cosine similarity of the global features, which can be expressed as formula (10) and formula (11):

[0200]

[0201] in and Represent the global features of the i-th image and the j-th text respectively, and the similarity from the i-th image to the j-th text is recorded as Measure the correspondence between visual elements and text descriptions, and obtain That is, the similarity between the i-th text and the j-th image;

[0202] The resulting hybrid similarity is used to calculate the image-text contrast loss, i.e., ITC loss, which is a method for training multimodal models to optimize the alignment between matching image-text pairs by maximizing the similarity between matching image-text pairs while minimizing the similarity between unmatched pairs;

[0203] In the last step of the MFE module, the similarity scores calculated in the previous step are used to calculate the ITC loss to enhance the effect of mismatched features;

[0204] By emphasizing these contrastive relationships, ITC loss significantly enhances the model's ability to distinguish visually similar individuals, as calculated in detail in the following formula group (12):

[0205]

[0206] where γ is the temperature hyperparameter.

[0207] The application of ID loss is: use ID loss to group individuals based on their identities to ensure matching at the identity level;

[0208] For an image-text pair (I,T), the cross entropy loss is applied to the global feature (g v ,g t );

[0209] ID Loss The definition is expressed as formula (13):

[0210]

[0211] Where W id represents the parameters of the fully connected layer used for classification, y id One-hot encoded vector representing the true label.

[0212] The application of SDM loss is: Use SDM loss to minimize the image-text similarity μ i,j The distribution of matching labels ξ i,j The KL divergence between the normalized distributions of is shown in the following formulas (14) and (15):

[0213]

[0214] The peak value of the probability distribution is controlled by the temperature hyperparameter τ, y i,j is the true matching label, y i,j =1 means are positive pairs from the same identity, while 0 represents negative pairs;

[0215] The SDM loss is expressed as formula (16):

[0216]

[0217] where ∈ is a decimal number to avoid numerical problems;

[0218] The total loss of training is the weighted sum of these losses:

[0219]

[0220] Where λ1 and λ2 are weight factors used to balance the contribution of ITC loss and MLM loss.

[0221] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A person re-identification method based on similarity guidance and mismatch feature enhancement, characterized by: At least the following steps are included: S1: Image and text feature extraction, using the pre-trained encoder in the contrastive language-image pre-trained model to extract image and text feature maps, that is, using the 12-layer Vision Transformer as the image encoder and the 12-layer Transformer architecture as the text encoder. The contrastive language-image pre-trained model is CLIP and the Vision Transformer is ViT. S2: Similarity guides multimodal interaction, using the correlation between image and text features to select text features to be masked. Text features that are highly correlated with image features are more likely to be masked. This strategy encourages the use of image information in the MLM process, thereby enhancing cross-modal interaction between images and texts. S3: mismatch feature enhancement is performed through a mismatch feature emphasis module, aiming to reduce the false positive rate and improve the overall retrieval accuracy. The mismatch feature emphasis module is MFE; S4: Perform training and inference, using a multi-task learning approach to optimize four loss functions in four parallel components, including an SGMI module, an MFE module, an ID loss, and an SDM loss. The SGMI module calculates the MLM loss, and the MFE module contains the ITC loss. At the same time, the ID loss and the SDM loss are calculated independently. During the inference process, a similarity metric S is calculated for each text query and image candidate pair. This calculation uses the cosine similarity method to quantify the correlation between text and image features. Subsequently, S is used to establish a ranked list of individuals in the image library, and the highest-ranked image matches that are closest to the given text description are retrieved and presented.

2. The pedestrian re-identification method based on similarity guidance and mismatch feature enhancement according to claim 1, characterized in that: The S1 at least comprises the following steps: S1.1: Image feature extraction, using the CLIP pre-trained VIT from the input image I∈R H×W×C Get image features from , where H, W, and C represent the height, width, and number of channels of the image, respectively; I is divided into N = H × W / P 2 A sequence of non-overlapping patches, P represents the patch size; Map these patches to 1D markers using a trainable linear projection F v is the feature sequence representing the entire image; represents the feature vector of the jth patch; N represents the number of feature sequences; After injecting position embedding and additional category tags, the tag sequence Input into the transformer block to model the correlation of each patch; Used as a global image representation, denoted as g v ; S1.2: Text feature extraction, for text input T, use CLIP's text encoder to obtain text representation; Processing begins by tokenizing the lowercase input text using byte pair encoding, using a vocabulary of 49,152 tokens; The text description is enclosed in [SOS] and [EOS] markers to indicate the start and end of the sequence; Then, the segmented text It is fed into the transformer to analyze the relationship between the tags; [EOS] mark on transformer top Treated as a global text representation g t .

3. The pedestrian re-identification method based on similarity guidance and mismatch feature enhancement according to claim 2, characterized in that: The S2 at least comprises the following steps: First, calculate the text feature F t The corresponding global image feature g v The similarity between them is then normalized using the softmax function, see formula (1): Where L represents the length of the text tag, express and g v The cosine similarity between them, ρ is the temperature hyperparameter, r i The sampling probability of a predetermined proportion of text tags used as a mask; exp is the abbreviation of the exponential function, which calculates the exponential value of the input. Here, exp maps each similarity score to a positive value; k is the summation symbol The index variable in represents the sum of the exponential similarity scores between all feature pairs from 1 to L; L represents the number of text features; Using the probability mask text feature in formula (1), we can get the partially masked text feature, denoted as F mt ; F mt and image features F v are jointly input into a multimodal interaction encoder, which consists of a multi-head criss-cross attention layer and four transformer blocks; In order to effectively integrate image and text information, the partially masked text feature F mt As query Q, and image features F v As key K and value V in multi-head cross attention layer; This interaction is implemented as follows: in represents the masked text representation that incorporates image information, is the set of masked text tag positions, and d represents the dimension of embedding; K T The T in the matrix usually refers to the transpose operation of the matrix. After transposing K, K is matched with the dimension of the query matrix Q, so as to calculate the dot product of the two and generate the attention weight matrix. For each mask position The multi-layer perceptron classifier is the probability distribution of the MLP prediction corresponding to the original label, calculated as follows (3): in represents the predicted probability of the jth token in the vocabulary at the i-th mask position; represents the masked text feature vector after fusion of image information, which refers to the text feature at the mask position i after being processed by the multimodal interactive encoder; The vocabulary used here is denoted as O, which is the same as that used in the text encoding process and consists of 49,152 tokens; At the final stage of the SGMI module, the MLM loss is calculated based on the prediction of the masked text tokens. This loss aims to optimize the model to align text and visual features at a fine-grained level. The MLM loss is the sum of the cross entropy between the masked text token and its label, defined as follows: in is the true label, the correct label is 1, and the others are 0.

4. The pedestrian re-identification method based on similarity guidance and mismatch feature enhancement according to claim 3 is characterized in that: The MFE operates in two different modes: unmatched image feature extraction and unmatched text feature extraction, each of which is equipped with a customized information aggregation strategy; The unmatched image feature extraction is UIE, the unmatched text feature extraction is UTE, and the information aggregation strategy is IG strategy; The IG strategy is a key component of UIE and UTE modes to address the challenge of identifying mismatched features in high-dimensional space; The process for UTE is similar to UIE. The difference between UTE and UIE is that it operates from the text aspect, focusing on text features that do not match any image features, and the IG strategy in UTE aggregates image tags in a similar way to UIE.

5. The pedestrian re-identification method based on similarity guidance and mismatch feature enhancement according to claim 4 is characterized in that: The MFE in UIE mode identifies image features that do not match any text features in a given pair; The MFE application in UIE mode includes at least the following steps: After aggregating L text tags into K text tags using the IG strategy, the similarity between each image tag and the K text tags is calculated; UIE’s IG strategy applies parameter-free adaptive average pooling to transform the text mark F t Aggregate into a more compact representation Where K <L; That is: the text tags are divided into K groups, and the index of the text features in the jth group is Indicates that multiple text features within each group are processed to form a single aggregate feature. Realizing that average pooling brings significant information loss, a single-layer neural network is used for projection and GELU activation before pooling. After pooling, layer normalization is applied to the output to ensure consistent feature scale; The IG strategy is expressed as formula (5): in represents the set of text features in the jth group, is the index set of the jth group, t j is the aggregated feature; Then, the text features and visual features F v Projected to the common feature space through 1x1 convolution, the similarity is then calculated, expressed as formula (6): Where W v , W t ∈R M×C , yes and W t t j The cosine similarity between i,j represents the similarity between the i-th image tag and the j-th text tag; When the similarity between an image feature and all K text features is less than 0, it is identified as an image feature that does not match any text feature; To highlight the degree of mismatch, the negative similarity scores of this image feature and all text features are added together, expressed as formula (7): where s i represents the cumulative mismatch score of the i-th image feature. Only negative similarity values ​​are included in the sum to emphasize the mismatch. The scalar s i is copied M times to form s i ∈R M , so that it is Dimension alignment; Then, this mismatch score s i Used as Perform element-by-element multiplication of the weight factor to obtain the visual feature v i ∈R M , which emphasizes the degree of mismatch with all text tags, expressed as formula (8): Using the same approach, we identify all image tags that do not match any text features, forming Where G represents the number of such recognized image features; For each image-text pair, these image features are calculated to emphasize the impact of mismatched features. With the text feature F t The similarity between them, the average result, is expressed as formula (9): s m represents the average similarity score of image and text features in the case of mismatch; G represents the number of image features identified as mismatching with the text; L is the number of text features; represents the i-th image feature v that is identified as not matching the text i and the jth text feature The cosine similarity of This is then combined with the global image-text similarity; For B image-text pairs, the intra-pair similarity is calculated by the global feature cosine similarity and s m The similarity between the i-th image and the j-th text, i≠j, is determined by the cosine similarity of the global features, which can be expressed as formula (10) and formula (11): in and Represent the global features of the i-th image and the j-th text respectively, and the similarity from the i-th image to the j-th text is recorded as Measure the correspondence between visual elements and text descriptions, and obtain That is, the similarity between the i-th text and the j-th image; The resulting hybrid similarity is used to calculate the image-text contrast loss, i.e., ITC loss, which is a method for training multimodal models to optimize the alignment between matching image-text pairs by maximizing the similarity between matching image-text pairs while minimizing the similarity between unmatched pairs; In the last step of the MFE module, the similarity scores calculated in the previous step are used to calculate the ITC loss to enhance the effect of mismatched features; By emphasizing these contrastive relationships, ITC loss significantly enhances the model's ability to distinguish visually similar individuals, as calculated in detail in the following formula group (12): where γ is the temperature hyperparameter.

6. The pedestrian re-identification method based on similarity guidance and mismatch feature enhancement according to claim 5, characterized in that: The application of the ID loss is: using the ID loss to group individuals based on their identities to ensure matching at the identity level; For an image-text pair (I,T), the cross entropy loss is applied to the global feature (g v ,g t ); ID Loss The definition is expressed as formula (13): Where W id represents the parameters of the fully connected layer used for classification, y id One-hot encoded vector representing the true label.

7. The method for person re-identification based on similarity guidance and mismatch feature enhancement according to claim 6, characterized in that: The application of the SDM loss is: Use the SDM loss to minimize the image-text similarity μ i,j The distribution of matching labels ξ i,j The KL divergence between the normalized distributions of is shown in the following formulas (14) and (15): The peak value of the probability distribution is controlled by the temperature hyperparameter τ, y i,j is the true matching label, y i,j =1 means are positive pairs from the same identity, while 0 represents negative pairs; The SDM loss is expressed as formula (16): where ∈ is a decimal number to avoid numerical problems; The total loss of training is the weighted sum of these losses: Where λ1 and λ2 are weight factors used to balance the contribution of ITC loss and MLM loss.

Images