Text-image pedestrian re-identification method oriented to semantic fuzziness and over-confidence decision

Abstract

The invention discloses a text-image pedestrian re-identification method oriented to semantic fuzziness and excessive decision confidence, and belongs to the technical field of pedestrian re-identification. The method mainly comprises the steps that based on a distributed feature alignment module, image and text modal features are coded into Gaussian distribution, a feature center and an intra-class semantic change range are jointly expressed through a mean value and a variance, and the modeling ability of cross-modal semantic difference and visual diversity is improved; the uncertainty penalty alignment module based on evidence deep learning explicitly models the uncertainty in the image-text matching process by introducing Dirichlet distribution and subjective logic theories, and applies adaptive penalty to a high-uncertainty matching relationship, so that the excessive confidence of the model on a mismatching result is inhibited. According to the method, a probability distribution-based feature expression mode and an uncertainty perception training mechanism are adopted, so that the matching accuracy and generalization ability of a cross-modal retrieval system in view angle change, semantic fuzziness and background interference scenes are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of pedestrian re-identification technology, and more specifically, to a text-image pedestrian re-identification method for addressing semantic ambiguity and overconfidence in decision-making. Background Technology

[0002] Person re-identification from text to image is an important research direction in the field of cross-modal retrieval. The most critical technical challenge in this research direction lies in establishing an effective cross-modal feature alignment mechanism to map visual and textual features to a unified semantic space.

[0003] In terms of technological development, research in this field has roughly gone through three stages: Early research primarily used basic network architectures such as VGG and LSTM to extract visual and textual features respectively, achieving feature alignment through the design of matching loss functions; mid-stage research introduced more powerful backbone networks such as ResNet and BERT, and optimized the design of cross-modal matching loss functions, significantly improving the alignment effect of global features; recent research focuses more on the mining of fine-grained features, improving retrieval accuracy by explicitly modeling local information such as human body parts and color attributes. In addition, some works have employed implicit methods such as attention mechanisms to capture local feature associations.

[0004] It is worth noting that most existing methods rely on unimodal pre-trained models to extract features, failing to fully leverage the advantages of state-of-the-art vision-language pre-trained models. Although existing research has attempted to apply the CLIP model to this task, it has not fully utilized the potential of the CLIP native architecture. Summary of the Invention

[0005] To address the issue of excessively high model confidence in erroneous results in visual-language person re-identification tasks, leading to mismatches due to intra-class visual differences and background similarity under the same text and image descriptions, this invention provides a text-image person re-identification method tackling semantic ambiguity and overconfidence in decision-making. This invention primarily utilizes feature representation upgrades from "point" to "surface" and explicit quantification of cross-modal alignment uncertainty. It parameterizes matching confidence through Dirichlet distribution and designs a confidence-aware contrastive loss to apply adaptive penalties to high-uncertainty matches, suppressing overconfident predictions. This process not only improves matching accuracy but also, to some extent, overcomes the shortcomings of traditional softmax classification.

[0006] The technical means employed in this invention are as follows: A text-image person re-identification method addressing semantic ambiguity and overconfidence in decision-making includes the following steps: Obtain pedestrian image data and pedestrian text data, and preprocess the pedestrian image data and pedestrian text data respectively; The preprocessed pedestrian text data is input into the Transformer text encoder in the CLIP model. First, it passes through a word embedding layer and positional encoding to generate a text feature sequence. Then, it passes through a multi-layer self-attention structure to extract its contextual features, and finally outputs a set of fixed-dimensional text feature sequence vectors. At the same time, the preprocessed pedestrian image data is input into the Transformer visual encoder in the CLIP model. First, the pedestrian image data is divided into several fixed-size image patches and embedded as patch tokens. After adding positional encoding, it is input into the ViT structure. The visual context features are extracted through the Transformer unit, and finally outputs a fixed-dimensional image visual feature sequence vector. The text feature sequence vector and the image visual feature sequence vector are input into the distributed feature alignment module. In the distributed feature alignment module, the text feature sequence vector and the image visual feature sequence vector are first encoded using Gaussian distribution. Then, by optimizing the alignment relationship between the mean vector and the variance vector, the intra-class semantic variation between different modal data is explicitly modeled. The text feature sequence vector and image visual feature sequence vector output by the distributed feature alignment module are input into the uncertainty modeling module based on evidence deep learning. The similarity between the text feature sequence vector and the image visual feature sequence vector is converted into a Dirichlet distribution. Based on the Dirichlet distribution, the possible values ​​of the alignment probability after evidence modeling are simulated. Based on the loss function, the evidence quantity of positive sample pairs is constrained to be higher than that of negative sample pairs.

[0007] Furthermore, the pedestrian image data is preprocessed, including: uniformly cropping the pedestrian image data into fixed-size RGB images and performing normalization processing.

[0008] Furthermore, the pedestrian text data is preprocessed, including converting the pedestrian text data to lowercase, removing special characters, and then performing word segmentation and encoding on a word-by-word basis.

[0009] Furthermore, the distributed feature alignment module includes a first Feed Forward module, a multi-head operation module, a second Feed Forward module, and a third Feed Forward module. The input data of the distributed feature alignment module first passes through the first Feed Forward module and then enters the multi-head operation module. Each operation module includes a mean path and a variance path. The mean path and variance path are used to map the input data to queries, keys, and values ​​for attention operations. After concatenating the mean path of the multi-head operation module, it is superimposed with the input data of the distributed feature alignment module and output through the second Feed Forward module. After concatenating the variance path of the multi-head operation module, it is output through the third Feed Forward module.

[0010] Compared with the prior art, the present invention has the following advantages: 1. The point-to-surface collaborative feature alignment module of this invention represents the modal features of images and text as Gaussian distributions, realizing point-to-surface feature representation, thereby more effectively expressing multimodal semantic information and complex relationships, and enhancing the robustness of the model to intra-class semantic changes.

[0011] 2. This invention addresses the problem that current models often exhibit overconfidence in incorrect identities due to a lack of uncertainty modeling. It introduces an evidence-based deep learning mechanism to implement an uncertainty-penalized alignment module based on evidence-based deep learning, constructing a confidence-aware matching training strategy for explicitly modeling the uncertainty of cross-modal alignment and reducing image-text mismatches. Experimental results on multiple mainstream datasets such as CUHK-PEDES show that the proposed method significantly outperforms many existing state-of-the-art methods, demonstrating good generalization ability and retrieval performance. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a diagram of the overall model framework used in a text-image person re-identification method for semantic ambiguity and overconfidence in decision-making, as described in an example of the present invention.

[0014] Figure 2 This is a network architecture diagram of the distributed feature alignment converter (PDFE) in an example of the present invention.

[0015] Figure 3This is a block diagram of the uncertainty penalty alignment module based on evidence deep learning in an embodiment of the present invention. Detailed Implementation

[0016] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0017] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0018] This invention provides a text-image person re-identification method for addressing semantic ambiguity and overconfidence in decision-making. The model framework used is as follows: Figure 1 As shown. The method includes the following steps.

[0019] S1. Preprocess the input pedestrian images and text to adapt them to the feature extraction network. The input images are uniformly cropped to a fixed size RGB image and normalized. The text descriptions are converted to lowercase, special characters are removed, and then segmented and encoded into words to facilitate subsequent embedding model processing.

[0020] In this application, the Clip algorithm is used to process pedestrian image sequences directly. Specifically, the input image is adjusted to a fixed size (224×224) using bicubic interpolation, center-cropped, and converted to RGB format. It is then normalized according to specific CLIP parameters. During training, random horizontal flipping and cropping enhancement are applied. After image preprocessing, the image becomes a fixed-size numerical matrix (e.g., 224×224×3), where each number represents a normalized pixel value. For the ViT model, this matrix is ​​further divided into 16×16 squares, each of which is flattened into a string of numbers. After adding positional encoding, it becomes a visual feature sequence, much like turning an image into a numerical password with location information.

[0021] S2. The text input is fed into the Transformer text encoder in the CLIP model. First, it passes through the word embedding layer and position encoding to generate a text feature sequence of length L. Then, it passes through a multi-layer self-attention structure to extract its contextual features. Finally, it outputs a set of text feature sequence vectors with a dimension of (L×D). In this invention, the Clip algorithm is used to process the text sequence of pedestrians directly. Specifically, the text is uniformly converted to lowercase and special characters are removed. Then, it is processed by the CLIP-specific BPE tokenizer, with start and end marks added and truncated to a maximum of 77 tokens. Finally, it is converted into a word index sequence and a corresponding attention mask is generated. Short sequences are padded with zeros to a fixed length. After text preprocessing, it becomes a series of numbered word fragments (up to 77), with each number corresponding to a specific text segment. These numbers are converted into numeric vectors through a lookup table (each word becomes a 512-dimensional numeric string), and special symbols marking the beginning and end of sentences and positional codes are added. Finally, it becomes a series of numeric passwords with positional information, aligned with the numeric password format of the image.

[0022] S3. The image input is fed into the visual Transformer encoder in the CLIP model. The image is first divided into several fixed-size image patches and embedded as patch tokens. After adding position encoding, it is input into the ViT structure. Visual context features are extracted through layers of Transformer units, and the final output is an image visual feature sequence with a dimension of (N×D). In this invention, specifically, the feature extraction process of the CLIP model is mainly divided into two parts: one is to extract visual features from the input image, and the other is to extract language features from the input text.

[0023] Specifically, the input to the vision branch is an RGB image (N, 3, 224, 224), where N represents the batch size, 3 represents the RGB three channels, and 224×224 is the image resolution. The input image is first segmented into 16×16 image patches. Each patch is flattened and transformed into a 768-dimensional patch token through a linear projection layer, with a learnable [CLS] token and positional encoding added. This is then processed by a 12-layer Transformer encoder, each layer containing a multi-head self-attention mechanism (12 attention heads), layer normalization, and an MLP expansion layer (expansion ratio of 4), maintaining information flow through residual connections. The first 6 layers are called the vision model Block 1, primarily extracting local visual features; the last 6 layers are called the vision model Block 2, focusing on establishing global semantic associations. The final output contains the [CLS] token features and all patch token features, which, after L2 normalization, yield a 512-dimensional visual feature vector.

[0024] The input to the text branch is a segmented text sequence (N, 77), where N is the batch size and 77 is the maximum token length. The text is first converted into 512-dimensional word vectors through a word embedding layer, with positional encoding and special tokens added. It then passes through a 12-layer Transformer encoder, each layer having the same structure as the visual encoder, but using masked multi-head attention to prevent information leakage. The first 6 layers are called Text Model Block 1, primarily establishing word-level semantic relationships; the last 6 layers are called Text Model Block 2, focusing on constructing sentence-level semantic representations. Finally, the token at the [EOF] position is selected as the text feature, and after L2 normalization, a 512-dimensional text feature vector is obtained, aligned with the visual features in the same embedding space.

[0025] S4. After obtaining the image-text modal feature sequence, it is first input into the distributed feature alignment module. The image-text features are encoded into a Gaussian distribution form, such as... Figure 2 As shown, each modal feature is represented by its mean μ and variance. This module optimizes the alignment between the mean vector and the variance vector, explicitly models intra-class semantic variations across modalities, enhances the expressive power of image-text alignment, and effectively alleviates the problem of inconsistent image-text feature alignment.

[0026] Specifically, the distributed feature converter (PDFE) of this invention predicts a mean vector (μ) and a variance vector (μ) for each input feature. ), where the mean vector (μ) represents the central location of the distribution in the probability space, and its physical meaning is represented by the central location of individual characteristics, and the variance vector ( The range of distribution across various dimensions is represented, and its physical meaning indicates the fluctuation of the feature semantics. In this application, the input features of PDFE are preprocessed text feature sequence vectors and image visual feature sequence vectors. The distributed feature converter (PDFE) proposed in this application considers both feature-level and sequence-level interactions in modeling the mean and variance vectors. The specific implementation is as follows: Input tensor: T is the sequence length (such as the number of image patches or text tokens), and H is split into μ and μ along the feature dimension. Two paths: FeedForward for feature-level interactions (feature-level refers to overall features), and Multi-Head operations for sequence-level interactions (sequence-level refers to each small text token and small image token). By applying MH operations, the input state... The sequence is divided into K heads, where T is the sequence length and D is the hidden size. Within each head, the features are split and sent to two paths (µ, σ). In each path, the hidden state is input. Projected to For example, the operation in the µ path is:

[0027]

[0028] The specific parameter definitions are as follows. This is the input matrix for the μ-th head of the l-th layer. Typically, for the first attention layer, the input is a sequence of word embeddings; for deeper layers, the input is the output of the previous layer.

[0029] This is a learnable weight matrix used to linearly project the input H into queries, keys, and values. A superscript <1 may indicate that this is the first set of projected weights for this header, or a specific label.

[0030] Query matrix. Obtained by multiplying the input H by the weight matrix. It represents the content that needs to be focused on.

[0031] The key matrix (Key) is obtained by multiplying the input H by the weight matrix. It represents the content that can be of interest.

[0032] Value matrix. Obtained by multiplying the input H by the weight matrix. It represents the information that should be extracted if it is of interest.

[0033] Multi-head integration, among which It is the output projection matrix:

[0034] The σ path is similar to the µ path; the computational force flow is symmetrical, but it uses an independent... and Since the input point representation is related to the average vector, an additive operation is used to learn the average vector representation μ. After PDFE, each visual and linguistic token is represented as a Gaussian distribution in a high-dimensional probability space with μ and σ. Constraints on point features to distribution features: Now, to change the traditional point features into Gaussian distribution features N(μ,σ), we need to constrain this new feature. The 2-Wasserstein distance between the text feature sequence vector and the image visual feature sequence vector is defined as:

[0035] The samples with different bottom corner values ​​represent the distance between the features of the two samples.

[0036] Where μ is the feature center vector and σ is the standard deviation vector, the vector obtained after PDFE is in the form N(μ,σ). After measuring their distance, we use the following formula to convert this distance into similarity. The formula for linearly transforming the distance into a similarity score is:

[0037] Where a>0: scaling factor (the negative sign ensures that the smaller the distance, the higher the similarity), and b is the offset (adjusting the similarity benchmark). For n image-text pairs in a batch, this application uses the Infonce loss as follows:

[0038] This is the contrastive learning loss from image to text.

[0039]

[0040] This is the contrastive learning loss from text to image.

[0041]

[0042] The parameters are: i: Represents the i-th positive sample pair in the current training batch. I_i: The image feature vector of the i-th positive sample pair. T_i: The text feature vector of the i-th positive sample pair. T_n: The text feature vector of the n-th sample in the batch. In the summation of the denominator, n iterates through the entire batch from 1 to N. N: The size of a training batch.

[0043] s(I, T): Similarity function. It takes feature vectors from the image and text and calculates the similarity score between them. It is usually a simple dot product or cosine similarity.

[0044] ε: Temperature parameter. This is a learnable or preset hyperparameter used to adjust the sharpness of the Logits distribution. A smaller ε makes the distribution sharper, and the model focuses more on difficult negative samples. A larger ε makes the distribution smoother, and the model learns more gently.

[0045] in This is a learned temperature parameter; the above is an introduction to the loss parameter. Positive sample enhancement and negative sample attenuation are performed based on this loss.

[0046] S5. Introduce an uncertainty modeling module based on evidence-based deep learning (EDL), such as... Figure 3 The similarity between the text feature sequence vector and the image visual feature sequence is converted into a Dirichlet distribution. Based on the Dirichlet distribution, the possible values ​​of the alignment probability are simulated after evidence modeling. Based on the loss function, the evidence quantity of positive sample pairs is higher than that of negative sample pairs, thereby achieving robust suppression of modality conflict and insufficient information samples, and improving matching accuracy and robustness.

[0047] Specifically, to address the limitations of traditional softmax classifiers in cross-modal alignment, we propose an uncertainty modeling method based on evidence theory. Inspired by Evidential Deep Learning (EDL), we introduce an evidence theory framework and quantify the uncertainty in the cross-modal matching process by constructing a Dirichlet distribution. Specifically, given image-text global feature pairs... First, a simple similarity measure is performed on them. In this application, cosine similarity is used as the measure. The purpose of this part is to quantify the desired image-text matching degree. Then, the ReLU nonlinear transformation activation function is used to calculate the amount of evidence supporting the matching pair. :

[0048] in This represents the ReLU composite activation function acting on two modalities. Based on this equation, we can extract bidirectional evidence, i.e., image-to-text evidence. Text-to-image evidence .

[0049] Based on subjective logic, it is possible to measure the belief probability of each pair of cross-modal alignments. and the uncertainty of overall alignment The specific calculation method is as follows:

[0050]

[0051] in It can be viewed as the concentration parameter of the Dirichlet distribution, while the belief probability... This can be understood as derived from the corresponding Dirichlet distribution parameters. The subjective confidence assessment is derived from [previous method]. In text-based person retrieval, we can intuitively view it as an N-way classification problem, where a query is classified as having a positive cross-modal alignment. Traditional neural networks output a probability distribution for each possible alignment of a given pair of samples, while by employing an evidence-parameterized Dirichlet distribution, we can characterize the density function of each probability assignment, thus modeling the second-order probability and the uncertainty of cross-modal alignment. Essentially, the Dirichlet distribution is a probability density function that can be used to model the possible values ​​of the alignment probability:

[0052] in It is the alignment probability obtained after evidence modeling. For N-dimensional beta functions, It is an N-dimensional unit simplex.

[0053] The Evidence-Based Penalty for Text-Based Person Retrieval extends traditional EDL to text-based person retrieval tasks. First, it employs the traditional EDL loss function, adjusting the cross-modal alignment probability... Approaching real labels (N-dimensional vector), assign an appropriate penalty term to each retrieval result. The density follows a parameterized Dirichlet distribution. Therefore, the loss function It can be represented as:

[0054] While the above formula ensures that the evidence value of positive sample pairs is higher than that of negative sample pairs, it does not guarantee that the evidence value of negative sample pairs will drop to zero. Therefore, a KL divergence term is introduced. To penalize deviations in uncertain alignment, it is defined as:

[0055] The final loss is as follows:

[0056] Finally, we perform end-to-end training on the overall framework, using two common loss functions. and The combined constraints ensure the implementation of our text-to-image person retrieval method, which integrates point-to-surface feature alignment and uncertainty modeling. This invention is the first in the field of text-to-image person re-identification to introduce a distributed feature alignment module and a Dirichlet distribution-based evidence modeling mechanism: the distributed feature alignment module is the first in this field to use a Gaussian distribution for multi-granular modeling of modal features, effectively improving the expressive power of intra-class variability between modalities; the uncertainty modeling module applies EDL theory to image-text modal matching based on person re-identification, modeling the matching confidence level, and possessing the advantages of explicitly expressing model confidence and adaptive fusion capabilities.

[0057] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A text-image pedestrian re-identification method for semantic ambiguity and decision overconfidence, characterized in that, The method comprises the following steps: obtaining pedestrian image data and pedestrian text data, and preprocessing the pedestrian image data and the pedestrian text data respectively; inputting the preprocessed pedestrian text data into a Transformer text encoder in a CLIP model, first passing through a word embedding layer and position encoding to generate a text feature sequence, then extracting its context correlation features through a multi-layer self-attention structure, and finally outputting a fixed-dimension text feature sequence vector; at the same time, inputting the preprocessed pedestrian image data into a Transformer visual encoder in the CLIP model, first dividing the pedestrian image data into a plurality of fixed-size image patches and embedding them into patch tokens, adding position encoding and inputting them into a ViT structure, extracting visual context features through a Transformer unit, and finally outputting a fixed-dimension image visual feature sequence vector; inputting the text feature sequence vector and the image visual feature sequence vector into a distributed feature alignment module, in which the text feature sequence vector and the image visual feature sequence vector are first encoded with Gaussian distribution, and then the alignment relationship between the mean vector and the variance vector is optimized to explicitly model the intra-class semantic variation between different modal data; inputting the text feature sequence vector and the image visual feature sequence vector output by the distributed feature alignment module into an uncertainty modeling module based on evidence deep learning, converting the similarity of the text feature sequence vector and the image visual feature sequence vector into a Dirichlet distribution form, simulating the possible values of the alignment probability based on evidence modeling based on the Dirichlet distribution, and constraining the evidence quantity of the positive sample pair to be higher than that of the negative sample pair based on a loss function.

2. The text-image pedestrian re-identification method for semantic ambiguity and overconfidence in decision-making according to claim 1, characterized in that, The preprocessing of the pedestrian image data comprises uniformly cropping the pedestrian image data into fixed-size RGB images and performing normalization processing.

3. The text-image pedestrian re-identification method for semantic ambiguity and overconfidence in decision making according to claim 1, characterized in that, The preprocessing of the pedestrian text data comprises converting the pedestrian text data to lowercase, removing special characters, and then performing word-based segmentation coding.

4. The text-image pedestrian re-identification method for semantic ambiguity and overconfidence in decision making according to claim 1, characterized in that, The distributed feature alignment module comprises a first Feed Forward module, a multi-head operation module, a second Feed Forward module and a third Feed Forward module. The input data of the distributed feature alignment module first passes through the first Feed Forward module and then enters the multi-head operation module. Each operation module comprises a mean path and a variance path, which are used to map the input data into queries, keys and values for attention operations. The mean path of the multi-head operation module is concatenated and superimposed with the input data of the distributed feature alignment module through the second Feed Forward module for output. The variance path of the multi-head operation module is concatenated and output through the third Feed Forward module.

Images