Pedestrian re-identification method based on extensible representation learning

Abstract

The invention provides a pedestrian re-identification method based on extensible representation learning. The system mainly comprises a visual representation learning module, a text representation learning module and a pedestrian retrieval module. The visual representation learning module utilizes visual coding features of ViT structures from two spaces to enhance the image distinguishing capability, and fusion is carried out through a causal attention mechanism; and the text representation learning module eliminates deviation existing in a data set by means of a demonstration sample and by adopting a causal attention mechanism, so that the pedestrian re-recognition performance is improved, and a pedestrian retrieval result is more accurate.

Description

Technical Field

[0001] The present invention relates to the field of multimedia information retrieval, and in particular to a pedestrian re-identification method based on scalable representation learning. Background Art

[0002] As a key technology in the intersection of computer vision and natural language processing, text-based pedestrian re-identification plays a key role in many practical scenarios such as intelligent security and pedestrian trajectory tracking. It can retrieve pedestrian images across cameras in different scenarios through text descriptions, effectively assisting security personnel to quickly lock targets, construct character movement trajectories, and improve the efficiency and accuracy of intelligent surveillance video analysis and retrieval.

[0003] Most existing studies focus on exploring the semantic alignment of image-text pairs in a common representation space. It can be observed that the model training process used by these methods on existing public benchmark datasets is based on a strong assumption: the modal information of images and texts is complete and the datasets have a uniform distribution, that is, the model can extract effective feature representations of images and texts, and at the same time they are not biased, which is obviously unrealistic. In addition, most current semantic representation-based methods lack scalability and have difficulty distinguishing similar images from biased texts (such as color bias), resulting in poor person re-identification performance. This is mainly due to two aspects: (1) First, a large number of cross-modal person re-identification methods rely on the features encoded by the CLIP model, and this text-aligned representation relies on the text. That is, if two similar pedestrian images can be described by the same text sentence, such as "this man is wearing a yellow shirt, gray shorts, tennis shoes, and a backpack", then their representations must be similar. (2) Secondly, the text descriptions currently used are biased, which may affect the semantics learned by the model. For example, according to the word frequency statistics of a certain dataset, "black" accounts for 38% of the color of pedestrians' clothes, while "red" accounts for only 2%. Similar uneven distribution biases will cause the model to favor high-frequency words, thereby affecting the overall performance of the model. Therefore, how to deal with bias in the modeling process to ensure that the model accurately captures the underlying semantics and performs well in multivariate scenarios still has room for improvement, but it is rarely considered.

[0004] Therefore, in view of the above problems, the present invention proposes a pedestrian re-identification method based on scalable representation learning. It consists of a visual representation learning module, a text representation learning module and a pedestrian retrieval module. Specifically, the visual representation learning module uses the visual encoding features of the ViT structure from two spaces to enhance the discriminative representation of the image, and performs representation fusion through a causal attention mechanism. For the text representation learning module, eliminating the dataset bias through demonstration samples is a reliable strategy. At the same time, a causal attention mechanism is adopted to eliminate the bias existing in the dataset, thereby causally intervening in the cross-modal input, thereby improving the performance of pedestrian re-identification. A structural causal model with a do operator is used here to treat the dataset bias as a confounding factor and subsequently eliminate their confounding effects. Summary of the invention

[0005] This paper proposes a pedestrian re-identification method based on scalable representation learning to solve the above problems and improve the accuracy of pedestrian re-identification. It mainly includes the following steps:

[0006] Step S1: obtaining training samples, each training sample including a training image and a training text description corresponding to the training image;

[0007] Step S2: using an image-text encoder to extract features of the training image and training text description respectively, to obtain training image features and training text features;

[0008] Step S3: Use the non-text aligned visual encoder to extract training image features, give priority to the image itself, and effectively separate similar images in the semantic space. Then use vector cosine similarity to obtain the image and text features of the dimensions corresponding to the backbone model, which are the demonstration sample features;

[0009] Step S4: Causal representation learning is performed using the geometric features of the do operator, which adopts a structural causal model, considers dataset biases as confounding factors, and subsequently eliminates their confounding effects;

[0010] Step S5: training the model until convergence. Then, the pedestrian image features and text features are input into the trained pedestrian re-identification model to obtain a pedestrian re-identification result.

[0011] Specifically, the step S2 includes:

[0012] Extracting training image features from the training images respectively by using the image feature extractor and the text feature extractor of the CLIP model, and extracting training text features from the training text description;

[0013] The step S3 comprises:

[0014] Use the non-text aligned visual encoder EVA to extract training image features. Use CLIP's text feature extractor to extract demonstration sample features;

[0015] Specifically, in step S2 and step S3, the image and text feature extractors are the image feature extractor of the CLIP model and the text feature extractor of CLIP, as well as the EVA image feature extractor. Since the text-aligned visual image encoder from CLIP depends on text and is difficult to distinguish similar images, another non-text-aligned visual encoder EVA with the same ViT architecture is used as a supplement to the semantic space, and the two representations are fused through a causal attention mechanism to enhance the robustness of the visual features.

[0016] Specifically, the causal representation learning formula of step S4 can be expressed as:

[0017]

[0018] where X is the independent variable (or cause) and Y is the relevant dependent variable (or outcome); z usually represents a confounder; P(y|do(X)=x) is the causal relationship and represents the distribution of Y under the intervention of X=x, where X is set to x. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to further understand the features and technical contents of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings are only provided for reference and explanation and are not intended to limit the present invention.

[0020] Figure 1 The whole process diagram of the present invention is implemented;

[0021] Figure 2 This is a causal diagram of intervention based on the structural causal model. DETAILED DESCRIPTION

[0022] In order to further illustrate the technical means and effects of the present invention, the following is a detailed description of the intelligent security scene application and its accompanying drawings. These implementations are only used to explain the technical principles of the present invention and are not intended to limit the scope of application of the present invention.

[0023] See also Figure 1 The present invention provides a pedestrian re-identification method based on scalable representation learning, and the specific steps are as follows:

[0024] Step S1: obtaining training samples, each training sample including a training image and a training text description corresponding to the training image;

[0025] Step S2: using an image-text encoder to extract features of the training image and training text description respectively, to obtain training image features and training text features;

[0026] Specifically, the step S2 includes:

[0027] Extracting training image features from the training images respectively by using the image feature extractor and the text feature extractor of the CLIP model, and extracting training text features from the training text description;

[0028] Furthermore, CLIP's image encoder adopts the ViT architecture. The image is first divided into blocks of equal size, and then each block is mapped to a vector with fixed dimension to extract its embedding. After being combined with the position embedding, the image embedding is input into the ViT-L / 14 network architecture to learn the visual context feature representation, which contains 12 Transformer layers. X pairs are sampled in a batch size, and the index of each image and text is represented as i∈{1,...,N}, where "N" is the number of images and texts in a batch size.

[0029] Step S3: Use the non-text aligned visual encoder to extract training image features, give priority to the image itself, and effectively separate similar images in the semantic space. Then use vector cosine similarity to obtain the image and text features of the dimensions corresponding to the backbone model, which are the demonstration sample features;

[0030] Specifically, step S3 includes:

[0031] The non-text aligned visual encoder EVA is used to extract training image features. The text feature extractor of CLIP is used to extract demonstration sample features.

[0032] Specifically, in step S2 and step S3, the image and text feature extractors are respectively the image feature extractor of the CLIP model and the text feature extractor of CLIP, and the EVA image feature extractor.

[0033] Furthermore, the step S3 comprises:

[0034] Since the text-aligned visual encoder from CLIP is dependent on text and has difficulty distinguishing similar images, another non-text-aligned visual encoder EVA with the same ViT architecture is used as a complement to the semantic space to obtain the extended embedding for visual representation encoded with EVA. Therefore, in this module, the image features f i m and extended visual features They can be formulated as:

[0035]

[0036] Where V CLIP (·) and V EVA (·) are all visual projection functions. In addition, it is necessary to consider the appropriate fusion strategy of ViT features in different spaces. The present invention adopts the causal attention mechanism mentioned above to fuse these two representations and enhance the robustness of visual features.

[0037] Furthermore, the step S3 comprises:

[0038] For text representation, the text encoder of the CLIP model is used as the backbone for feature extraction to obtain text features f i t and demonstration sample characteristics The formula is:

[0039]

[0040] where L(·) is the text projection function.

[0041] Step S4: Utilize the geometric features of the do operator to perform causal representation learning, which adopts a structural causal model to treat dataset biases as confounding factors and subsequently eliminate their confounding effects;

[0042] Specifically, step S4 includes:

[0043] See also Figure 2 , specifically showing the causal graph: Q (text query), I (image), Y (prediction) and C (confounding factors, i.e., dataset bias), where nodes and edges represent endogenous variables and causal relationships, respectively. The causal representation learning formula can be expressed as:

[0044]

[0045] Where X is the independent variable (or cause) and Y is the dependent variable (or outcome) of interest. z usually represents a confounder. P(y|do(X)=x) is the causal relationship and represents the distribution of Y under the intervention X=x, where X is set to x.

[0046] Q,I→Y: In most methods, the model uses the text description Q and the image I to produce a prediction Y. This prediction process is usually considered biased because it ignores the dataset bias C, which is a factor that needs to be considered.

[0047] C→Q,I,Y: Dataset bias is represented as a confounding factor C, which can lead to spurious correlations between text input and target. Bias often varies in different datasets, such as spatial bias and temporal bias, most of which are difficult to quantify and usually unobservable.

[0048] Based on this, a decongested scalable representation learning framework with a causal attention mechanism is introduced here to perform causal intervention on the input, and the causal do-operator technique is used to derive an unbiased estimation formula.

[0049] Since the cross-modal pedestrian re-identification task mainly uses pedestrian images in the text retrieval gallery, the features from the CLIP model still need to be dominant. Therefore, a one-way process is adopted to apply causal attention for feature interaction, as shown below:

[0050]

[0051] Where M is the visual representation extracted by the CLIP model image encoder, and S is the visual representation encoded by the non-text aligned visual encoder EVA. In addition, i and j are row and column indices, and M[i,j] indicates that the i-th image can attend to the j-th image in the Transformer layer. Finally, the resulting dual visual representation f i ′m The expression is:

[0052]

[0053] In addition, dataset bias (e.g., color information) in person re-ID acts as an unobservable confounder, and we promote deconfounding learning and mitigate dataset bias by leveraging analogical reasoning, thereby achieving unbiased estimation without explicitly modeling the confounders. In particular, this process is biased because it does not consider the spurious correlations caused by the confounder C. The representation and target of the query sample can be approximated by the output of similar samples. This suggests that in order to compute the causal relationship on Y, the model input should contain a query sample and similar demonstration samples. Therefore, the demonstration samples should pay attention to their own visual-textual information to guide the query sample to learn analogically, which can be represented by the causal attention mechanism.

[0054] Similarly, text representation learning performs causal attention mechanism as follows:

[0055]

[0056] in and is the text representation encoded by the CLIP text encoder. In addition, i and j are row and column indices, and W[i,j] indicates that the i-th text can pay attention to the j-th text representation in the Transformer layer.

[0057] Step S5: training the model until convergence. Then inputting the pedestrian image features and text features into the trained pedestrian re-identification model to obtain the pedestrian re-identification result;

[0058] Specifically, step S5 includes:

[0059] Similarity learning is used to determine whether feature vectors belong to the same person. Euclidean distance is mainly used to calculate the distance score between the query information and the gallery image. Among them, pedestrian images of positive sample pairs will produce higher scores than negative sample pairs.

[0060]

[0061] where e M (·) and e T (·) are linear layers that project the embeddings into a multimodal embedding space. The losses are summarized as follows. For text, the text-to-image contrast loss L t2m It is expressed as:

[0062]

[0063] Use the obtained features to calculate the image-to-text loss L m2t ,

[0064]

[0065] The above text-image loss formula is the similarity of the two embeddings from the matching pair. Therefore, the contrast loss L can be obtained Con as follows:

[0066] L Con =L t2m +L m2t

[0067] Similarly, retrieval tasks based on text description query modality are also considered.

[0068]

[0069] in represents the set of masked text tags, and |C| represents the size of the vocabulary C. The predicted tag probability distribution is denoted by c h It means that q h is a one-hot vocabulary distribution, where the probability of the true tag is 1. Correspondingly, the ID loss L in pedestrian re-identification is ID ,

[0070]

[0071] where y i is the probability distribution of the true label, pi is the predicted probability distribution of the model. is the predicted output of the model, G represents the number of images in the training mini-batch, and i is the index of the category, i.e., i in 1-G. Finally, the overall loss is expressed as:

[0072]

[0073] Specifically, the pedestrian re-identification model retrieves pedestrian image features through the text description features of the queried pedestrian, and outputs a preset Rank-K sorted list corresponding to the pedestrian image features as the result of pedestrian re-identification. The pedestrian images in the ranked list are arranged according to the similarity between the pedestrian image features and the queried text description.

Claims

1. A person re-identification method based on scalable representation learning, characterized in that: The following steps are involved: Step S1: obtaining training samples, each training sample including: a training image and a training text description corresponding to the training image; Step S2: using an image-text encoder to extract features of the training image and training text description respectively, to obtain training image features and training text features; Step S3: Use the non-text aligned visual encoder to extract training image features, give priority to the image itself, and effectively separate similar images in the semantic space. Then use vector cosine similarity to obtain the image and text features of the dimensions corresponding to the backbone model, which are the demonstration sample features; Step S4: Causal representation learning is performed using the geometric features of the do operator, which adopts a structural causal model, considers dataset biases as confounding factors, and subsequently eliminates their confounding effects; Step S5: training the model until convergence. Then, the pedestrian image features and text features are input into the trained pedestrian re-identification model to obtain a pedestrian re-identification result.

2. The method for person re-identification based on scalable representation learning according to claim 1, characterized in that: The step S2 comprises: The image feature extractor and the text feature extractor of the CLIP model are used to extract training image features from the training images respectively, and the training text features are extracted from the training text descriptions.

3. The pedestrian re-identification method based on scalable representation learning according to claim 1, characterized in that: The step S3 comprises: Use the non-text aligned visual encoder EVA to extract training image features. Use CLIP's text feature extractor to extract demonstration sample features; Specifically, in step S2 and step S3, the image and text feature extractors are the image feature extractor of the CLIP model and the text feature extractor of CLIP, and the EVA image feature extractor. Since the text-aligned visual image encoder from CLIP depends on text and has difficulty distinguishing similar images, another non-text-aligned visual encoder EVA, which also has the ViT architecture, is used as a supplement to the semantic space.

4. The method for person re-identification based on scalable representation learning according to claim 1, characterized in that: The step S4 comprises: It is necessary to consider the appropriate fusion strategy of ViT visual encoding features in different spaces. Here, the causal attention mechanism is used to fuse these two representations to enhance the robustness of visual features. Since the cross-modal pedestrian re-identification task mainly uses pedestrian images in the text retrieval gallery, the features from the CLIP model still need to be dominant. Therefore, the causal attention mechanism is used for feature interaction using a one-way process as follows: where M is the visual representation encoded by the image encoder of the CLIP model with text supervision, and S is the visual representation encoded by the non-text aligned visual encoder; in addition, i and j are row and column indices, and M[i,j] indicates that the i-th image can attend to the j-th image representation in the Transformer layer; finally, the dual visual representation The expression is: Similarly, text representation learning performs causal attention mechanism as follows: in and is the text representation encoded by the text encoder of the CLIP model; in addition, i and j are row and column indices, and W[i,j] means that the i-th text can pay attention to the representation of the j-th text in the Transformer layer.

Images