A pedestrian retrieval method based on image and text fusion and attention guidance

By introducing key points information of human body in the CLIP model, using the cross attention mechanism to reduce background and occlusion interference, the problem of poor retrieval effect in graphic and text pedestrian search is solved, efficient and accurate pedestrian graphic and text retrieval and reduce labeling cost .

CN118736625BActive Publication Date: 2024-11-15GUANGZHOU BAOLUN ELECTRONICS CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411210396.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-11-15
Estimated Expiration
2044-08-30

AI Technical Summary

Technical Problem

The existing graphic pedestrian search methods are poor in retrieval when dealing with background information and occlusion, and relying on self-supervised comparison learning leads to high labeling costs.

Method used

The human body key point information is introduced through the cross attention mechanism, combined with the CLIP model and the human body key point detection model, reduce interference from background and occlusion information, improve retrieval accuracy and reduce labeling costs.

Benefits of technology

It improves the accuracy of pedestrian graphic and text retrieval, reduces the cost of manual labeling, and enhances the training efficiency and retrieval ability of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118736625B_ABST
    Figure CN118736625B_ABST
Patent Text Reader

Abstract

The invention discloses a method for image-text pedestrian retrieval based on fusion key point attention guidance, comprising the following steps: obtaining an image-text pedestrian data set and an image data set; constructing a human key point detection model, training the human key point detection model, and obtaining a trained human key point detection model; constructing a CLIP model, wherein the CLIP model includes a text branch and an image branch, and according to a cross-attention mechanism, the CLIP model is trained in combination with the output of the human key point detection model; inputting pedestrian images and pedestrian texts into the trained CLIP model in real time, and the CLIP model retrieves and matches corresponding pedestrians in the pedestrian images according to the pedestrian texts. The invention introduces human key point information into the CLIP model through the cross-attention mechanism, adds additional supervisory signals, reduces interference from occlusion and background information, and reduces manual annotation costs while improving the accuracy of pedestrian image-text retrieval.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method for image-text pedestrian retrieval based on fusion of key point attention guidance. Background Art

[0002] Due to the inconsistency between image and text models, most of the early image-text multimodal pedestrian retrieval methods used a combination of convolutional neural networks (CNN) and long short-term memory neural networks (LSTM) or gated recurrent units (GRU) to align image and text information. In recent years, with the development of the Transformer model, the CLIP model with stronger image-text retrieval capabilities has been proposed. CLIP is a self-supervised contrastive learning model trained from a large number of image-text pairs. It has very strong image-text retrieval generalization capabilities, but this generalization capability cannot be directly applied to fine-grained tasks such as pedestrian retrieval. It is necessary to add pedestrian information to fine-tune the model on its basis. In actual scene applications, the pedestrian images collected often contain a lot of background information, or some parts of the body are occluded, and the corresponding text cannot accurately describe the occlusion or missing of the body, resulting in poor retrieval results. This is a difficult problem in image-text pedestrian retrieval tasks.

[0003] In previous image-text pedestrian retrieval methods, it is common to add an attention mechanism on the basis of the own network to enhance the ability to extract fine-grained features of images and texts, or to improve the fusion effect of image and text information. Although these methods can enhance the feature extraction and fusion capabilities of image-text pedestrian retrieval models to a certain extent, relying solely on the model's own self-supervised comparative learning often requires a larger image and text sample size and complexity, which greatly increases the cost of manual labeling.

[0004] Therefore, there is a need for a pedestrian image and text retrieval method that can improve efficiency and accuracy while reducing annotation costs. Summary of the invention

[0005] To solve the above problems, the present invention provides a method for pedestrian retrieval in images and texts based on fusion key point attention guidance. The human key point information is introduced into the CLIP model through the cross-attention mechanism, and additional supervision signals are added to reduce the interference of occlusion and background information. While improving the accuracy of pedestrian image and text retrieval, the cost of manual annotation is reduced.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A method for image-text pedestrian retrieval based on fusion key point attention guidance includes the following steps:

[0008] S1. Obtain a text-and-graphic pedestrian dataset and an image dataset, wherein the text-and-graphic pedestrian dataset is a dataset with labels, the text-and-graphic pedestrian dataset includes a number of pedestrian image data and pedestrian text data corresponding to the images, and the image dataset is an image dataset with labels;

[0009] S2. Build a human key point detection model, input an image data set into the human key point detection model, train the human key point detection model, and obtain a trained human key point detection model;

[0010] S3. construct a CLIP model, wherein the CLIP model includes a text branch and an image branch, input a text-image pedestrian dataset into the CLIP model and the trained human key point detection model, and train the CLIP model according to the cross attention mechanism in combination with the output of the trained human key point detection model to obtain a trained CLIP model;

[0011] S4. Input pedestrian images and pedestrian texts to the trained CLIP model in real time. The CLIP model retrieves and matches the corresponding pedestrians in the pedestrian images based on the pedestrian texts.

[0012] Furthermore, in step S2, the backbone network of the human key point detection model is a ViT-B model, and the ViT-B model includes a Patch Embedding module, a Transformer module and a decoder module. The Transformer module includes an encoder, and the encoder in the Transformer module includes a normalization layer, a multi-head self-attention layer and a feedforward layer, and the multi-head attention layer includes multiple self-attention modules.

[0013] Furthermore, the formula of the self-attention module is:

[0014]

[0015] Among them, Q is the query vector, K is the key vector, V is the value vector, and Softmax() is the normalization function. is the scaling factor.

[0016] Furthermore, the workflow of the human key point detection model includes: inputting an image vector into the human key point detection model, the Patch Embedding module divides the image vector into several patches of the same size, and inputting the several patches into the Transformer module in turn. For each patch, the same patch is simultaneously input into all self-attention modules in the multi-head self-attention layer, each self-attention module outputs a vector after calculating the patch, all the obtained vectors are concatenated and linearly transformed to obtain a prediction result with the same dimension as the patch.

[0017] Furthermore, in step S2, the image data set is input into the human key point detection model to train the human key point detection model. The specific implementation method is: the image vectors in the image data set are input to the human key point detection model one by one, the human key point detection model outputs the prediction result, the loss between the prediction result and the data set label is calculated, and the parameters of the human key point detection model are optimized according to the loss until the number of training times reaches a preset threshold or the loss is less than the preset loss, the training is terminated, and a trained human key point detection model is obtained.

[0018] Furthermore, in step S3, the specific principle of the cross-attention mechanism is: the cross-attention mechanism has two different vector inputs, namely vector X1 and vector X2. Vector X1 is transformed into Query through linear transformation, and vector X2 is transformed into Key and Value through linear transformation. Query and Key are dot-producted, and the result after dot product is normalized by exponential function to obtain attention weight Ws. Ws is multiplied by Value to obtain the output of the cross-attention mechanism.

[0019] Furthermore, in step S3, the CLIP model includes an image branch and a text branch, the image branch is used for image encoding, and the text branch is used for text encoding. When performing image-text pedestrian retrieval, pedestrian image data is input into the image branch of the CLIP model, and an image feature vector is output; corresponding pedestrian text data is input into the text branch of the CLIP model, and a text feature vector is output; the image feature vector and the text feature vector are multiplied to obtain a similarity matrix, and the loss of the CLIP model is calculated.

[0020] Furthermore, the calculation of the loss of the CLIP model is specifically implemented as follows: the formula for calculating the loss of the CLIP model is:

[0021] , where LOSS represents the loss of the CLIP model, is the loss of image feature vector, is the loss of text feature vector;

[0022] The formula for calculating the loss of the image feature vector is:

[0023] , where CELoss() is the cross entropy loss function;

[0024] The formula for calculating the loss of the text feature vector is:

[0025] , where CELoss() is the cross entropy loss function.

[0026] Further, in step S3, the CLIP model is trained according to the cross-attention mechanism in combination with the output of the trained human key point detection model. The specific implementation method includes: in each training, a pedestrian image data is input into the image branch of the CLIP model, and the feature sequence output by the image branch is used as the input vector X2 of the cross-attention mechanism; the same pedestrian image data is input into the human key point detection model, and the feature sequence output by the human key point detection model is used as the input vector X1 of the cross-attention mechanism to obtain the output of the cross-attention mechanism; the output of the cross-attention mechanism is used as the output of the image branch; the corresponding pedestrian text data is input into the text branch of the CLIP model to obtain the output of the text branch; the loss between the output of the image branch and the output of the text branch and the dataset label is calculated; the parameters of the CLIP model are optimized according to the loss; one training is completed; several trainings are performed until the number of trainings reaches a preset threshold or the loss is less than the preset loss, and the training is terminated to obtain a trained CLIP model.

[0027] Furthermore, the image branch in the CLIP model is a ViT-B model, and the text branch is a BERT model.

[0028] Through the above technical scheme, the present invention has the following beneficial effects: on the basis of the CLIP model, a human key point detection model is added, and the human key point information is passed through the Cross-Attention mechanism module, so as to more efficiently guide the image encoding branch of CLIP to focus on the human body itself, while reducing the influence of interference information such as occlusion or background, thereby improving the training efficiency of the model and the pedestrian retrieval accuracy, and reducing the labeling cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 The present invention is a schematic diagram of the overall process of a method for image-text pedestrian retrieval based on fusion of key point attention guidance. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] See also Figure 1 , a method for image-text pedestrian retrieval based on fusion key point attention guidance, comprising the following steps:

[0033] S1. Obtain a text-and-graphic pedestrian dataset and an image dataset, wherein the text-and-graphic pedestrian dataset is a dataset with labels, the text-and-graphic pedestrian dataset includes a number of pedestrian image data and pedestrian text data corresponding to the images, and the image dataset is an image dataset with labels;

[0034] S2. Build a human key point detection model, input an image data set into the human key point detection model, train the human key point detection model, and obtain a trained human key point detection model;

[0035] S3. construct a CLIP model, wherein the CLIP model includes a text branch and an image branch, input a text-image pedestrian dataset into the CLIP model and the trained human key point detection model, and train the CLIP model according to the cross attention mechanism in combination with the output of the trained human key point detection model to obtain a trained CLIP model;

[0036] S4. Input pedestrian images and pedestrian texts to the trained CLIP model in real time. The CLIP model retrieves and matches the corresponding pedestrians in the pedestrian images based on the pedestrian texts.

[0037] In an optional embodiment, in step S2, the backbone network of the human key point detection model is a ViT-B model, and the ViT-B model includes a Patch Embedding module, a Transformer module and a decoder module. The Transformer module includes an encoder, and the encoder in the Transformer module includes a normalization layer, a multi-head self-attention layer and a feedforward layer, and the multi-head attention layer includes multiple self-attention modules.

[0038] The ViT-B model is the VisionTransformerBase model. The Patch Embedding module is used to divide the input image data into several patches, and then convert each patch into a vector of fixed dimension. Through the above operation, the original two-dimensional image is converted into a one-dimensional sequence, which is convenient for the Transformer module to process. Then, these vectors will be sent to the Transformer encoder and decoder for processing to complete tasks such as image classification and target detection.

[0039] Transformer is a deep learning model based on the self-attention mechanism. It abandons the recurrent neural network widely used in previous sequence processing tasks and instead uses the self-attention mechanism to directly calculate the relationship between elements in the sequence, thereby effectively capturing long-distance dependencies. The self-attention mechanism enables the model to weigh the importance of different elements in the input sequence and dynamically adjust their impact on the output.

[0040] In an optional embodiment, the formula of the self-attention module is:

[0041]

[0042] Among them, Q is the query vector, K is the key vector, V is the value vector, and Softmax() is the normalization function. is the scaling factor.

[0043] The self-attention mechanism module has only one input X, and obtains three intermediate values ​​Query, Key, and Value through linear transformation.

[0044] The query vector represents part of the query, which usually corresponds to information related to the current position (such as the current position in the decoding phase) or the entire sequence (encoding phase). The query is used to measure the relevance of information at all other positions. When the model needs to generate the output at the current moment, the query vector plays a role in finding relevant information.

[0045] The Key vector represents the "index" or "lookup table" of the information at each position in the input sequence, which can be understood as the identification feature of each position. When calculating the attention weight, the Key vector of each position is compared with the Query vector, and the correlation between the Query and each position in the sequence is evaluated by calculating the dot product or other similarity functions. The Key vector determines which input position information is most important to the current Query.

[0046] The Value vector is the carrier of the "content" or "context information" associated with each position. It contains the information that is actually to be paid attention to and combined into the output, that is, the information at each position that we believe is valuable for subsequent processing. When the Key and Query at a certain position have a high match, the weight of the corresponding Value vector will increase, so this part of information will be considered more when calculating the final output. After calculating the Attention weight, the Value vector does not directly participate in the similarity calculation, but is weighted by the corresponding Attention weight, and then the accumulated result is a new vector that comprehensively considers the context information of the entire sequence.

[0047] In an optional embodiment, the workflow of the human key point detection model includes: inputting an image vector into the human key point detection model, the Patch Embedding module divides the image vector into several patches of the same size, and inputting the several patches into the Transformer module in turn. For each patch, the same patch is simultaneously input into all self-attention modules in the multi-head self-attention layer, each self-attention module outputs a vector after calculating the patch, all the obtained vectors are spliced ​​and linearly transformed to obtain a prediction result with the same patch dimension.

[0048] The multi-head self-attention mechanism uses the self-attention module to perform multiple parallel repeated calculations, and each parallel calculation is called an attention head. Each self-attention module splits its query, key, and value parameter matrix, and each split passes through a separate attention head independently. Finally, all these identical attention calculations are merged together to produce the final attention score. It can capture and express the multiple connections and subtle differences between each word in a more detailed way.

[0049] In an optional embodiment, in step S2, the image data set is input into the human key point detection model to train the human key point detection model. The specific implementation method is: the image vectors in the image data set are input to the human key point detection model one by one, the human key point detection model outputs the prediction result, the loss between the prediction result and the data set label is calculated, and the parameters of the human key point detection model are optimized according to the loss until the number of training times reaches a preset threshold or the loss is less than the preset loss, the training is terminated, and a trained human key point detection model is obtained.

[0050] The model training uses the MSCOCO dataset, which is a large-scale image dataset developed and maintained by Microsoft. The dataset annotation types correspond to tasks including object detection, key point detection, instance segmentation, stuff segmentation (objects without a specific shape), panoramic segmentation of human key points, human density detection, etc.

[0051] In an optional embodiment, in step S3, the specific principle of the cross-attention mechanism is: the cross-attention mechanism has two different vector inputs, namely vector X1 and vector X2. Vector X1 is transformed into Query through linear transformation, and vector X2 is transformed into Key and Value through linear transformation. Query is dot-producted with Key, and the result after dot product is normalized by exponential function to obtain attention weight Ws. Ws is multiplied by Value to obtain the output of the cross-attention mechanism.

[0052] Cross attention is mainly used to handle multimodal tasks or scenarios where inputs from different sources need to be associated. The core idea is that the elements of one input sequence pay attention to the elements of another input sequence, thereby establishing a connection between different inputs. The cross attention mechanism is used to process the semantic relationship between two sequences by calculating attention on two different sequences. The cross attention mechanism combines two embedding sequences of the same dimension asymmetrically, where one sequence is used as the query Q input and the other sequence is used as the key K and value V input.

[0053] In an optional embodiment, in step S3, the CLIP model includes an image branch and a text branch, the image branch is used for image encoding, and the text branch is used for text encoding. When performing image-text pedestrian retrieval, pedestrian image data is input into the image branch of the CLIP model, and an image feature vector is output; corresponding pedestrian text data is input into the text branch of the CLIP model, and a text feature vector is output; the image feature vector and the text feature vector are multiplied to obtain a similarity matrix, and the loss of the CLIP model is calculated.

[0054] The CLIP model (Contrastive Language-Image Pre-training) is a model for joint representation learning of images and texts. The core idea of ​​CLIP is to pre-train a model through contrastive learning so that it can understand the relationship between images and texts.

[0055] In an optional embodiment, the calculation of the loss of the CLIP model is specifically implemented as follows: the formula for calculating the loss of the CLIP model is:

[0056] , where LOSS represents the loss of the CLIP model, is the loss of image feature vector, is the loss of text feature vector;

[0057] The formula for calculating the loss of the image feature vector is:

[0058] , where CELoss() is the cross entropy loss function;

[0059] The formula for calculating the loss of the text feature vector is:

[0060] , where CELoss() is the cross entropy loss function.

[0061] The loss function used by CLIP is a symmetric contrast loss function. Specifically, for each image-text pair, the model calculates losses in two directions: Image-to-text loss: Calculates the similarity between the image feature vector and the text feature vector, and optimizes to maximize the similarity of the matching image-text pairs. Text-to-image loss: Calculates the similarity between the text feature vector and the image feature vector, and optimizes to maximize the similarity of the matching text-image pairs.

[0062] In an optional embodiment, in step S3, the CLIP model is trained according to the cross-attention mechanism in combination with the output of the trained human key point detection model. The specific implementation method includes: in each training, a pedestrian image data is input into the image branch of the CLIP model, and the feature sequence output by the image branch is used as the input vector X2 of the cross-attention mechanism; the same pedestrian image data is input into the human key point detection model, and the feature sequence output by the human key point detection model is used as the input vector X1 of the cross-attention mechanism to obtain the output of the cross-attention mechanism; the output of the cross-attention mechanism is used as the output of the image branch; the corresponding pedestrian text data is input into the text branch of the CLIP model to obtain the output of the text branch; the loss between the output of the image branch and the output of the text branch and the dataset label is calculated; the parameters of the CLIP model are optimized according to the loss; one training is completed; several trainings are performed until the number of trainings reaches a preset threshold or the loss is less than the preset loss, and the training is terminated to obtain a trained CLIP model.

[0063] The 12th layer output in CLIP's image encoder ViT-B is used as the input vector X2 of the Cross-Attention model; the weights of the ViT-Point model are frozen, the ViT-Point model is not updated, and the 12th layer output in its backbone network ViT-B is used as another input vector X1 of the Cross-Attention module. In this way, the cross-attention mechanism can be used to let the key point information of the human body guide the CLIP model to complete the fine-tuning training of pedestrian image-text comparison, and more effectively achieve the pedestrian retrieval task.

[0064] In an optional embodiment, the image branch in the CLIP model is a ViT-B model, and the text branch is a BERT model.

[0065] The BERT model (Bidirectional Encoder Representation from Transformer) is a pre-trained language model based on the Transformer architecture. It captures the deep bidirectional representation of language by pre-training on large-scale text data, and then fine-tunes it for different natural language processing tasks.

[0066] The embodiment disclosed in this specification is only an example of a unilateral feature of the present invention, and the protection scope of the present invention is not limited to this embodiment, and any other functionally equivalent embodiments fall within the protection scope of the present invention. For those skilled in the art, various other corresponding changes and deformations can be made according to the technical solutions and concepts described above, and all of these changes and deformations should fall within the protection scope of the claims of the present invention.

Claims

1. A pedestrian retrieval method based on image and text fusion with key point attention guidance. It is characterized in that The following steps are involved: S1. Obtain a text-and-graphic pedestrian dataset and an image dataset, wherein the text-and-graphic pedestrian dataset is a dataset with labels, the text-and-graphic pedestrian dataset includes a number of pedestrian image data and pedestrian text data corresponding to the images, and the image dataset is an image dataset with labels; S2. construct a human key point detection model, input an image data set into the human key point detection model, train the human key point detection model, and obtain a trained human key point detection model, wherein the backbone network of the human key point detection model is a ViT-B model, and the ViT-B model includes a Patch Embedding module, a Transformer module, and a decoder module, wherein the Transformer module includes an encoder, and the encoder in the Transformer module includes a normalization layer, a multi-head self-attention layer, and a feedforward layer, and the multi-head self-attention layer includes multiple self-attention modules; The workflow of the human key point detection model includes: inputting an image vector into the human key point detection model, the Patch Embedding module divides the image vector into several patches of the same size, and the several patches are sequentially input into the Transformer module. For each patch, the same patch is simultaneously input into all self-attention modules in the multi-head self-attention layer, each self-attention module outputs a vector after calculating the patch, and all the obtained vectors are concatenated and linearly transformed to obtain a prediction result with the same dimension as the patch; S3. Construct a CLIP model, which includes a text branch and an image branch. Input a text-and-image pedestrian dataset into the CLIP model and the trained human key point detection model. According to the cross-attention mechanism, the specific principle of the cross-attention mechanism is as follows: the cross-attention mechanism has two different vector inputs, namely vector X1 and vector X2. Vector X1 is transformed into Query through linear transformation, and vector X2 is transformed into Key and Value through linear transformation. Query and Key are dot-producted, and the result after dot product is normalized by exponential function to obtain the attention weight Ws. Ws is multiplied by Value to obtain the output of the cross-attention mechanism. The CLIP model is trained in combination with the output of the trained human key point detection model. In each training, the cross-attention mechanism is input into the CLIP model. The image branch inputs a pedestrian image data, and uses the feature sequence output by the image branch as the input vector X2 of the cross-attention mechanism. The same pedestrian image data is input to the human key point detection model, and the feature sequence output by the human key point detection model is used as the input vector X1 of the cross-attention mechanism to obtain the output of the cross-attention mechanism. The output of the cross-attention mechanism is used as the output of the image branch. The corresponding pedestrian text data is input to the text branch of the CLIP model to obtain the output of the text branch. The loss between the output of the image branch and the output of the text branch and the dataset label is calculated, and the parameters of the CLIP model are optimized according to the loss. A training is completed, and several trainings are performed until the number of trainings reaches a preset threshold or the loss is less than the preset loss. The training is terminated to obtain a trained CLIP model. S4. Input pedestrian images and pedestrian texts to the trained CLIP model in real time. The CLIP model retrieves and matches the corresponding pedestrians in the pedestrian images based on the pedestrian texts.

2. According to claim 1, a method for image-text pedestrian retrieval based on fusion key point attention guidance, It is characterized in that The formula of the self-attention module is: Among them, Q is the query vector, K is the key vector, V is the value vector, and Softmax() is the normalization function. is the scaling factor.

3. According to claim 2, a method for image-text pedestrian retrieval based on fusion key point attention guidance, It is characterized in that In step S2, the image data set is input into the human key point detection model to train the human key point detection model. The specific implementation method is: the image vectors in the image data set are input to the human key point detection model one by one, the human key point detection model outputs the prediction result, the loss between the prediction result and the data set label is calculated, and the parameters of the human key point detection model are optimized according to the loss until the number of training times reaches a preset threshold or the loss is less than the preset loss, the training is terminated, and a trained human key point detection model is obtained.

4. According to claim 1, a method for image-text pedestrian retrieval based on fusion key point attention guidance, It is characterized in that In step S3, the CLIP model includes an image branch and a text branch, the image branch is used for image encoding, and the text branch is used for text encoding. When performing image-text pedestrian retrieval, pedestrian image data is input into the image branch of the CLIP model, and an image feature vector is output. Corresponding pedestrian text data is input into the text branch of the CLIP model, and a text feature vector is output. The image feature vector and the text feature vector are multiplied to obtain a similarity matrix, and the loss of the CLIP model is calculated.

5. According to claim 4, a method for image-text pedestrian retrieval based on fusion key point attention guidance, It is characterized in that The specific implementation of calculating the loss of the CLIP model is as follows: the formula for calculating the loss of the CLIP model is: , where LOSS represents the loss of the CLIP model, is the loss of image feature vector, is the loss of text feature vector; The formula for calculating the loss of the image feature vector is: , where CELoss() is the cross entropy loss function; The formula for calculating the loss of the text feature vector is: , where CELoss() is the cross entropy loss function.

6. According to claim 5, a method for image-text pedestrian retrieval based on fusion key point attention guidance, It is characterized in that The image branch in the CLIP model is the ViT-B model, and the text branch is the BERT model.