Text-object retrieval method based on dynamic self-evolution information extraction and alignment

Abstract

The present invention discloses a text-target retrieval method based on dynamic self-evolution information extraction and alignment, and constructs a text-guided dynamic visual information filtering and component-level alignment network, including a basic image-text feature extraction module, a pedestrian-independent visual content filtering module and a pedestrian modality dynamic self-evolution module; in the basic image-text feature extraction module, a visual convolutional neural network and a language processing model are used to extract basic features of images and descriptions; in the pedestrian-independent visual content filtering module, a Transformer encoder layer is used to enhance fine-grained pixel-level visual features, and a prototype-guided attention layer is used to filter out noise unrelated to pedestrians from the enhanced visual features; finally, in the pedestrian modality dynamic self-evolution module, text descriptions are used to guide the autonomous evolution of image-level component masks, and the mask description is used to describe the emphasis degree of each pedestrian component to perform detailed component-level alignment and matching.

Description

Technical Field

[0001] The present invention belongs to the technical field of image-text retrieval and pedestrian re-identification, and particularly relates to a description-based pedestrian image retrieval method. Background Art

[0002] In recent years, pedestrian retrieval has received widespread attention and has been widely used in many fields, such as missing persons search and suspect tracking. This task requires retrieving corresponding pedestrian images from the image library for queries such as pedestrian photos or sentence descriptions. According to the type of query, pedestrian retrieval can be mainly divided into image-based pedestrian retrieval and language description-based pedestrian retrieval. Among them, image-based pedestrian retrieval requires at least one photo of the pedestrian to be retrieved as an index, and such photos are difficult to obtain in some actual scenarios. For example, in the field of criminal investigation, sometimes the surveillance camera does not capture the photo of the suspect. However, we can get the description of the suspect by the witness. At the same time, the information contained in the description is more comprehensive and accurate than the attribute features. Therefore, the research on description-based pedestrian image retrieval has great scenario needs and application value.

[0003] In description-based pedestrian image retrieval tasks, an important reason that limits image-text alignment and retrieval performance is the information asymmetry between images and descriptions, such as Figure 1 As shown in Figure 3, it mainly manifests itself in two aspects: (1) the image contains pedestrian-irrelevant content that is not mentioned in the description, such as background and occlusion; (2) the descriptions of different pedestrian parts are not equal, such as some pedestrian parts are not mentioned. Summary of the Invention

[0004] In order to overcome the shortcomings of the existing technology, the present invention provides a text-to-object retrieval method based on dynamic self-evolutionary information extraction and alignment, and constructs a text-guided dynamic visual information filtering and component-level alignment network, including a basic image-text feature extraction module, a pedestrian-independent visual content filtering module and a pedestrian modality dynamic self-evolution module; in the basic image-text feature extraction module, a visual convolutional neural network and a language processing model are used to extract the basic features of the image and description; in the pedestrian-independent visual content filtering module, a Transformer encoder layer is used to enhance the fine-grained pixel-level visual features, and a prototype-guided attention layer is used to filter out noise unrelated to pedestrians from the enhanced visual features; finally, in the pedestrian modality dynamic self-evolution module, text descriptions are used to guide the autonomous evolution of image-level component masks, and the masks are used to describe the emphasis degree of each pedestrian component to perform detailed component-level alignment and matching.

[0005] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0006] Step 1: Construct a text-guided dynamic visual information filtering and component-level alignment network, including a basic image-text feature extraction module, a pedestrian-independent visual content filtering module, and a pedestrian modality dynamic self-evolution module;

[0007] Step 1-1: The basic image-text feature extraction module includes a semantic feature extraction branch and an image feature extraction branch;

[0008] The semantic feature extraction branch uses a BERT network pre-trained on text data. For each text input, a special classification token is added to the beginning of each description. After the special classification token is processed by the BERT model, the hidden state in the BERT model's final output is used as the overall feature representation of the description.

[0009] The image feature extraction branch uses ResNet50 pre-trained on ImageNet as the baseline network backbone to extract pedestrian image features. The last downsampling operation in the image feature extraction branch is removed to increase the spatial scale of the feature map. The size of the feature map output by the model is set to (C, H, W), where C and (H, W) represent the channel dimension and feature map size, respectively.

[0010] Step 1-2: The pedestrian-independent visual content filtering module includes a Transformer encoder layer and a prototype-guided attention module;

[0011] Step 1-2-1: The Transformer encoder layer uses the correlation between feature vectors to enhance visual features. It includes a multi-head self-attention mechanism and a feedforward network. First, the features extracted by the Transformer encoder layer are flattened into H*W visual feature vectors as the input of the Transformer encoder layer. Then, the H*W visual feature vectors are passed through a fully connected layer to calculate the query vector, key vector, and value vector of each visual feature vector:

[0012] q i =W q v i ,k j =W k v j ,value j =W v v j

[0013] Among them, q i is the component in the query vector, v i is the component in the value vector, k j is the component in the key vector, W q 、Wk 、W v are weight matrices respectively;

[0014] Utilize the similarity s between the query vector and the key vector of each two visual feature vectors i,j As the attention weight a i,j :

[0015]

[0016] where d k is the channel dimension of the query vector and key vector;

[0017] Calculate the weighted visual feature vector as follows:

[0018]

[0019] The weighted visual feature vector is passed through the layer regularization term and then input into a feedforward network with two fully connected layers to obtain enhanced visual features.

[0020] Step 1-2-2: The prototype-guided attention module includes the construction and update of the prototype vector and the prototype-based cross-modal attention mechanism; first, a learnable prototype vector is constructed, whose dimension is consistent with the dimension of the visual feature vector; in order to make this prototype vector contain all the pedestrian semantic knowledge described in the text, a similarity loss function L is designed between the prototype vector and the text feature vector extracted in step 1-2-1. SM , a fully connected layer is used to calculate the key vector and value vector of the visual feature vector generated by the Transformer encoder layer. The similarity between the prototype vector and the visual feature vector is used as the weight of the cross-modal attention mechanism, and the final visual feature vector is calculated as the semantically significant visual information to separate the pedestrian content from the visual noise. Finally, the prototype-guided attention module will generate the enhanced visual feature V original ;

[0021] Step 1-3: In the pedestrian modality dynamic self-evolution module, text descriptions are used to guide the autonomous evolution of image-level component masks, including visual mask filters and text mask filters;

[0022] The input of the pedestrian modality dynamic self-evolution module is the enhanced visual feature V generated by the prototype-guided attention module in step 1-2-2 original , V original The H dimension is evenly divided into k local visual features, the size of which is expressed as (C, H / k, W). Each local feature is first averagely pooled, and then passed through multiple fully connected layers that do not share parameters to generate a local visual feature V partAt the same time, V original The global feature V is obtained through a fully connected layer global ; Among them, the identity classification loss function is inserted after each local visual feature and global feature to extract the significant features related to the pedestrian identity; thereafter, the text features extracted in step 1-2-1 are guided to the local visual feature V part Dynamically evolve the global visual features to obtain the final visual feature V final , the calculation process is shown in the following formula:

[0023]

[0024] Among them, V global Represents the global visual features, represents the partial level visual features of the i-th horizontal strip area, α i is the weight score corresponding to the i-th visual horizontal strip area guided by the text, i = 1, 2, ..., k;

[0025] Step 2: Use the dataset of the text-based pedestrian retrieval task to train the text-guided dynamic visual information filtering and part-level alignment network, set the training hyperparameters, and complete the training using the Adam optimizer;

[0026] When testing the network, cosine similarity is used as the similarity evaluation metric to calculate the distance between the text feature vector and the final visual feature;

[0027] Step 3: Combine the modules built in the above steps, input the description and the pedestrian image to be retrieved, sort the images according to similarity, and if the query result given is the same as the actual pedestrian ID, the query is considered successful.

[0028] Preferably, C, H, W and k are set to 2048, 24, 8 and 6 respectively.

[0029] Preferably, the learning rate is set to 1.1×10 -4 , the number of iterations is 60, and λ=0.1.

[0030] Preferably, the similarity loss function L SM Expressed as:

[0031]

[0032] Where P represents the prototype vector, represents the text feature vector of the i-th description, N represents the batch size, w and b represent the parameters of the fully connected layer that processes the text feature vector.

[0033] The beneficial effects of the present invention are as follows:

[0034] This paper utilizes text-guided dynamic visual information filtering and a component-level alignment network to effectively dynamically focus on the visual features of pedestrian images based on the text. This not only eliminates visual content irrelevant to the description, but also focuses on visual features that are semantically aligned but inconsistent. This dynamically and adaptively captures visual representations that align with the description, facilitating the calculation of accurate text-image similarity, further improving the accuracy of description-based pedestrian image retrieval tasks. In particular, it achieves a rank-1 accuracy of 64.64% on the international benchmark dataset CUHK-pedestrians, the highest accuracy achieved to date internationally. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A comparative diagram of cross-modal information asymmetry.

[0036] Figure 2 Schematic diagram of the text-guided dynamic visual information filtering and component-level alignment network structure for the implementation method of the present invention.

[0037] Figure 3 This is a diagram illustrating cross-modal component-level information matching in the present invention.

[0038] Figure 4 This is the result of text-based pedestrian retrieval performed by the present invention.

[0039] Figure 5 This is a diagram of a specific embodiment of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described below with reference to the accompanying drawings and examples.

[0041] In order to solve the information asymmetry problem between the image and description, the present invention provides a text-guided dynamic visual information filtering and component-level alignment target retrieval method, the structure of which is as follows: Figure 2 As shown;

[0042] The main structure of the present invention can be divided into three functional modules: a basic image-text feature extraction module, a pedestrian-independent visual content filtering module, and a pedestrian modality dynamic self-evolution module. Specifically, in the basic image-text feature extraction module, a visual convolutional neural network and a language processing model are used to extract basic features of the image and description. Subsequently, in the pedestrian-independent visual content filtering module, a Transformer encoder layer is used to enhance fine-grained pixel-level visual features, and a prototype-guided attention layer is used to filter out pedestrian-independent noise from the enhanced visual features. In the prototype-guided attention layer, a trainable prototype vector is used to learn pedestrian knowledge, guiding the module to suppress pedestrian-independent noise, and using text descriptions to guide the module to extract key features of pedestrian images. Finally, in the pedestrian modality dynamic self-evolution module, text-level query descriptions are used to guide image-level component masks. The mask descriptions are used to describe the emphasis of each pedestrian component, and the text descriptions are used to guide the autonomous evolution of the image-level component masks. This allows the focus to be placed on description-biased image modality features for detailed component-level alignment and matching.

[0043] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0044] Step 1: Build a text-guided dynamic visual information filtering and component-level alignment network. The network framework is shown in the figure below. Figure 2 As shown in the figure, it mainly includes basic image and text feature extraction module, pedestrian-independent visual content filtering module and pedestrian modality dynamic self-evolution module;

[0045] Step 1-1: The basic image-text feature extraction module includes a semantic feature extraction branch and an image feature extraction branch;

[0046] The semantic feature extraction branch uses a BERT network pre-trained on a large amount of text data. For each text input, a special classification tag ([CLS]) is added to the beginning of each description. After it is processed by the BERT model, the hidden state in the final model output is used as the overall feature representation of the description.

[0047] The image feature extraction branch uses ResNet50 pre-trained on ImageNet as the baseline network (Backbone) to extract pedestrian image features. In particular, we remove the last downsampling operation in the image feature extraction network to increase the spatial scale of the feature map. The size of the feature map output by the model is set to (C, H, W), where C and (H, W) represent the channel dimension and feature map size, respectively.

[0048] Step 1-2: The pedestrian-independent visual content filtering module includes a Transformer encoder layer and a prototype-guided attention module;

[0049] Step 1-2-1: The Transformer encoder layer uses the correlation between feature vectors to enhance visual features. It includes a multi-head self-attention mechanism and a feed-forward network. First, the features extracted by the visual feature network are flattened into H*W feature vectors as the input of the encoder. Then, a fully connected layer is used to calculate the query, key, and value vectors of each feature vector. Then, the similarity s between the query and key vectors of each two vectors is used. i,j As the attention weight a i,j , the weighted visual feature vector is calculated according to the weights. Finally, the weighted visual feature vector is input into the feedforward network of two fully connected layers after passing the layer regularization term to obtain the enhanced visual features.

[0050] Step 1-2-2: The prototype-guided attention module is dedicated to using a learnable prototype vector to learn the semantic information of pedestrians in all language descriptions and use it as a guide to focus on the visual information related to pedestrians in the image. It includes the construction and update of the prototype vector and the prototype-based cross-modal attention mechanism. First, a learnable prototype vector is constructed, whose dimension is consistent with the dimension of the visual feature vector. In order to make this prototype vector contain the semantic knowledge of pedestrians in all text descriptions, a similarity loss function L is designed between the prototype vector and all text feature vectors. SM Afterwards, a fully connected layer is used to calculate the key and value vectors of the visual feature vector generated by the encoder. The similarity between the prototype vector and the visual feature vector is used as the weight of the cross-modal attention mechanism, and the final visual feature vector is calculated as the semantically significant visual information to separate the pedestrian content from the visual noise.

[0051] Step 1-3: The pedestrian modality dynamic self-evolution module uses text descriptions to guide the autonomous evolution of image-level component masks, which can focus on describing biased image modality features.

[0052] The input of this module is the enhanced global visual feature V generated by the prototype-guided attention module in 1-2-2 original , V original The H dimension is divided into k local features, the size of which is expressed as (C, H / k, W). Each local feature is first averagely pooled, and then passed through multiple fully connected layers that do not share parameters to generate local visual features V partAt the same time, V original The global feature V is obtained through a fully connected layer global ; Among them, in each local visual feature V part and global feature V global The identity classification loss function is then inserted to extract the salient features related to pedestrian identity. After that, the local level visual features are fused with the output of the text-based filter generator, and the fusion result is calculated with the global visual features to obtain the enhanced visual features V guided by the text description. final , for detailed component-level alignment and matching. Figure 2 Shown is an illustration of cross-modal component-level information matching.

[0053] Step 2: Train the network using the dataset for the text-based person retrieval task, set the training hyperparameters, and complete the training using the Adam optimizer.

[0054] When testing the network, cosine similarity is used as the similarity evaluation metric to calculate the distance between the text feature vector and the final visual feature;

[0055] Step 3: Combine the modules built in the above steps, input the description and the pedestrian image to be retrieved, sort the images according to similarity, and if the query result given is the same as the actual pedestrian ID, the query is considered successful. Figure 4 shown. Specific embodiment:

[0057] The present invention provides a text-guided dynamic visual information filtering and component-level description-pedestrian image retrieval method. This method can use a text-guided model to extract pedestrian-related information, filter out pedestrian-irrelevant noise, dynamically extract aligned visual features, and guide the model to focus on cross-modal local fine-grained feature alignment with biased descriptions, thereby further optimizing the performance of description-based pedestrian image retrieval.

[0058] Figure 5 Two examples of description-based pedestrian image retrieval using the method of the present invention are demonstrated. The prototype-guided attention layer can comprehensively capture semantically significant visual areas in positive and negative image samples. These visual contents are frequently mentioned in pedestrian description datasets. The filter generated by text guidance can dynamically focus on the mentioned visual areas according to each description, while ignoring the areas that are not mentioned, which are the visual contents of the image that are of interest. The method of the invention uses a text-guided model to extract pedestrian-related information, filter out pedestrian-irrelevant noise, and dynamically and adaptively guide the model to focus on cross-modal local fine-grained feature alignment that is biased towards descriptions, providing an important foundation and favorable conditions for cross-modal matching and retrieval.

Claims

1. A text-object retrieval method based on dynamic self-evolution information extraction and alignment, characterized in that: The steps include: Step 1: Construct a text-guided dynamic visual information filtering and component-level alignment network, including a basic image-text feature extraction module, a pedestrian-independent visual content filtering module, and a pedestrian modality dynamic self-evolution module; Step 1-1: The basic image-text feature extraction module includes a semantic feature extraction branch and an image feature extraction branch; The semantic feature extraction branch uses a BERT network pre-trained on text data. For each text input, a special classification token is added to the beginning of each description. After the special classification token is processed by the BERT model, the hidden state in the BERT model's final output is used as the overall feature representation of the description. The image feature extraction branch uses ResNet50 pre-trained on ImageNet as the baseline network backbone to extract pedestrian image features. The last downsampling operation in the image feature extraction branch is removed to increase the spatial scale of the feature map. The size of the feature map output by the model is set to (C, H, W), where C and (H, W) represent the channel dimension and feature map size, respectively. Step 1-2: The pedestrian-independent visual content filtering module includes a Transformer encoder layer and a prototype-guided attention module; Step 1-2-1: The Transformer encoder layer uses the correlation between feature vectors to enhance visual features. It includes a multi-head self-attention mechanism and a feedforward network. First, the features extracted by the Transformer encoder layer are flattened into H*W visual feature vectors as the input of the Transformer encoder layer. Then, the H*W visual feature vectors are passed through a fully connected layer to calculate the query vector, key vector, and value vector of each visual feature vector: q i =W q v i ,k j =W k v j ,value j =W v v j Among them, q i is the component in the query vector, v i is the component in the value vector, k j is the component in the key vector, W q 、W k 、W v are weight matrices respectively; Utilize the similarity s between the query vector and the key vector of each two visual feature vectors i，j As the attention weight a i，j : where d k is the channel dimension of the query vector and key vector; Calculate the weighted visual feature vector as follows: The weighted visual feature vector is passed through the layer regularization term and then input into a feedforward network with two fully connected layers to obtain enhanced visual features. Step 1-2-2: The prototype-guided attention module includes the construction and update of the prototype vector and the prototype-based cross-modal attention mechanism; first, a learnable prototype vector is constructed, whose dimension is consistent with the dimension of the visual feature vector; in order to make this prototype vector contain all the pedestrian semantic knowledge described in the text, a similarity loss function L is designed between the prototype vector and the text feature vector extracted in step 1-2-1. SM , a fully connected layer is used to calculate the key vector and value vector of the visual feature vector generated by the Transformer encoder layer. The similarity between the prototype vector and the visual feature vector is used as the weight of the cross-modal attention mechanism, and the final visual feature vector is calculated as the semantically significant visual information to separate the pedestrian content from the visual noise. Finally, the prototype-guided attention module will generate the enhanced visual feature V original ; Step 1-3: In the pedestrian modality dynamic self-evolution module, text descriptions are used to guide the autonomous evolution of image-level component masks, including visual mask filters and text mask filters; The input of the pedestrian modality dynamic self-evolution module is the enhanced visual feature V generated by the prototype-guided attention module in step 1-2-2 original , V original The H dimension is evenly divided into k local visual features, the size of which is expressed as (C, H / k, W). Each local feature is first averagely pooled, and then passed through multiple fully connected layers that do not share parameters to generate a local visual feature V part At the same time, V original The global feature V is obtained through a fully connected layer global ; Among them, the identity classification loss function is inserted after each local visual feature and global feature to extract the significant features related to the pedestrian identity; thereafter, the text features extracted in step 1-2-1 are guided to the local visual feature V part Dynamically evolve the global visual features to obtain the final visual feature V final , the calculation process is shown in the following formula: Among them, V global Represents the global visual features, represents the partial level visual features of the i-th horizontal strip area, α i is the weight score corresponding to the i-th visual horizontal strip area guided by the text, i = 1, 2, ..., k; Step 2: Use the dataset of the text-based pedestrian retrieval task to train the text-guided dynamic visual information filtering and part-level alignment network, set the training hyperparameters, and complete the training using the Adam optimizer; When testing the network, cosine similarity is used as the similarity evaluation metric to calculate the distance between the text feature vector and the final visual feature; Step 3: Combine the modules built in the above steps, input the description and the pedestrian image to be retrieved, sort the images according to similarity, and if the query result given is the same as the actual pedestrian ID, the query is considered successful.

2. A text-object retrieval method based on dynamic self-evolution information extraction and alignment according to claim 1, characterized in that: The C, H, W and k are set to 2048, 24, 8 and 6, respectively.

3. The text-object retrieval method based on dynamic self-evolution information extraction and alignment according to claim 1, characterized in that: The learning rate during training is set to 1.1×10 -4 , the number of iterations is 60, and λ=0.

1.

4. The text-object retrieval method based on dynamic self-evolution information extraction and alignment according to claim 1, characterized in that: The similarity loss function L SM Expressed as: Where P represents the prototype vector, represents the text feature vector of the i-th description, N represents the batch size, w and b represent the parameters of the fully connected layer that processes the text feature vector.

Images