Visible light infrared pedestrian re-identification method based on text assistance

Abstract

The invention provides a visible light infrared pedestrian re-identification method based on text assistance. The method comprises the following steps: acquiring an infrared pedestrian image to be retrieved and a visible light pedestrian image library, and labeling rough text description on the infrared pedestrian image; preprocessing the infrared pedestrian image and the visible light pedestrian image; constructing an asymmetric double-branch cross-modal feature extractor, introducing text description as auxiliary information, and extracting features of a to-be-retrieved pedestrian image and all pedestrian images in an image library; calculating the cosine similarity of the two, sorting the similarities from high to low, and generating an initial retrieval sequence; calculating the similarity between the pedestrian images in the image library, and generating a neighbor set according to the similarity; and calculating neighbor similarity, and reordering the initial retrieval sequence according to the neighbor similarity. According to the method, text description is introduced as cross-modal retrieval auxiliary information, and a cross-modal retrieval reordering algorithm is designed, so that the problem of cross-modal pedestrian feature alignment can be well solved, and a pedestrian re-identification result which is more accurate than that of an existing method is obtained.

Description

Technical Field

[0001] The present invention belongs to the field of computer software technology and relates to image recognition technology, in particular to a text-assisted visible light infrared pedestrian re-identification method. Background Art

[0002] In recent years, with the development of deep learning technology and the increase in data volume, pedestrian re-identification technology based on deep learning has made great progress. Among them, in order to meet the needs of 24-hour monitoring, the visible light infrared pedestrian re-identification task aims to retrieve visible light pedestrian images based on infrared pedestrian images, which poses a challenge to the algorithm's cross-modal feature extraction capabilities. Most of the work extracts modal consistency features through shared feature learning and maps pedestrian images from different modalities to a unified feature space. However, this will cause modal-specific information in visible and infrared modalities to be discarded, including some discriminative detail features. Other works use feature compensation learning to supplement the missing modal-specific information from one modality to another, but some information is difficult to compensate through the model, such as the missing color information in the infrared modality.

[0003] The task of pedestrian re-identification from text images is one of the current research hotspots. It can retrieve the visible light pedestrian image corresponding to the target through the eyewitness's language description, so it has a wide range of application scenarios. The success of this task proves that language description can provide discriminative features for pedestrian retrieval. Based on the above analysis, the present invention proposes a text-assisted visible light infrared pedestrian re-identification framework, that is, using a rough language description (such as "a man wearing a dark blue short-sleeved shirt and light blue jeans") to assist infrared pedestrian images in retrieving visible light pedestrian images, and proposes a cross-modal k-nearest neighbor re-ranking algorithm to further improve the retrieval accuracy. Summary of the invention

[0004] The present invention provides a text-assisted visible light infrared pedestrian re-identification method, characterized in that the method comprises the following steps:

[0005] Step 1: Obtain a retrieval library of infrared pedestrian images and visible light pedestrian images of pedestrians to be retrieved, and annotate a rough text description for the infrared pedestrian images;

[0006] Step 2, preprocessing the infrared pedestrian image and the visible light pedestrian image;

[0007] Step 3: construct an asymmetric dual-branch cross-modal feature extractor to extract visible light pedestrian image features and extract infrared pedestrian image features based on text descriptions; introduce texture feature loss, color feature loss, and joint feature loss for training in the training phase;

[0008] Step 4, calculate the cosine similarity between pedestrian images and generate a retrieval sequence;

[0009] Step 5: Calculate the nearest neighbor similarity between pedestrian images, reorder the retrieval sequence, and output the results.

[0010] Specifically, in step 1, given an infrared pedestrian image to be retrieved, a text description is annotated based on a visible light pedestrian image with the same ID, and the annotation template is:

[0011] "A person dressed in _, _ and _", such as "a person dressed in a light blue short-sleeved shirt, dark black pants and a yellow backpack".

[0012] Finally, we get a rough text description set:

[0013] Where N i is the number of infrared pedestrian images and corresponding text descriptions.

[0014] Specifically, in step 2, the single-channel infrared pedestrian image is first converted into a three-channel image, and then the infrared and visible light pedestrian images are scaled to obtain a three-channel image with a height of 384 pixels and a width of 144 pixels.

[0015] During the training phase, additional data enhancement processing such as random grayscale, random cropping, random horizontal flipping, and random erasing is required. Finally, the processed pedestrian visible light image set is obtained. And a collection of infrared pedestrian images Where N v and N i are the number of visible light pedestrian images and infrared pedestrian images respectively. The corresponding pedestrian ID is and

[0016] Specifically, in step 3, a batch of triplet pedestrian images is input:

[0017] Where N B is the batch size.

[0018] Using two texture encoders and To extract texture features:

[0019] Where *∈{v,i}. The texture encoder uses ResNet-50 as the backbone network, and the stride of the last module is set to 1. The first convolution modules of the two texture encoders are independent of each other, and the parameters of other convolution modules are shared.

[0020] Use a color coder Extract color features from visible light pedestrian images:

[0021] Its network structure and Same but independent parameters.

[0022] Using a RoBERTa model as a color encoder Extract color features from infrared pedestrian images:

[0023]

[0024] Using two joint relationship modules *∈{v,i} dynamically aggregates and decouples color features and texture features Output joint features:

[0025]

[0026] During the training phase, three sets of encoders and Use a Kullback-Leibler divergence loss and two ReID losses and To optimize, the KL divergence loss uses the color word overlap to obtain the regular probability as the training truth value. In the inference phase, the triple features are concatenated along the channel dimension To get the final features:

[0027] f, *∈{v,i}.

[0028] Specifically, in step 4, given N q query pedestrian image features and N g The pedestrian image features of the image library are first calculated to obtain the initial distance matrix It is expressed as:

[0029]

[0030] in is the distance between all query pedestrian images, and To query the distance between pedestrian images and pedestrian images in the image library, is the distance between pedestrian images in all image libraries.

[0031] Based on the distance matrix D ori Sort the pedestrian images by distance and obtain the retrieval sequence.

[0032] Specifically, in step 5, based on the matrix D ori , get the query pedestrian image q i K The mutual neighbor set is:

[0033] R(q i , k1)={g j |(g j ∈N(q i , k1))∩(q i ∈N(g j , k1))},

[0034] Where N(q i , k1) is based on the pedestrian image distance D ori The retrieved pedestrian image q i Then use the cross-modal nearest pedestrian image set R q→g (q i , k1) for R(q i , k1) to expand:

[0035] R ext (q i , k1)=R(q i , k1)∪R q→g (q i , k1).

[0036] Then the expanded mutual nearest neighbor set R ext (q i , k1) is transformed into the nearest neighbor feature vector And further optimize it using the local retrieval expansion algorithm to obtain:

[0037]

[0038] in Explicitly aggregate (k2-k3) intra-modal nearest pedestrian images and k3 cross-modal nearest pedestrian images. Finally, calculate the Jaccard distance of the optimized nearest neighbor features between pedestrian images to re-rank the retrieval results:

[0039]

[0040] in is the feature vector The nth element of . BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0042] Figure 1 It is a flow chart of the text-assisted visible light infrared pedestrian re-identification method of the present invention;

[0043] Figure 2 This is a diagram of the inference framework for training the cross-modal feature extractor in the present invention.

[0044] Figure 3 This is a flow chart of the cross-modal k-nearest-neighbor reordering algorithm in the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Visible light infrared pedestrian re-identification can be used in 24-hour monitoring systems and is of great significance to the fields of intelligent security, smart cities, etc. At present, the solutions based on deep learning mainly include feature sharing learning and feature compensation learning, but these methods cannot make up for the missing color information in the infrared modality. In addition, the degradation problem of the re-ranking algorithm in cross-modal scenarios is often ignored, which constrains the accuracy of the pedestrian re-identification retrieval sequence. To this end, the present invention proposes a text-assisted visible light infrared pedestrian re-identification method. First, an asymmetric dual-branch cross-modal feature extractor is constructed to extract pedestrian image features in visible light and infrared modalities based on rough text descriptions. Then, a cross-modal k-nearest neighbor re-ranking algorithm is designed to further optimize the retrieval sequence.

[0047] like Figure 1 As shown, the specific steps of the present invention are as follows:

[0048] S101: Obtain an infrared pedestrian image and a visible light pedestrian image retrieval library of pedestrians to be retrieved, and annotate a rough text description for the infrared pedestrian image.

[0049] Specifically, given an infrared pedestrian image to be retrieved, a text description is annotated based on the visible light pedestrian image with the same ID, and the annotation template is "a person wearing _, _ and _", such as "a person wearing light blue short-sleeved shirt, dark black pants and a yellow backpack". Finally, a rough text description set is obtained. Where N i is the number of infrared pedestrian images and corresponding text descriptions.

[0050] S102: Preprocessing the infrared pedestrian image and the visible light pedestrian image.

[0051] Specifically, the single-channel infrared pedestrian image is first converted into a 3-channel image, and then the infrared and visible light pedestrian images are scaled to obtain a 3-channel image with a height of 384 pixels and a width of 144 pixels. During the training phase, additional data enhancement processing such as random grayscale, random cropping, random horizontal flipping, and random erasing is required. Finally, the processed pedestrian visible light pedestrian image set is obtained. and pedestrian infrared image collection Where N v and N i are the number of visible light pedestrian images and infrared pedestrian images respectively. The corresponding pedestrian ID is and

[0052] S103: Construct an asymmetric dual-branch cross-modal feature extractor to extract visible light pedestrian image features, and extract infrared pedestrian image features based on text descriptions; introduce texture feature loss, color feature loss and joint feature loss for training during the training phase.

[0053] Specifically, input a batch of triplet pedestrian images Where N B is the batch size. Two texture encoders are used and To extract texture features Where *∈{v,i}, the texture encoder uses ResNet-50 as the backbone network, and the stride of the last module is set to 1. The first convolution modules of the two texture encoders are independent of each other, and the parameters of other convolution modules are shared. Then a color encoder is used Extracting color features from visible light pedestrian images Its network structure and The same but the parameters are independent of each other. In addition, a RoBERTa model is used as the color encoder Extracting color features from infrared pedestrian images Finally, use two joint relationship modules *∈{v,i} dynamically aggregates and decouples color features and texture features Output joint features During the training phase, three sets of encoders and Use a Kullback-Leibler divergence loss and two ReID losses and To optimize, the KL divergence loss uses the color word overlap to obtain the regular probability as the training truth value. In the inference phase, the triple features are concatenated along the channel dimension To get the final feature f* ,*∈{v,i}.

[0054] S104: Calculate the cosine similarity between pedestrian images and generate a retrieval sequence.

[0055] Specifically, given N q query pedestrian image features and N g The pedestrian image features of the image library are first calculated to obtain the initial distance matrix It is expressed as: in is the distance between all query pedestrian images, and To query the distance between pedestrian images and pedestrian images in the image library, is the distance between pedestrian images in all image libraries. Based on the distance matrix D ori Sort the pedestrian images by distance and obtain the retrieval sequence.

[0056] S105: Calculate the nearest neighbor similarity between pedestrian images, reorder the retrieval sequence, and output the result.

[0057] Specifically, based on the matrix D ori , get the query pedestrian image q i The set of k nearest neighbors is R(q i , k1)={g j |(g j ∈N(q i , k1))∩(q i ∈N(g j , k1))}, where N(q i , k1) is based on the pedestrian image distance D ori The retrieved pedestrian image q i Then use the cross-modal nearest pedestrian image set R q→g (q i , k1) for R(q i , k1) to expand: R ext (q i , k1)=R(q i , k1)∪R q→g (q i , k1). Then the expanded mutual nearest neighbor set R ext (q i , k1) is transformed into the nearest neighbor feature vector And further optimize it using the local search expansion algorithm in Explicitly aggregate (k2-k3) intra-modal nearest pedestrian images and k3 cross-modal nearest pedestrian images. Finally, calculate the Jaccard distance of the optimized nearest neighbor features between pedestrian images to re-rank the retrieval results. in is the feature vector The nth element of .

[0058] like Figure 2 As shown, the cross-modal feature extractor training reasoning framework in the present invention specifically includes:

[0059] S201: Build 4 feature extractors, including text feature (color feature) extractor Infrared feature (texture feature) extractor Visible light color feature extractor And visible light texture feature extractor The text feature extractor uses the pre-trained RoBERTa as the backbone network; other feature extractors use the pre-trained ResNet-50 as the backbone network, and the infrared feature extractor shares the deep convolution module parameters with the visible light texture feature extractor.

[0060] S202: Building an infrared joint feature extractor and visible light joint feature extractor The two have the same structure, which includes 3 relationship modules and for Color Features and texture features First, two mapping layers are used for encoding, and then the mapped texture features are After the average pooling layer and the sigmoid layer, the channel modulation weight is obtained:

[0061]

[0062] Then use Color features after modulation mapping And introduce residual connection:

[0063] Where × is element-wise multiplication.

[0064] for Color Features and texture features Similarly, first do the mapping encoding to get and and will As a dynamic check Perform depth-wise separable convolution and introduce residual connections:

[0065] Where DWConv(·;·) is a depthwise separable convolution.

[0066] In obtaining modulation characteristics and After that, use Perform feature aggregation. First, Make spatial dimension copies and compare with Concatenate in the channel dimension Then it passes through a 2D convolution layer, a residual connection and an average pooling layer to obtain the joint encoding features:

[0067]

[0068] S203: Inference stage, concatenating features along the channel dimension and Get infrared joint features and visible light joint features

[0069] S204: In the training phase, ReID loss (cross entropy loss and triplet loss) is used as texture feature loss and joint feature loss to supervise texture features and joint features And use KL divergence loss as color feature loss to supervise color features

[0070] where p n，m is a feature pair {f n′ f m}The probability of matching, q n，m is the matching truth value, ∈=1e -8 is the compensation term. Considering q n，m Supervision errors may be introduced, and the overlap of the color word set corresponding to the pedestrian image needs to be calculated:

[0071] In the selling, / / is the basis of the set.

[0072] Then the IoU n，m Do batch normalization as regularization probability Replace the original true value.

[0073] like Figure 3 As shown, it is a flow chart of the cross-modal k-nearest neighbor reordering algorithm in the present invention, which specifically includes:

[0074] S301: Given N q query pedestrian image features and Ng The pedestrian image features of the image library are first calculated to obtain the initial distance matrix

[0075] S302: Based on the distance matrix D ori Get the k nearest neighbor set of each pedestrian image, such as N(q i , k1) is based on the pedestrian image distance D ori The retrieved pedestrian image q i The k1 nearest neighbor pedestrian image collection.

[0076] S303: Calculate k mutual nearest neighbor sets based on k nearest neighbor sets: R(q i , k1)={g j |(g j ∈N(q i , k1))∩(q i ∈N(g j , k1))}.

[0077] S304: Based on the cross-modal k nearest neighbor set, cross-modal expansion is performed on the k mutual nearest neighbor set: R ext (q i , k1)=R(q i , k1)∪R q→g (q i , k1), transform it into a mutual neighbor feature necklace V i .

[0078] S305: Calculate modality-aware neighbor features based on intra-modality and inter-modality neighbors: in Aggregate (k2-k3) intra-modal nearest neighbor pedestrian images and k3 cross-modal nearest neighbor pedestrian images.

[0079] S306: Calculate the Jaccard distance of neighbor features between pedestrian images for reordering the retrieval sequence: in is the feature vector The nth element of .

[0080] The text-assisted visible light infrared pedestrian re-identification method of the present invention can make full use of the discriminative information in the text description and provide retrieval context information based on cross-modal neighbor features, thereby greatly improving the retrieval accuracy of existing methods.

Claims

1. A text-assisted visible light infrared pedestrian re-identification method, characterized in that: The method comprises the following steps: Step 1: obtain the infrared pedestrian image to be retrieved and the visible light pedestrian image library, and annotate the infrared pedestrian image with a rough text description; Step 2, preprocessing the infrared pedestrian image and the visible light pedestrian image; Step 3: construct an asymmetric dual-branch cross-modal feature extractor to extract visible light pedestrian image features and extract infrared pedestrian image features based on text descriptions; introduce texture feature loss, color feature loss, and joint feature loss for training in the training phase; Step 4, calculating the cosine similarity between the infrared pedestrian image and all visible light pedestrian images in the image library to generate an initial retrieval sequence; Step 5: Calculate the similarity between visible light pedestrian images and generate a neighbor set. Reorder the initial search sequence according to the similarity of the neighbors and output the result.

2. The text-assisted visible light infrared pedestrian re-identification method according to claim 1, characterized in that: The infrared pedestrian image is labeled with a rough text description. Specifically, given a certain infrared pedestrian image to be retrieved, a text description is labeled based on the visible light pedestrian image with the same ID. The annotation template is "a person wearing _, _ and _", such as "a person wearing a light blue short-sleeved shirt, dark black pants and a yellow backpack". Finally, a rough text description set is obtained. Where N i is the number of infrared pedestrian images and corresponding text descriptions.

3. The text-assisted visible light infrared pedestrian re-identification method according to claim 1, characterized in that: Preprocess the infrared pedestrian images and visible light pedestrian images. Specifically, first convert the single-channel infrared pedestrian images into 3-channel images, then scale the infrared and visible light pedestrian images to obtain 3-channel images with a height of 384 pixels and a width of 144 pixels. In the training phase, additional data enhancement processing such as random grayscale, random cropping, random horizontal flipping, and random erasing is required. Finally, the processed visible light pedestrian image set is obtained. and infrared pedestrian image collection Where N v and N i are the number of visible light pedestrian images and infrared pedestrian images, and the corresponding pedestrian IDs are and 4. The text-assisted visible light infrared pedestrian re-identification method according to claim 1, characterized in that: Construct an asymmetric dual-branch cross-modal feature extractor to extract pedestrian image features. Specifically, input a batch of triples of pedestrian images pedestrian images Where N B is the batch size. Two texture encoders are used and To extract texture features Where *∈{v,i}, the texture encoder uses ResNet-50 as the backbone network, and the stride of the last module is set to 1. The first convolution modules of the two texture encoders are independent of each other, and the parameters of other convolution modules are shared. Then a color encoder is used Extracting color features from visible light pedestrian images Its network structure and The same but the parameters are independent of each other. In addition, a RoBERTa model is used as the color encoder Extracting color features from infrared pedestrian images Finally, use two joint relationship modules Dynamically aggregate and decouple color features and texture features Output joint features During the training phase, three sets of encoders and Use a Kullback-Leibler divergence loss and two ReID losses and To optimize, the KL divergence loss uses the color word overlap to obtain the regular probability as the training truth value. In the inference phase, the triple features are concatenated along the channel dimension To get the final feature f * ,*∈{v,i}.

5. The text-assisted visible light infrared pedestrian re-identification method according to claim 1, characterized in that: Get the pedestrian image retrieval sequence and reorder it based on the neighbor similarity. Specifically: given N q query image features and N g Image features of the image library, first calculate the pairwise simple cosine distance to get the initial distance matrix It is expressed as: in is the distance between all query images, and To find the distance between the query image and the image library, is the distance between all pedestrian images in the image library. Based on this, query pedestrian image q i The set of k nearest neighbors is: R(q i ,k1)={g j |(g j ∈N(q i ,k1))∩(q i ∈N(q j ,k1))}, Where N(q i ,k1) is based on the pedestrian image distance D ori The retrieved pedestrian image q i Then use the cross-modal nearest pedestrian image set R q→g (q i ,k1) for R(q i ,k1) to expand: R ext (q i ,k1)=R(q i ,k1)∪R q→g (q i ,k1) Then the expanded mutual nearest neighbor set R ext (q i ,k1) is transformed into the nearest neighbor feature vector And further optimize it using the local retrieval expansion algorithm: in Explicitly aggregate (k2-k3) intra-modal nearest pedestrian images and k3 cross-modal nearest pedestrian images. Finally, calculate the Jaccard distance of the optimized nearest neighbor features between pedestrian images to re-rank the retrieval results: in is the feature vector The nth element of .

Images