Cross-modal pedestrian re-identification method based on self-supervised learning and pre-training model

Abstract

The application discloses a cross-modal pedestrian re-identification method based on self-supervised learning and a pre-training model, and steps of the method comprise the following steps: 1, a large-scale unlabeled pedestrian dataset is acquired, and the collected pedestrian dataset is screened by using a cross-modal catastrophic forgetting score; 2, positive and negative samples are generated from the screened pedestrian data by using a channel combination module, and a self-supervised contrast learning network is constructed to extract negative sample and positive sample features respectively; 3, the feature distance between the positive sample and the negative sample is constrained by using a bidirectional contrast loss function, so that a pre-training model is trained; and 4, after the trained encoder is migrated to other cross-modal pedestrian re-identification models for fine-tuning, a fine-tuned cross-modal pedestrian re-identification model is obtained. According to the application, the positive and negative samples are separated from the same image by using the channel combination method, and self-supervised training is completed, so that the accuracy of infrared and visible light pedestrian re-identification can be improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to a cross-modal pedestrian re-identification method based on self-supervised learning and a pre-training model. BACKGROUND

[0002] In recent years, with the continuous popularization of visible light-near infrared dual-mode cameras, conditions are provided for constructing a 24-hour all-time pedestrian re-identification system. In the face of the limitation that visible light cameras are difficult to shoot effective information in insufficient light, infrared cameras make up for their shortcomings and can still capture the appearance features of pedestrians at night or in poor lighting conditions, and a new type of cross-modal re-identification task, i.e. visible light-near infrared cross-modal pedestrian re-identification, is derived. Cross-modal pedestrian re-identification is generally divided into two stages: a pre-training stage and a fine-tuning stage.

[0003] However, in the pre-training stage, the existing cross-modal pedestrian re-identification technology only uses a visible light image set as pre-training data. In the pre-training process, a single visible light image will cause the model to only focus on pedestrian images with sufficient light. On the other hand, the current pre-training method is to classify objects with labels, but there is a big gap between pedestrian recognition and object classification tasks. This pre-training method cannot help the model well acquire the features of near-infrared pedestrian images, resulting in unsatisfactory transfer identification results. SUMMARY

[0004] The present application is proposed to overcome the problems in the prior art, and provides a cross-modal pedestrian re-identification method based on self-supervised learning and a pre-training model, so as to realize the generalization ability of the model to various samples, and at the same time consider the domain gap between the training samples and the downstream task samples, so as to better extract the texture features of visible light images and near-infrared images, and thus improve the accuracy of cross-modal pedestrian re-identification.

[0005] The present application adopts the following technical solutions to solve the technical problems:

[0006] The cross-modal pedestrian re-identification method based on self-supervised learning and a pre-training model of the present application has the following steps:

[0007] Step 1, acquisition and preprocessing of pre-training data;

[0008] Step 1.1, M visible light pedestrian images are collected from a video using a YOLO algorithm and normalized for preprocessing, to obtain a visible light data set wherein, represents the mth visible light pedestrian image;

[0009] N near-infrared pedestrian images of pedestrians are collected using a near-infrared camera and normalized for preprocessing, to obtain a near-infrared data set Dt = (X t , Y t ), wherein X t represents a near-infrared image dataset, and represents the nth near-infrared pedestrian image, Y t is an identity label dataset of X t ;

[0010] Step 1.2, converting the visible light dataset X s into a third modality closer to the near-infrared using a channel enhancement method to obtain a visible light enhanced dataset wherein, represents the mth visible light enhanced pedestrian image;

[0011] Step 1.3, calculating the cross-modal catastrophic forgetting score between the mth visible light enhanced pedestrian image and the mth near-infrared pedestrian image , for measuring the feature distance between the mth visible light enhanced pedestrian image and the near-infrared pedestrian image ;

[0012]

[0013] In formula (1), θ s represents an encoder pre-trained unsupervised on the visible light dataset X s , and θ t represents an encoder trained supervised on the near-infrared image dataset X t ;

[0014] Step 1.4, obtaining the feature distance between all visible light enhanced pedestrian images and respective near-infrared pedestrian images according to the process of step 1.3, and screening out M' visible light pedestrian images closest in distance for constructing an unlabeled pre-training dataset wherein, represents the mth visible light pedestrian image screened out;

[0015] Step 2, constructing a channel combination module for generating positive sample images and a contrast learning network for extracting image features;

[0016] Step 2.1, the channel combination module comprises a channel separation and duplication unit, a channel erasing unit, and a channel superposition unit; and inputting the channel combination module for processing to obtain a set of positive sample image pairs of ;

[0017] Step 2.2: The contrastive learning network consists of two encoders E Q E K It consists of a linear dimensionality reduction module M and a memory storage module;

[0018] Will The combined image of any positive sample pair in the set is regarded as an anchor point and input into encoder E. Q In the process, the other combined image in the positive sample pair is considered a positive sample, and the pre-training dataset X* is divided into... Other images are treated as a negative sample set and input together with it into encoder E. K The data is processed to obtain preliminary features of the anchor points, positive samples, and negative sample sets.

[0019] The linear dimensionality reduction module M consists of an MLP layer containing two fully connected layers and a ReLU function, and is used to perform linear dimensionality reduction on the preliminary features of the anchor points, positive samples, and negative sample sets to obtain the features of the anchor points, positive samples, and negative sample sets after dimensionality reduction.

[0020] The memory repository module is a queue structure used to dynamically store the features of the negative sample set after dimensionality reduction;

[0021] Step 3: Use equation (2) to establish the double contrastive loss function L for backpropagation:

[0022]

[0023] In equation (2), α and β are proportionality coefficients, and α + β = 1; q is the characteristic of the anchor point after dimensionality reduction; k + k represents the features of the positive samples after dimensionality reduction. i - Let be any i-th feature in the feature set of the negative samples after dimensionality reduction; τ is the temperature coefficient; K represents the number of features in the negative sample set after dimensionality reduction;

[0024] Step 4: Train the contrastive learning network using gradient descent and calculate the dual contrastive loss function L to update the network parameters. When the value of the loss function L is less than the set threshold or the number of training iterations reaches the set requirement, the training is completed and the optimal pre-trained model is obtained.

[0025] Step 5: Transfer the pre-trained model to the cross-modal person re-identification model, and apply it to the near-infrared dataset D. t =(X t ,Y t) input to the migrated cross-modal pedestrian re-identification model for training to fine-tune the parameters of the cross-modal pedestrian re-identification model, so as to obtain a fine-tuned cross-modal pedestrian re-identification model for cross-modal pedestrian re-identification on the to-be-predicted near-infrared image data.

[0026] The cross-modal pedestrian re-identification method based on self-supervised learning has the characteristics that the step 2.1 specifically includes the following processes:

[0027] The channel separation and replication unit separates and replicates the R, G and B channels of any one visible light pedestrian image to obtain a plurality of replicated images of the visible light pedestrian image on the three channels.

[0028] The channel erasing unit uses a channel erasing function to erase part of the pixels of the plurality of replicated images of the visible light pedestrian image on the three channels.

[0029] The channel superposition unit randomly selects three replicated images from the plurality of replicated images of the visible light pedestrian image on the three channels after erasing to combine to obtain a combined image, thereby obtaining a plurality of combined images of the visible light pedestrian image, and any two combined images form a positive sample image pair.

[0030] The electronic device comprises a memory and a processor, and the memory is used to store a program supporting the processor to execute the cross-modal pedestrian re-identification method, and the processor is configured to execute the program stored in the memory.

[0031] The computer readable storage medium stores a computer program, and when the computer program is run by a processor, the steps of the cross-modal pedestrian re-identification method are executed.

[0032] Compared with the prior art, the present application has the following advantages:

[0033] 1. The present application is based on cross-modal catastrophic forgetting score, which measures the feature similarity of pre-training data and target task data under the target domain model, reduces the domain gap between RGB single-modal data set and RGB-IR double-modal data set, realizes efficient screening of large amount of data, avoids the long training time of traditional pre-training method facing large amount of low-quality pre-training data, and improves the efficiency of model training facing large data.

[0034] 2. The present application designs a channel superposition and erasing method, which can better utilize the color information correlation between channels to better generate samples and increase the diversity of pre-training samples, thereby improving the robustness of the model facing complex samples.

[0035] 3. The present application generates positive and negative samples with discriminability by constructing a self-supervised contrastive learning network and its joint application with the channel superposition erasure strategy, so that the model completes training under no label condition and obtains a good pre-training model, finally improving the recognition accuracy of the downstream task. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 The structural diagram of the cross-modal pedestrian re-identification pre-training network of the present application;

[0037] Figure 2 The flowchart of the present application. DETAILED DESCRIPTION

[0038] In this embodiment, a cross-modal pedestrian re-identification pre-training method based on self-supervised learning is to first construct a large-scale unlabeled visible light pedestrian dataset, then filter the dataset using a cross-modal catastrophic forgetting index, and finally input the filtered dataset into a constructed self-supervised contrastive learning network to generate positive and negative samples for training by combining a channel superposition erasure method, as shown in Figure 1 The specific steps are as follows:

[0039] Step 1, acquisition and preprocessing of pre-training data;

[0040] Step 1.1, collect M visible light pedestrian images from videos using YOLO algorithm and perform normalization preprocessing to obtain a visible light dataset wherein, represents the mth visible light pedestrian image. In this embodiment, the LUPerson dataset is used as the visible light dataset to be filtered The data of this dataset is obtained by intercepting videos from videos retrieved from video websites using YOLO algorithm, including 4180243 unlabeled pedestrian visible light images of 46260 scenes of more than 200000 pedestrian identities;

[0041] Collect N near-infrared pedestrian images of pedestrians using a near-infrared camera and perform normalization preprocessing to obtain a near-infrared dataset D t =(X t ,Y t ), wherein X t represents the near-infrared image dataset, and represents the nth near-infrared pedestrian image, Y t is the identity label dataset of X t . In this embodiment, SYSY-MM01 is selected as the near-infrared dataset The dataset is captured by 4 visible light cameras and 2 near-infrared cameras. SYSU-MM01 contains 491 pedestrian identities in total, each of which is captured by at least two different cameras, and there are 287628 visible light RGB images and 15792 near-infrared images in total. 395 pedestrian identity images are selected as the training set, and the remaining 96 pedestrian identity images are selected as the test set. 301 visible light images are randomly selected as the retrieval images, and 3803 infrared images are selected as the searched images;

[0042] Step 1.2, using a channel enhancement method to convert the visible light dataset X s into a third modality closer to the near-infrared to obtain a visible light enhanced dataset wherein, represents the m-th visible light enhanced pedestrian image;

[0043] In this embodiment, LUPerson, i.e. is input into the channel enhancement module to obtain third modality data corresponding to the original data one by one The purpose is to reduce the domain gap between the visible light image and the near-infrared image with the help of the third modality;

[0044] Step 1.3, calculating the cross-modal catastrophic forgetting score between the m-th visible light enhanced pedestrian image and the m-th near-infrared pedestrian image , which is used to measure the feature distance between the m-th visible light enhanced pedestrian image and the near-infrared pedestrian image

[0045]

[0046] In formula (1), θ s represents an encoder pre-trained unsupervised on the visible light dataset X s , and θ t represents an encoder trained supervised on the near-infrared image dataset X t ; in this embodiment, θ s is a ResNet50 encoder trained on the MoCo contrast learning network using unfiltered LUPerson, and θ t is a ResNet50 encoder trained on SYSU-MM01 using the AGW model, is an enhanced image obtained by the channel enhancement strategy CAJ in LUPerson, is the original image of LUPerson;

[0047] Step 1.4, as Figure 1 ​As shown, the feature distance between all visible light enhanced pedestrian images and each near-infrared pedestrian image is obtained according to the process of step 1.3, and the M' closest visible light pedestrian images are selected from the feature distance for constructing the unlabeled pre-training data set wherein, represents the m'th visible light pedestrian image selected; the cross-modal catastrophic forgetting score The feature distance is the feature gap between the LUPerson data and the SYSU-MM01 data set, and the selection process selects data with high similarity to the near-infrared image features as pre-training data; in this embodiment, all 4 million visible light data are input into the formula after channel enhancement to calculate the score, and the top 1 million images with the highest feature similarity are selected as the pre-training data set

[0048] Step 2, construct a channel combination module and a contrast learning network, the channel combination module is used to generate positive sample images, and the contrast learning network is used to extract image features;

[0049] Step 2.1, the channel combination module includes: a channel separation and duplication unit, a channel erasing unit, and a channel superposition unit; and Input into the channel combination module for processing to obtain a set of positive sample image pairs .

[0050] The channel separation and duplication unit separates and duplicates the R, G, and B channels of any visible light pedestrian image to obtain a plurality of duplicate images of the visible light pedestrian image on the three channels.

[0051] The channel erasing unit uses a channel erasing function to erase part of the pixels of the plurality of duplicate images of the visible light pedestrian image on the three channels.

[0052] The channel superposition unit randomly selects three duplicate images from the plurality of duplicate images of the visible light pedestrian image on the three channels after erasing to combine them to obtain a combined image, thereby obtaining a plurality of combined images of the visible light pedestrian image, and any two combined images form a positive sample image pair.

[0053] After inputting to the channel combination module, the image first passes through a channel separation and copying unit to separate and copy the R, G, and B channels, resulting in nine single-channel images. Each of the R, G, and B channels has three images. Then, a channel erasing unit randomly erases each of the single-channel images obtained in the previous unit. Finally, the erased single-channel images are input to a channel overlay unit for combination. The combination method involves randomly selecting three single-channel images from the nine images to create a single three-channel image; the same image is repeated twice to obtain two three-channel images, thus forming a positive sample image pair. In this embodiment, the pre-training dataset is first... Visible light images The input channel separation copy unit produces nine single-channel color images. Then, each single-channel image is input into a random erasure unit to obtain nine erased single-channel images. These nine images are then input into a channel combining unit, and the average values ​​of the R, G, and B channels obtained from the large-scale dataset LUPerson are selected as the erase value for each image, resulting in a three-channel image. Repeat the above process twice to obtain positive sample pairs. and Meanwhile, other images in the pre-training dataset will be used as negative samples during training, as follows: The combination of channel erasure and channel swapping strategies generates a large number of multimodal images with diverse styles. During processing, random selection and overlay of erasure regions are performed, resulting in color-level overlay erasure. Simultaneously, hard erasure of texture is preserved when the R, G, and B channel erasure regions overlap. As a lightweight data augmentation method, the channel overlay erasure strategy further enhances the color transformation of images, helping the network train more suitable parameters to extract modality-invariant features in visible and near-infrared pedestrian re-identification tasks.

[0054] Step 2.2: The contrastive learning network consists of two encoders E Q E K It consists of a linear dimensionality reduction module M and a memory storage module;

[0055] Will The combined image of any positive sample pair in the set is regarded as an anchor point and input into encoder E. Q The pre-training dataset X is processed in the following way: the other combined image in the positive sample pair is treated as a positive sample. * Except Other images are considered as negative samples, and positive samples are input together with the negative sample set into encoder E. K The process is performed to obtain preliminary features of the anchor points, positive samples, and negative sample sets; in this embodiment, E QE K All are ResNet50 encoders, such as Figure 1 As shown, due to the limitation of video memory size, the negative sample set and the positive sample set cannot be input into E simultaneously. K However, the parameter updates during the training process are iterative. Therefore, in order to ensure that the feature differences between positive and negative samples change little, E K With E Q The continuous iteration of parameters will automatically update the momentum, as shown in equation (3):

[0056] E K =mE K +(1-m)E Q (3)

[0057] In equation (3), m is a controllable parameter. In this example, m is taken as 0.999, E K Initialization and E Q Consistent;

[0058] The anchor points obtained through step 2.1 Input encoder E Q Obtain feature q, and Input another encoder E K Obtain feature k + In addition, other images in the current batch during training... After encoder E k A series of negative sample features can be obtained. After obtaining the features, contrastive learning can be viewed as a dictionary lookup task, given a query q from the dictionary and the dictionary contents. During training, we hope that q can find its corresponding positive sample k. + Contrastive learning essentially encourages q to continuously approach the positive sample k during the convergence process. + Stay away from negative samples The statement is as follows:

[0059] score(E q (q),E k (k + ))>>score(E q (q),E k (k - (4)

[0060] The linear dimensionality reduction module M consists of an MLP layer containing two fully connected layers and a ReLU function, and is used to perform linear dimensionality reduction on the preliminary features of the anchor points, positive samples, and negative sample sets to obtain the features of the anchor points, positive samples, and negative sample sets after dimensionality reduction.

[0061] The memory storage module is a queue structure and is used for dynamically storing features of the reduced dimension negative sample set;

[0062] In step 3, a back propagation double contrast loss function L is established by using formula (2):

[0063]

[0064] In formula (2), α and β are proportional coefficients, and α+β=1; q is a feature of the reduced dimension anchor point; k + is a feature of the reduced dimension positive sample, k i - is any ith feature of the reduced dimension negative sample set; τ is a temperature coefficient; in this embodiment, α and β are both 0.5, and τ is 0.07 after testing;

[0065] In step 4, the contrast learning network is trained by using the gradient descent method, and the double contrast loss function L is calculated to update the network parameters. When the value of the loss function L is less than a set threshold or the number of training iterations reaches a set requirement, the training is completed, and an optimal pre-training model is obtained. In this embodiment, the training is ended after 600 epochs, and a pre-training completed ResNet50 encoder is obtained.

[0066] In step 5, the pre-training model is migrated to a cross-modal pedestrian re-identification model, and a near-infrared dataset D t =(X t ,Y t ) is input to the migrated cross-modal pedestrian re-identification model for training, so as to fine-tune the parameters of the cross-modal pedestrian re-identification model, thereby obtaining a fine-tuned cross-modal pedestrian re-identification model, which is used for cross-modal pedestrian re-identification of a to-be-predicted near-infrared image data. In this example, the downstream test model is an AGW model, the ResNet50 network obtained in step 4 is migrated to the AGW model, the training set of the SYSU-MM01 dataset is used for training, and after the training is completed, the test set of the SYSU-MM01 dataset is used for testing to obtain a recognition result.

[0067] In this embodiment, an electronic device includes a memory for storing a program supporting a processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0068] In this embodiment, a computer readable storage medium has a computer program stored thereon, and the computer program is run by a processor to execute the steps of the above method.

Claims

1. A cross-modal pedestrian re-identification method based on self-supervised learning and pre-training model, characterized in that, The method comprises the following steps: Step 1, obtaining and preprocessing pre-training data; Step 1.1, collect M visible light pedestrian images from the video using the YOLO algorithm and perform normalization preprocessing to obtain a visible light dataset wherein, represents the mth visible light pedestrian image; Near-infrared images of N pedestrians are collected by a near-infrared camera and normalized to obtain a near-infrared data set wherein, represents a near-infrared image data set, and , represents the nth near-infrared pedestrian image, is an identity label data set; Step 1.

2. Using the channel enhancement method to convert the visible light dataset into a third modality closer to near-infrared, obtaining a visible light enhanced dataset Step 1.

2. Using the channel enhancement method to convert the visible light dataset into a third modality closer to near-infrared, obtaining a visible light enhanced dataset ; wherein, represents the mth visible light enhanced pedestrian image; Step 1.

3. Compute the cross-modal catastrophic forgetting score between the mth visible light augmented pedestrian image and the mth near-infrared pedestrian image using formula (1) Step 1.

4. Compute the feature distance between the mth visible light augmented pedestrian image and the mth near-infrared pedestrian image using formula (2) Step 1.

5. Compute the feature distance between the mth visible light augmented pedestrian image and the mth near-infrared pedestrian image using formula (3) Step 1.

6. Compute the feature distance between the mth visible light augmented pedestrian image and the mth near-infrared pedestrian image using formula (4) Step 1.

7. Compute the feature distance between the mth visible light augmented pedestrian image and the mth near-infrared pedestrian image using formula (5) : (1) In formula (1), an encoder trained unsupervised on a visible light dataset an encoder trained unsupervised on a visible light dataset an encoder trained supervised on a near-infrared image dataset an encoder trained supervised on a near-infrared image dataset Step 1.

4. Obtain the feature distance between each visible light enhanced pedestrian image and each near-infrared pedestrian image according to the process of step 1.3, and select the closest distance from them visible light pedestrian images, for constructing the unlabeled pre-training data set ; wherein, represents the selected first visible light pedestrian images; Step 2, constructing a channel combination module and a contrast learning network, the channel combination module is used for generating a positive sample image, and the contrast learning network is used for extracting image features; Step 2.1, the channel combination module comprises: a channel separation and replication unit, a channel erasing unit, and a channel superposition unit; and inputting the positive sample image pair set into the channel combination module for processing to obtain a negative sample image pair set. The channel separation and replication unit separates and replicates the R, G and B three channels of any one visible light pedestrian image to obtain a plurality of replication images of the visible light pedestrian image on the three channels; The channel erasing unit uses a channel erasing function to erase part of the pixels of the plurality of replication images of the visible light pedestrian image on the three channels respectively; The channel superposition unit randomly selects three replication images from the plurality of replication images of the visible light pedestrian image after erasing on the three channels to combine to obtain a combined image, thereby obtaining a plurality of combined images of the visible light pedestrian image, and any two combined images constitute a positive sample image pair; Step 2.2, the contrast learning network is composed of two encoders , , a linear dimension reduction module , a memory storage module Will The combined image of any positive sample pair in the set is regarded as an anchor point and input into the encoder. In the process, the other combined image in the positive sample pair is treated as a positive sample, and the pre-training dataset is processed. Except Other images are treated as a negative sample set and input into the encoder along with them. The data is processed to obtain preliminary features of the anchor points, positive samples, and negative sample sets. The linear dimension reduction module The MLP layer is composed of two full connection layers and a Relu function, and is used for linear dimension reduction on the preliminary features of the anchor point, the positive sample and the negative sample set to obtain the features of the anchor point, the positive sample and the negative sample set after dimension reduction. The memory storage library module is a queue structure and is used for dynamically storing the features of the negative sample set after dimension reduction; Step 3, use of formula (2) to establish a backpropagation double contrast loss function : (2) In formula (2), , is a proportional coefficient, and ; is a feature of the anchor point after dimension reduction; is a feature of the positive sample after dimension reduction, is any ith feature of the feature of the negative sample set after dimension reduction; is a temperature coefficient; indicates the number of features of the negative sample set after dimension reduction; Step 4, training the contrast learning network by using gradient descent method and calculating the dual contrast loss function updating the network parameters when the value of the loss function is less than a set threshold value or the number of training iterations reaches a set requirement, completing the training and obtaining an optimal pre-training model; Step 5, migrating the pre-trained model to a cross-modal pedestrian re-identification model, and inputting the near-infrared dataset training on the migrated cross-modal pedestrian re-identification model to fine-tune the parameters of the cross-modal pedestrian re-identification model, so as to obtain a fine-tuned cross-modal pedestrian re-identification model, which is used for cross-modal pedestrian re-identification on the to-be-predicted near-infrared image data.

2. An electronic device comprising a memory and a processor, characterized in that The memory is used for storing a program supporting the processor to execute the cross-modal pedestrian re-identification method of claim 1, and the processor is configured to execute the program stored in the memory.

3. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to execute the steps of the cross-modal pedestrian re-identification method of claim 1.

Images