Cross-modal pedestrian re-identification method based on diffusion model

Abstract

The cross-modal pedestrian re-identification method based on diffusion model belongs to the field of image retrieval. The problem of identifying the same pedestrian caused by the dramatic change of light conditions is solved by using diffusion model, so as to solve the cross-modal retrieval problem. 1) Divide multiple infrared data sets and visible light data sets, divide infrared sample data sets and visible light sample data sets in each training stage, and each data contains two positive samples and multiple negative samples; 2) using resnet50 network to extract picture information, and forming a smaller size tensor. 3) using two diffusion models to reconstruct the tensor of two sample data sets and calculate the loss function, so that the model can extract the information in the sample; 4) apply the diffusion model trained in one sample data set to the sample reconstruction in another sample data set, and use the guide network to guide the model to generate positive samples, enhance the cross-modal pedestrian re-identification ability of the model.

Description

Technical Field

[0001] The present invention belongs to the field of image retrieval and relates to a visual target detection method, and in particular to a cross-modal pedestrian re-identification method based on a diffusion model. Background Art

[0002] As more and more cameras are deployed in various scenarios, the scale of images captured by them is also increasing. Finding the same person in a massive amount of images is a challenging yet highly practical technology. Due to issues with camera resolution and shooting angle, obtaining high-quality images of pedestrians is often impossible. When facial information is inadequate, person re-identification technology becomes a crucial alternative. Person re-identification technology offers novel solutions to these challenges and has become a new research hotspot in academia and industry in recent years.

[0003] Person re-identification (PIR) uses computer vision to determine whether a specific person exists in an image or video sequence. In other words, PIR identifies a target person from images or video sequences captured within non-overlapping camera fields of view. This technology has broad applications in scenarios such as unmanned supermarkets, smart urban surveillance systems, and image organization.

[0004] The cross-modal retrieval task is to find the image of the pedestrian in another modality using computer vision technology, given an image of the pedestrian in one modality. However, in the cross-modal pedestrian retrieval task, images from different modalities result in differences between the distributions generated by the two modalities. The traditional single-modality method of bridging the difference between the distributions will make the features of some pedestrians closer to the features of different pedestrians with similar features in the other modality, resulting in retrieval errors. Therefore, traditional single-modality pedestrian re-identification methods have difficulty solving the problem of pedestrian re-identification in poor lighting conditions and dark environments, which greatly limits the all-weather deployment and practical application of pedestrian re-identification technology. How to eliminate modal differences and obtain discriminative cross-modal retrieval features under given infrared-visible light image conditions is the key to cross-modal pedestrian re-identification.

[0005] The main challenge in cross-modal person re-identification lies in the discrepancy between the distributions of images from different modalities. Early research focused on representation learning and metric learning, followed by modality conversion methods. The key idea of ​​these methods is to convert images between the two modalities to reduce the modality discrepancy.

[0006] Representation learning-based methods primarily study how to design appropriate network structures to extract discriminative pedestrian features shared between two modalities. Wu et al. (Wu, Ancong, et al. "RGB-Infrared Cross-Modality Person Re-identification." 2017 IEEE International Conference on Computer Vision (ICCV) (2017): 5390-5399) first proposed the cross-modal person re-identification problem in the field of person re-identification. They analyzed three network architectures and contributed a new multimodal ReID dataset, SYSU-MM01: a single-stream network, a two-stream network, and an asymmetric fully connected layer structure. They also proposed a data preprocessing method with deep zero padding and compared the performance of these network architectures.

[0007] Methods based on metric learning aim to learn the similarity between two images through a network. The key to this method lies in designing a reasonable metric function and loss function, so that the features of the same pedestrian in the same and different modalities are close to each other, while the distance between the features of different pedestrians is as large as possible. Ye et al. (YE M, WANG Z, LAN X, et al. Visible thermal person re-identification via dual-constrained top-ranking [C] / / Twenty-Seventh International Joint Conference on Artificial Intelligence IJCAI-18, 2018) proposed a bidirectional ranking loss to bring sample images of the same pedestrian in different modalities closer together, integrating it with the traditional cross-entropy loss to achieve the purpose of fusing features from different modalities.

[0008] In the generation task, diffusion model is a kind of generative model that can generate observation values randomly with respect to some implicit parameters. At present, diffusion model has gradually become a popular generative model due to its strong generation ability. The goal of diffusion model is to transform the prior data distribution into random noise, and gradually reverse transform to reconstruct a new instance with the same distribution as the prior. For today's generative model, a milestone work is DDPM (Ho, J., Jain, A., & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. ArXiv, abs / 2006.11239). Diffusion model neither needs to align the posterior distribution like VAE, nor needs to deal with the tricky partition function like EBM, nor needs to train additional classifiers like GAN and introduce network constraints like NF. At present, diffusion model has been widely used in CV, speech generation, bioinformatics and NLP fields due to its strong generation ability, and has achieved exciting results.

[0009] Compared with other generative models, diffusion models require a large number of sampling steps to generate high-quality instances. Therefore, the main disadvantage of diffusion models is that sampling consumes a large amount of time, and since both the forward process and the reverse process follow a Markov chain, this greatly increases the time required for sampling. At present, general diffusion models are accelerated for sampling. The method of accelerating sampling can be divided into four categories: changing the training mechanism, non-training sampling, hybrid models and score diffusion unification. Salimans et al. (T. Salimans and J. Ho, “Progressive distillation for fast sampling of diffusion models,” arXiv preprint arXiv:2202.00512, 2022) borrowed the core idea of accelerating training with knowledge distillation to increase the sampling speed of diffusion models by gradually distilling knowledge from one sampling model to another. In each distillation step, the student model is re-parameterized by the teacher model, and then trained to make the generated instances close to the instances generated by the teacher network. In each distillation process, the student model can reduce the sampling time by half. Further improvements along the distillation path include de-noising the student network, that is, the teacher network guides the sampling mode of the student network by minimizing the L2 loss of the two network-generated instances. These methods effectively reduce the sampling time without losing the expressiveness of the model. TDPM (H. Zheng, et al. “Truncated diffusion probabilistic models,” arXiv preprint arXiv:2202.09671, 2022) proposes to truncate the diffusion process and the sampling process to reduce the sampling time, which is conducive to shortening the sampling time and improving the generation quality. The key idea of the truncation mode is to generate less diffusion data with the help of other generative models, such as GAN or VAE. TDPM uses GAN and conditional conversion to learn the implicit generative distribution from random noise. In addition to the idea of reducing steps, Franzese et al. (G. Franzese, et al. “How much is enough? a study on diffusion times in score-based generative models,” 2022) define the number of training steps as a variable to train the model more flexibly and find the optimal steps.Cold diffusion (A.Bansal, et al. “Cold diffusion: Inverting arbitrary image transforms without noise,” 2022) shows that the data distribution of noise can be set to an arbitrary distribution by cold diffusion improved sampling to reduce the prediction error introduced by reconstructor design errors. Summary of the Invention

[0010] The purpose of the present invention is to address the shortcomings of general pedestrian re-identification methods that cannot achieve good results under conditions of drastic changes in lighting conditions. Considering the drastic changes in lighting and the failure of face recognition in different application scenarios, a cross-modal pedestrian re-identification method based on a diffusion model is provided.

[0011] The present invention comprises the following steps:

[0012] 1) Divide multiple infrared datasets and visible light datasets into two sets of infrared sample datasets and visible light sample datasets in each training stage, and each set contains two positive samples and multiple negative samples;

[0013] 2) Use the resnet50 network to extract image information and form a smaller tensor;

[0014] 3) Use two diffusion models to reconstruct the tensors of the two sample data sets and calculate the loss function, so that the model can extract information from the samples;

[0015] 4) Apply the diffusion model trained in one sample dataset to another sample dataset to reconstruct the samples, and use the guidance network to guide the model to generate positive samples to enhance the model's cross-modal pedestrian re-identification capability.

[0016] In step 1), multiple infrared datasets and visible light datasets are divided. In each training stage, an infrared sample dataset and a visible light sample dataset are divided, and each dataset contains two positive samples and multiple negative samples. The specific division method is shown in Table 1:

[0017] Table 1 Data division method

[0018] subset ids images visible 395 22258 Infra 395 11909 query 96 3803 gallery 96 301

[0019] Among them, visible represents the visible light training set, which has a total of 395 pedestrians and contains 22,258 images; infra represents the infrared training set, which contains the same pedestrians as visible but only has 111,909 images; query and gallery are test sets, both containing 96 pedestrians, with set sizes of 3,803 and 301 images, respectively.

[0020] In step 2), the image information is extracted using the resnet50 network and a smaller tensor is formed by inputting the visible and infra images into the trained resnet50 network respectively to obtain a smaller but larger dimensional tensor.

[0021] In step 3), the two diffusion models are used to reconstruct the tensors of the two sample data sets and calculate the loss function. The tensors output from RensNet50 are respectively passed through the diffusion models of the corresponding modes to obtain generated images to extract the structural information of the images. The specific steps are as follows:

[0022] 3a) The noise tensor x at time t∈[0,T] is obtained by the following formula: t ;

[0023]

[0024] Among them, x0 is the tensor output by resnet50, ε is Gaussian noise, is a constant related to time t;

[0025] 3b) The noise tensor x t Input it into the UNet network together with time t, obtain the output of the UNet network for tensor noise, sample a Gaussian noise ε, and then train the UNet network by minimizing the following loss function;

[0026]

[0027] After 100 training cycles, the diffusion model can effectively extract the structural information of the tensor.

[0028] In step 4), the diffusion model trained in one sample dataset is applied to another sample dataset to reconstruct the samples, and the guidance network is used to guide the model to generate positive samples to enhance the model's cross-modal pedestrian re-identification capability. The specific steps are as follows:

[0029] 4a) Randomly initialize the encoder E in the visible light modality v and encoder E in infrared mode i,The model structure of the two encoders is the same, both of which are a classifier with a fully connected network added after resnet50;

[0030] 4b) Randomly sample a visible light image x from the dataset v ,At time t=T, the distribution of p~N(0,1) is initially pure Gaussian;

[0031] 4c) Sample a noise from the p distribution And input the noise and time t into UNetDF v In the network; get the output of the network Then with Perform linear combination to obtain the output of the non-classifier guided diffusion model at time t-1

[0032]

[0033] 4d) Input middle;

[0034] 4e) The noise state of the visible light image at time t is obtained by the following formula: and enter

[0035]

[0036] 4f) Calculate the encoder by the following formula and Similarity between outputs;

[0037]

[0038] 4g) Calculate the derivative of the obtained similarity with respect to ε and sample a noise z from a pure Gaussian distribution; then derivative The state of the classifier-guided diffusion model at time t-1 is obtained by summing and z through the following formula Distribution

[0039]

[0040] 4h) Repeat from step 4b) until t = t-1; end the loop at t = 0 and obtain the distribution

[0041] 4i) From the distribution Sampling a tensor Input it into the fully connected network to get one-hot, and calculate the distance with the one-hot vector of the pedestrian image in the visible light dataset. The one with the closest distance is the corresponding pedestrian.

[0042] The present invention pre-trains a classifier-free diffusion model by reconstructing samples from a single modality, enabling each diffusion model to learn the structural information of the samples. Diffusion model generation is guided by a unique encoder E for each modality, whose input is either the current state of the diffusion model or an intermediate state obtained by adding noise to the guidance image. Diffusion model generation is guided by calculating the similarity between the outputs of the encoders of the two modalities.

[0043] Compared with the prior art, the present invention has the following outstanding advantages:

[0044] 1) This invention uses a generative model to generate images of the other modality, thereby bridging the gap between modalities and resolving the difficulty of identifying the same person in low-light conditions. Traditional methods, however, extract features and directly find the nearest neighbor. However, due to low image clarity, the distance between distributions is small, leading to misclassification.

[0045] 2) This method can control the image generation process and the training process, effectively extracting the image's structural information. GAN-based methods, on the other hand, can only be controlled by the final min-max loss, making the training process uncontrollable, prone to mode collapse, and lacking diversity in the generated images.

[0046] 3) The present invention can be used in fields such as intelligent monitoring, target tracking and unmanned supermarkets. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a step framework diagram of the present invention.

[0048] Figure 2 It is the DF+EN module in the step diagram.

[0049] Figure 3 It is a schematic diagram of the diffusion model.

[0050] Figure 4 It is a feature visualization graph. DETAILED DESCRIPTION

[0051] The specific implementation steps of the present invention are further described below.

[0052] This invention proposes a method for generating samples based on a diffusion model, taking into account not only the differences between samples within the same modality but also between samples in different modalities. This method bridges the modality gap by using a generative model to generate samples from another modality. Furthermore, considering the differences in model generalization performance across different samples and the diversity of generated samples, this invention uses a similarity metric to control the generation of the generative model and employs intra-class and inter-class loss functions to enhance the model's discriminative capabilities. Furthermore, considering the time-consuming nature of the diffusion model, cross-modal samples are generated starting from an intermediate state. Specifically, this invention comprises two main components. First, a classifier-free diffusion model is pre-trained by reconstructing samples from a single modality, enabling each diffusion model to learn the structural information of the samples. Second, the generation of the diffusion model is guided by a modality-specific encoder E. Its input is either the current state of the diffusion model or an intermediate state obtained by noise processing of the guidance image. The generation of the diffusion model is then guided by calculating the similarity between the outputs of the encoders of the two modalities.

[0053] In this paper, the number of layers in both the encoder and UNet networks was set to 6. The backbone network was set to ResNet50 and pre-trained. Between epochs 0 and 10, the learning rate was gradually increased from 0.01 to 0.1. The learning rate was then kept constant between epochs 10 and 20, and then set to 0.01 between epochs 20 and 30, and then to 0.001 for subsequent epochs. The model was trained in parallel on four cards with a batch size of 16 per card. The Adam optimizer was used throughout the training process.

[0054] The overall framework is as follows Figure 1 As shown, it specifically includes the following aspects:

[0055] 1. Obtain a training sample set.

[0056] To extract modality-independent discriminative structural information, the source dataset is divided into a visible light dataset (DR) and an infrared dataset (DI) at each training stage, with the same pedestrians in both datasets. For each batch, B pedestrians are selected, and N images are randomly selected from each dataset for each pedestrian.

[0057] 2. Pre-train backbone.

[0058] In order to make the training stable and converge quickly, we first use the dataset with the intra-class loss L id and triplet loss L tri train Figure 1 The left part is respectively v and B iRepresents resnet50 for visible light dataset and resnet50 for infrared dataset.

[0059]

[0060] in, and Represents the anchor of visible light and the anchor of infrared, x p Indicates that x a The same pedestrian, x n Indicates that x a Different pedestrians.

[0061] 3. Pre-trained diffusion model.

[0062] Train through the following steps Figure 3 The unconditional diffusion model in . The forward direction is the diffusion direction, that is, x i The direction in which i increases is the generation direction.

[0063] (3a) Randomly sample B*N time t from a uniform distribution and form a tensor of t=[B*N,1].

[0064] (3b) Randomly sample B*N Gaussian noises from the Gaussian distribution and form a tensor of ε=[B*N,1].

[0065] (3c) The sample x of the intermediate state corresponding to time t in the diffusion model is obtained by the following formula t .

[0066]

[0067] (3d) The intermediate state x t and time t is input to the UNet network ε θ and by minimizing the following loss function L t Train the UNet network.

[0068]

[0069] 4. Load the pre-trained backbone and unconditional diffusion models into the model.

[0070] The following process takes the generation process from visible light to infrared as an example, in which the main generation processes are as follows: Figure 2 As shown, the process of generating infrared to visible light is the same.

[0071] (4a) Input the visible light image into the backbone to reduce the image dimension and thus reduce the amount of computation.

[0072] (4b) Starting from the hyperparameter T, iteratively generate infrared modality images. The initial state is x T ~N(0,1). For time t=[0,T], the state of the current image is x t , first use Noise state of sampled visible light and input to the encoder E in the visible light modality v In the above example, we get the characteristic f of the state v . Then the generated image x t Input to encoder E in infrared mode i The feature f of the state is obtained i ; Calculate the cosine similarity of the two features and calculate the similarity with respect to x t The gradient of x t The input is fed into the diffusion model trained in the infrared mode, and the output is added to the gradient obtained previously to finally obtain the generated image x at the current moment. t-1 The specific formula is as follows:

[0073]

[0074] in, Right now

[0075] Considering the time-consuming disadvantage of the diffusion model, we start generating cross-modal samples from a certain state in the middle. Starting from the hyperparameter T, we iteratively generate infrared images, and the initial state is

[0076] Through the continuous iteration of the above formula, the cross-modal sample x0 is finally generated. The schematic diagram of this process is as follows Figure 2 shown.

[0077] (4c) The generated image is input into the backbone network to obtain its features, and its category is obtained through the fully connected layer, and the network is trained by minimizing the cross entropy loss and triplet loss.

[0078] 5. Analysis

[0079] The network first extracts structural information from the corresponding modality using a diffusion model. It then uses a modality-specific encoder to extract features from the generated state. It also uses another modality-specific encoder to extract features from a reference image that has been subjected to a certain amount of noise, ensuring that the generated image and the reference image have similar texture information. However, due to the use of a modality-specific diffusion model, only images within that modality are generated. Therefore, by using the texture information provided by the encoder and a pre-trained, dedicated diffusion model, this method can be used to guide the diffusion model in generating images from one modality to the other, thereby bridging the modality gap.

[0080] Compared with traditional methods, this invention generates another modality image through generative model, so as to bridge the difference between modalities and solve the problem of difficulty in identifying the same pedestrian under insufficient light conditions. Traditional methods directly find the nearest neighbor by extracting features, which will lead to a small distance between distributions due to the low clarity of the image, resulting in incorrect classification. Figure 4 As shown, Figure 4 The leftmost figure shows the feature distribution of each mode extracted by the traditional method. Different shapes represent different pedestrians, and the same shape represents the same pedestrian. id and L tri When training the network, we get Figure 4 The feature distribution in the middle will make some features closer to the features of different pedestrians in another mode (for example, the features with arrows on the way). Figure 4 As shown in the right part, by mapping the image in the visible light (RGB) mode to the infrared (IR) mode, it is only necessary to use traditional methods in a single mode to classify the features, thereby increasing the accuracy.

Claims

1. A cross-modal person re-identification method based on a diffusion model, characterized by The following steps are involved: 1) Divide multiple infrared datasets and visible light datasets into two sets of infrared sample datasets and visible light sample datasets in each training stage, and each set contains two positive samples and multiple negative samples; 2) Use the resnet50 network to extract image information and form a smaller tensor; 3) Use two diffusion models to reconstruct the tensors of the two sample data sets and calculate the loss function, so that the model can extract information from the samples; 4) Apply the diffusion model trained in one sample dataset to another sample dataset to reconstruct the samples, and use the guidance network to guide the model to generate positive samples to enhance the model's cross-modal pedestrian re-identification capability. The specific steps are as follows: 4a) Randomly initialize the encoder E in the visible light modality v and encoder E in infrared mode i ,The model structure of the two encoders is the same, both of which are a classifier with a fully connected network added after resnet50; 4b) Randomly sample a visible light image x from the dataset v ,At time t=T, the distribution of p~N(0,1) is initially pure Gaussian; 4c) Sample a noise from the p distribution And input the noise and time t into UNet DF v In the network; get the output of the network Then with Perform linear combination to obtain the output of the non-classifier guided diffusion model at time t-1 4d) Input to middle; 4e) The noise state of the visible light image at time t is obtained by the following formula: and enter 4f) Calculate the encoder by the following formula and Similarity between outputs; 4g) Calculate the derivative of the obtained similarity with respect to ε and sample a noise z from a pure Gaussian distribution; then derivative The state of the classifier-guided diffusion model at time t-1 is obtained by summing and z through the following formula Distribution 4h) Repeat from step 4b) until t = t-1; end the loop at t = 0 and obtain the distribution 4i) From the distribution Sampling a tensor Input it into the fully connected network to get one-hot, and calculate the distance with the one-hot vector of the pedestrian image in the visible light dataset. The one with the closest distance is the corresponding pedestrian.

2. The cross-modal person re-identification method based on the diffusion model as claimed in claim 1 is characterized in that In step 1), the plurality of infrared data sets and visible light data sets are divided into visible, infra, query and gallery; visible represents the training set of visible light; infra represents the training set of infrared light; query and gallery are test sets.

3. The cross-modal person re-identification method based on the diffusion model as claimed in claim 1, characterized in that In step 2), the image information is extracted using the resnet50 network and a smaller tensor is formed by inputting the visible and infra images into the trained resnet50 network respectively to obtain a smaller but larger dimensional tensor.

4. The cross-modal person re-identification method based on the diffusion model as claimed in claim 1, characterized in that In step 3), the two diffusion models are used to reconstruct the tensors of the two sample data sets and calculate the loss function. The tensors output from RensNet50 are respectively passed through the diffusion models of the corresponding modes to obtain generated images to extract the structural information of the images. The specific steps are as follows: 3a) The noise tensor x at time t∈[0,T] is obtained by the following formula: t ; Among them, x0 is the tensor output by resnet50, ε is Gaussian noise, is a constant related to time t; 3b) The noise tensor x t Input it into the UNet network together with time t, obtain the output of the UNet network for tensor noise, sample a Gaussian noise ε, and then train the UNet network by minimizing the following loss function; After 100 training cycles, the diffusion model can effectively extract the structural information of the tensor.

Images