An adaptive feature fusion and cross-modal person re-identification method

Abstract

The present invention discloses an adaptive feature fusion and cross-modal pedestrian re-identification method, comprising the following steps: (1) obtaining a SYSU-MM01 dataset or a RegDB dataset from an open source website, and preprocessing the dataset using a multi-channel hybrid supplementation strategy; (2) constructing a two-stream network model, wherein the two-stream network model is composed of two window shift transformer second-generation backbone network modules connected in series; inputting the dataset into the two-stream network model for training to obtain the prediction probability of each category of the image and the extracted feature vector; (3) calculating the distance matrix between the query feature and the library feature by matrix multiplication, and calculating the cumulative matching characteristic curve CMC, the mean average precision mAP, and the mean inverse precision mINP according to the distance matrix to evaluate the retrieval performance of the model; the performance of the present invention is superior to that of the prior art.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an adaptive feature fusion and cross-modal pedestrian re-identification method. Background Art

[0002] Cross-modal pedestrian re-identification is an important and challenging specific pedestrian retrieval task that needs to solve the feature differences between visible light and infrared modalities. It has high application value for the construction of security systems. Due to different imaging mechanisms, the feature representations of visible light images and infrared images are very different. In order to improve the feature information mining of the two modalities, the existing methods have two core strategies. The first one adopts image-level processing methods, that is, converting one modality into another or constructing an intermediate state, and strives to reduce the modal differences during data initialization. The second one ensures the alignment of modal features by designing a cross-modal network architecture and loss, effectively mapping cross-modal pedestrian features to the feature space.

[0003] The cross-modal person re-identification method that uses conventional CNN as the backbone has the risk of losing spatial information during model training due to the existence of its pooling layer, which indirectly increases the difficulty of aligning the features of the two modalities.

[0004] The image-level processing method lacks reliable feature mapping, and the additional noise and increased computational complexity make it difficult for model training to converge. Designing a cross-modal network architecture and loss method does not really solve the problem of alienation of cross-modal positive sample pairs in the feature space. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to provide an adaptive feature fusion and cross-modal pedestrian re-identification method to solve the problems existing in the background technology.

[0006] Technical solution: The method for adaptive feature fusion and cross-modal person re-identification described in the present invention comprises the following steps:

[0007] (1) Obtain the SYSU-MM01 dataset or RegDB dataset from the open source website and preprocess it using a multi-channel mixed supplementation strategy;

[0008] (2) Construct a two-stream network model, which consists of two window shift transformer second-generation backbone network modules (build_swin_transformerV2) connected in series; input the data set into the two-stream network model for training to obtain the predicted probability of each category of the image and the extracted feature vector;

[0009] (3) The distance matrix between the query features and the library features is calculated by matrix multiplication. The cumulative matching characteristic curve CMC, mean average precision (mAP), and mean inverse precision (mINP) are calculated based on the distance matrix to evaluate the retrieval performance of the model.

[0010] Furthermore, step (1) is specifically as follows: for each data set, visible light and infrared images are loaded respectively; the preprocessing is specifically as follows: a multi-channel mixed supplementation strategy is used to replace the color channel of the visible light image and grayscale it, and the visible light image and the monochrome channel grayscale image are mixed through a random scheduler, and at the same time, the relationship between the image under the monochrome channel grayscale change and the feature information of the visible light and infrared images is obtained.

[0011] Furthermore, the dual-stream network model in step (2) includes two stages, each stage is composed of a window shift transformer second-generation backbone network module; wherein, the first stage and the second stage window shift transformer second-generation backbone network module both contain two normalization layers, and a multi-layer perceptron is introduced between the two normalization layers; in the first stage, a multi-head self-attention module is added after the normalization layer; in the second stage, a sliding window self-attention module is added after the normalization layer.

[0012] Furthermore, the first stage is as follows: the multi-head self-attention mechanism is used to capture the global relationship of features, and the feature representation is enhanced through normalization and MLP; among them, the first stage adopts the difficult sample mining triple loss function and identity loss function, and the formula is as follows:

[0013] ;

[0014] Among them, the triplet loss of difficult sample mining is expressed as:

[0015] ;

[0016] in, represents the Euclidean distance between samples, is the anchor sample, It is a difficult sample. is a hard negative sample, It is the margin hyperparameter, which is used to increase the distance between positive samples and negative samples;

[0017] The identity loss is expressed as:

[0018] ;

[0019] in, is the number of training samples, It is The true identity labels of samples, The model predicts The probability distribution of samples belonging to each identity.

[0020] Furthermore, the second stage is as follows: receiving the predicted probability of each category of the image output in the first stage and the extracted feature vector, further refining the features through the sliding window self-attention mechanism, performing self-attention calculations in a local range, while maintaining the integrity of local information. Among them, the second stage adopts the ratio-based center difference loss function and the cross-modal intra-class denoising loss function based on the loss function of the first stage; the formula is as follows:

[0021]

[0022] in, and is a hyperparameter; , The value range is 0 or 0.1 to 0.9, and the interval is 0.2; represents the triplet loss of difficult sample mining, Indicates identity loss, represents the ratio-based center difference loss, represents the cross-modal intra-class denoising loss.

[0023] Furthermore, the ratio-based center difference loss function formula is as follows:

[0024] ;

[0025] in, It is the arithmetic mean of a group of values, calculated by adding all the values ​​and then dividing by the number of values. Total number of representative identities, Representative status The characteristics of negative samples;

[0026] ;

[0027] in, Representative status The characteristic center of Represents the number of samples in a single sample with a single identity, Represents the features in the visible light mode after multi-channel mixing and supplementation. Representing the features in the original visible light mode, the average distance between the feature center and the negative sample is used as the dynamic boundary formula as follows:

[0028] ;

[0029] The formula of the cross-modal intra-class denoising loss function is as follows:

[0030] ;

[0031] ;

[0032] ;

[0033] in, Total number of representative identities, The number of samples in a single identity for a single sample, is the anchor sample (AnchorSample), represents the identity of the anchor sample, represents the sample value of the anchor sample, is the average distance between the anchor sample and other positive samples in its internal mode, is the average distance between the anchor sample and other positive samples under cross-modality. In the endomembrane state, is cross-modal.

[0034] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: in order to avoid the problem that the cross-modal samples of the same pedestrian are far away in the feature space, a ratio-based central difference loss is designed, and the cross-modal intra-class denoising loss is designed to dynamically calculate the average distance of positive and negative samples, strengthen the intra-class and inter-class differences, and adjust the feature space in stages; a data set multi-channel hybrid supplementation strategy is proposed to enrich the data set and reduce the low-level modal differences. Experiments have proved that the present invention has better performance than the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a flow chart of the present invention;

[0036] Figure 2 This is a schematic diagram of the network structure of the present invention;

[0037] Figure 3 Schematic diagram of hyperparameters of the present invention. DETAILED DESCRIPTION

[0038] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0039] like Figure 1 As shown, an embodiment of the present invention provides an adaptive feature fusion and cross-modal pedestrian re-identification method, which includes the following steps:

[0040] (1) Obtain the SYSU-MM01 dataset or RegDB dataset from the open source website and perform preprocessing using the multi-channel mixed supplementation strategy. Specifically, for each dataset, load the visible light and infrared images separately. Preprocessing is as follows: Use the multi-channel mixed supplementation strategy to replace the color channel of the visible light image and perform grayscale processing. Use a random scheduler to mix the visible light image and the monochrome channel grayscale image. At the same time, obtain the relationship between the feature information of the image under the grayscale change of the monochrome channel and the visible light and infrared images. The formula is as follows:

[0041] Assume that the original cross-modal dataset is represented as:

[0042] , ;

[0043] in, represents the training set of visible light images. Each image is defined as , k represents the ID value defined by the person in the data set, , , The components of the three color channels of the image.

[0044] Represents a training set of infrared images, where each element Represents an infrared image with a single oversaturated grayscale channel. With the previous training set representation of visible light images, a new training set image with a single channel is formed by randomly selecting one channel to replace the other two channels. .

[0045] This process can be expressed as: ;

[0046] in, Represents the components of random single channels of R, G, and B. In the loss function, the original modal feature relationship and the modal feature relationship after adopting the strategy can be expressed as:

[0047] ;

[0048] in represents the features in the original visible light mode, Represents the features in the visible light mode after multi-channel mixing and supplementation. and represents the annotated training identity labels in each image, and Represents the loss function used.

[0049] (2) Construct a two-stream network model, such as Figure 2As shown in the figure, the two-stream network model consists of two window-shift transformer second-generation backbone network modules (build_swin_transformerV2) connected in series; the data set is input into the two-stream network model for training to obtain the predicted probability of each category of the image and the extracted feature vector; the two-stream network model includes two stages, each stage is composed of a window-shift transformer second-generation backbone network module; among them, the first stage and the second stage window-shift transformer second-generation backbone network module both contain two normalization layers, and a multi-layer perceptron is introduced between the two normalization layers; in the first stage, a multi-head self-attention module is added after the normalization layer; in the second stage, a sliding window self-attention module is added after the normalization layer. Among them,

[0050] Multi-head self-attention module: used to capture the global relationship of input features. Through multiple heads, the model can focus on features at different locations in parallel, thereby achieving cross-regional communication of information;

[0051] Normalization layer: used to standardize the output of the multi-head self-attention module to stabilize the training process of the model.

[0052] Multilayer Perceptron (MLP): It is used to further process and transform features. It consists of a series of fully connected layers and non-linear activation functions.

[0053] Sliding window self-attention module: It is used to limit the calculation scope of self-attention using a sliding window, thereby reducing the computational complexity.

[0054] The first stage is as follows: Use the multi-head self-attention mechanism to capture the global relationship of features, and enhance the feature representation through normalization and MLP. This stage ensures that the model has a global understanding of the overall structure of the input data. (Responsible for modeling global relationships, suitable for processing relatively broad features.); Among them, the first stage uses the difficult sample mining triple loss function and identity loss function, the formula is as follows:

[0055] ;

[0056] Among them, the triplet loss of difficult sample mining is expressed as:

[0057] ;

[0058] in, represents the Euclidean distance between samples, is the anchor sample, It is a difficult sample. is a hard negative sample, It is the margin hyperparameter, which is used to increase the distance between positive samples and negative samples;

[0059] The identity loss is expressed as:

[0060] ;

[0061] in, is the number of training samples, It is The true identity labels of samples, The model predicts The probability distribution of samples belonging to each identity.

[0062] The second stage is as follows: receive the predicted probability of each category of the image output in the first stage and the extracted feature vector, further refine the features through the sliding window self-attention mechanism, perform self-attention calculations in the local range, and maintain the integrity of the local information. Among them, the second stage adds the ratio-based center difference loss function and the cross-modal intra-class denoising loss function to the loss function of the first stage; the formula is as follows:

[0063] ;

[0064] in, and is a hyperparameter; , The value range is 0 or 0.1 to 0.9, and the interval is 0.2; represents the triplet loss of difficult sample mining, Indicates identity loss, represents the ratio-based center difference loss, represents the cross-modal intra-class denoising loss.

[0065] For hyperparameters and Experiment on the impact of different values ​​on training performance. and The value range is 0 and 0.1 to 0.9, with a value interval of 0.2. The experimental results are as follows Figure 3 As shown in the figure, it can be found that when the value is between 0.3 and 0.7, the model training is relatively stable. The value is 0.5, The value is 0.7, that is, the highest point in the figure is used as the configuration of this method.

[0066] The formula of the ratio center difference loss function is as follows:

[0067]

[0068] in, It is the arithmetic mean of a group of values, calculated by adding all the values ​​and then dividing by the number of values. Total number of representative identities, Representative status The characteristics of negative samples;

[0069] ;

[0070] in, Representative status The characteristic center of Represents the number of samples in a single sample with a single identity, Represents the features in the visible light mode after multi-channel mixing and supplementation. Representing the features in the original visible light mode, the average distance between the feature center and the negative sample is used as the dynamic boundary formula as follows:

[0071] ;

[0072] The formula of the cross-modal intra-class denoising loss function is as follows:

[0073] ;

[0074] ;

[0075] ;

[0076] in, Total number of representative identities, The number of samples in a single identity for a single sample, is the anchor sample (AnchorSample), represents the identity of the anchor sample, represents the sample value of the anchor sample, is the average distance between the anchor sample and other positive samples in its internal mode, is the average distance between the anchor sample and other positive samples under cross-modality. In the endomembrane state, is cross-modal.

[0077] (3) The distance matrix between the query features and the library features is calculated by matrix multiplication. The cumulative matching characteristic curve CMC, mean average precision (mAP), and mean inverse precision (mINP) are calculated based on the distance matrix to evaluate the retrieval performance of the model.

[0078] Experimental setup

[0079] The invention is presented on the PyTorch framework, and the training and evaluation are all performed on the NVIDIA 4090 GPU. The second-generation window shift transformer backbone network module is used, and the ImageNet pre-trained weight SwinV2-T is used for initialization and accelerated convergence. The size of the input image is adjusted to 256×256. In terms of basic data enhancement, random erasing and random flipping are adopted for visible light images, and random color jitter and random Gaussian blur are additionally added to infrared images. The adaptive gradient (Adamw) optimizer is used for training. For both data sets, the initial learning rate is set to , the weight decay is set to SYSU-MM01 and RegDB are trained for 20 and 28 epochs respectively, and the second phase of SMS starts from epoch 10. In each training, 4 visible light images and 4 infrared images under 4 different identities are randomly selected, and the 4 identities are taken as a batch. The margin parameter of the triplet loss for hard sample mining is set to 0.3, and the 1000-dimensional features after the BN layer are used for testing.

[0080] The present invention conducts experiments on the SYSU-MM01 dataset in full search and indoor search modes using a single camera setting, and conducts experiments on the RegDB dataset using two evaluation standard experiments using visible light images to query infrared images and infrared images to query visible light images. The experimental results of the present invention are compared with other current advanced cross-modal pedestrian re-identification algorithms, as shown in Table 1. Obviously, the present invention has a very good performance, and is superior to the prior art in both evaluation modes. In the SYSU-MM01 full search mode and the RegDB evaluation standard of using visible light images to query infrared images, the Rank1 of the present invention reaches 73.4% and 90.2% respectively, and the mAP reaches 71.1% and 87.1% respectively. The design of CID makes up for the problem that the traditional back propagation does not specifically focus on the position proximity between positive and negative sample pairs under cross-modality due to modal differences. Therefore, the performance is better than the classic cross-modal pedestrian re-identification AGW baseline that focuses more on representation learning, as well as the cutting-edge optimization method MCLNet proposed on this basis. By introducing Swin-TransformerV2 as the backbone of the entire model, combined with CHF in data preprocessing and SMS in the training stage, the method based on CNN dual-stream network, such as DDAG, LBA, and the method using VIT feature fusion network, such as PMT, has obvious advantages. The above results all show that the present invention has better performance than the existing methods.

[0081] Table 1 Comparison with state-of-the-art methods on RegDB and SYSU-MM01 datasets

[0082] .

Claims

1. An adaptive feature fusion and cross-modal person re-identification method, characterized in that: The following steps are involved: (1) Obtain the SYSU-MM01 dataset or RegDB dataset from the open source website and perform preprocessing using the multi-channel mixed supplementation strategy. Specifically, for each dataset, load the visible light and infrared images separately. Preprocessing is as follows: Use the multi-channel mixed supplementation strategy to replace the color channel of the visible light image and perform grayscale processing. Use a random scheduler to mix the visible light image and the monochrome channel grayscale image. At the same time, obtain the relationship between the image under the grayscale change of the monochrome channel and the feature information of the visible light and infrared images. (2) Construct a two-stream network model, which is composed of two window-shift transformer second-generation backbone network modules connected in series; input the data set into the two-stream network model for training to obtain the predicted probability of each category of the image and the extracted feature vector; the two-stream network model includes two stages, each stage is composed of a window-shift transformer second-generation backbone network module; among them, the first stage and the second stage window-shift transformer second-generation backbone network module both contain two normalization layers, and a multi-layer perceptron is introduced between the two normalization layers; in the first stage, a multi-head self-attention module is added after the normalization layer; in the second stage, a sliding window self-attention module is added after the normalization layer; among them, the first stage is as follows: the multi-head self-attention mechanism is used to capture the global relationship of the features, and the feature representation is enhanced by normalization and MLP; among them, the first stage adopts the difficult sample mining triplet loss function and identity loss function, and the formula is as follows: ； Among them, the triplet loss of difficult sample mining is expressed as: ； in, represents the Euclidean distance between samples, is the anchor sample, It is a difficult sample. is a hard negative sample, It is the margin hyperparameter, which is used to increase the distance between positive samples and negative samples; The identity loss is expressed as: ； in, is the number of training samples, It is The true identity labels of samples, The model predicts The probability distribution of samples belonging to each identity; The second stage is as follows: receive the predicted probability of each category of the image output in the first stage and the extracted feature vector, further refine the features through the sliding window self-attention mechanism, perform self-attention calculation in the local range, and maintain the integrity of the local information; among them, the second stage adopts the ratio-based center difference loss function and the cross-modal intra-class denoising loss function based on the loss function of the first stage; the formula is as follows: ； in, and is a hyperparameter; , The value range is 0 or 0.1 to 0.9, and the interval is 0.2; represents the triplet loss of difficult sample mining, Indicates identity loss, represents the ratio-based center difference loss, represents the cross-modal intra-class denoising loss; Among them, the formula of the ratio center difference loss function is as follows: ； in, It is the arithmetic mean of a group of values, calculated by adding all the values ​​and then dividing by the number of values. Total number of representative identities, Representative status The characteristics of negative samples; ； in, Representative status The characteristic center of Represents the number of samples in a single sample with a single identity, Represents the features in the visible light mode after multi-channel mixing and supplementation. Representing the features in the original visible light mode, the average distance between the feature center and the negative sample is used as the dynamic boundary formula as follows: ； The formula of the cross-modal intra-class denoising loss function is as follows: ； ； ； in, Total number of representative identities, The number of samples in a single identity for a single sample, is the anchor sample, represents the identity of the anchor sample, represents the sample value of the anchor sample, is the average distance between the anchor sample and other positive samples in its internal mode, is the average distance between the anchor sample and other positive samples under cross-modality; In the endomembrane state, is cross-modal; (3) The distance matrix between the query features and the library features is calculated by matrix multiplication. The cumulative matching characteristic curve CMC, mean average precision (mAP), and mean inverse precision (mINP) are calculated based on the distance matrix to evaluate the retrieval performance of the model.

Images