Cross-modal pedestrian re-identification method based on image alignment and channel enhancement

Abstract

The invention discloses a cross-modal pedestrian re-identification method based on image alignment and channel enhancement. The method mainly comprises the following steps: (1) generating a color-independent pedestrian image by using random channel exchange enhancement; (2) carrying out self-adaptive alignment adjustment on the visible light and infrared pedestrian images with dislocation by using a spatial transformation network; (3) respectively extracting the characteristics of the visible light pedestrian image and the infrared pedestrian image by using a ResNet50 network embedded with CBR non-local attention; and (4) the identity loss, the weighted regularization triple loss and the visible light-infrared sharing loss are jointly used to constrain the model. The method is mainly applied to the field of pedestrian re-identification, and has wide application prospects in the fields of video monitoring, image retrieval and the like.

Description

Technical Field

[0001] The present invention relates to a cross-modal pedestrian re-identification method based on image alignment and channel enhancement, and belongs to the field of computer vision and intelligent information technology. Background Art

[0002] Person re-identification (Re-ID) is a cross-camera pedestrian image retrieval work first proposed by Gheissari et al. in 2006, aiming to retrieve a given pedestrian query image from a gallery collected by different cameras. However, most of the current Re-ID algorithms focus on searching for visible pedestrian images taken by visible light cameras, and cannot cope with night scenes such as weak light and low illumination. Wu et al. proposed visible-infrared cross-modal person re-identification (VI-ReID) in 2017. Compared with traditional pedestrian re-identification research, VI-ReID is more challenging because of the significant modal differences and unknown environmental factors (such as occlusion and background interference, light changes, camera viewpoint differences, etc.).

[0003] In order to reduce the visible-infrared modality difference, generating cross-modal images through generative adversarial networks (GANs) is a popular method to mitigate modality differences by aligning pixel levels. However, image generation usually requires additional computational costs, and such algorithms rely heavily on training samples. In addition, some studies assume that color information is irrelevant and use grayscale images for cross-modal matching. Although these methods eliminate color differences, they lose discriminative information in color channels. Another method is to measure similarity by mapping visible and infrared pedestrian images to a common feature space. A dual-channel framework is usually used, which consists of two specific networks for extracting features and a parameter-sharing network for mapping the extracted features to a common feature space. This method can effectively extract modality-specific features and reduce modality differences, but it is difficult to extract modality-shared features. In addition, there is also spatial misalignment of pedestrians caused by factors such as camera perspective and pedestrian posture changes. In view of the above problems, the present invention proposes a cross-modal pedestrian re-identification method based on image alignment and channel enhancement to improve the performance of cross-modal pedestrian re-identification. Summary of the invention

[0004] The present invention proposes a cross-modal pedestrian re-identification method based on image alignment and channel enhancement. The purpose is to design a dual-channel network, use a spatial transformation module to adaptively align images, use the classic ResNet50 to extract the features of visible light and infrared pedestrian images, and jointly use multiple loss functions to optimize the model to achieve the purpose of improving the cross-modal pedestrian re-identification performance.

[0005] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0006] (1) Use Random Channel Exchangeable Enhancement (CE) to generate color-independent pedestrian images and learn the relationship between each color channel of single-channel infrared images and visible light images;

[0007] (2) Use the Spatial Transformer Network (STN) to adaptively align the misaligned visible light and infrared pedestrian images, and fuse the aligned visible light and infrared pedestrian images with the original images at the pixel level;

[0008] (3) Then, the ResNet50 network embedded with CBR non-local attention is used to extract the features of visible light pedestrian images and infrared pedestrian images respectively;

[0009] (4) The identity loss (ID loss), weighted regularization triplet loss (Weighted regularization Triplet loss) and visible-infrared shared loss (Visible-Infrared loss) are used jointly to constrain the model and achieve the purpose of improving the accuracy of cross-modal pedestrian re-identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a framework diagram of the cross-modal pedestrian re-identification method based on image alignment and channel enhancement of the present invention;

[0011] Figure 2 This is a structural diagram of the space transformation module STN of the present invention;

[0012] Figure 3 This is the CBR non-local attention structure diagram of the present invention. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not used to limit the present invention.

[0014] (1) Overall network framework

[0015] The network model framework is trained in an end-to-end manner. Figure 1As shown in the figure, random channel exchangeable enhancement is first used to improve the robustness to color changes. For pedestrian feature extraction, ResNet50 pre-trained on ImageNet is used as the backbone network. The spatial transformer network is used to transform the original visible light and infrared images, fuse the original image and the transformed image pixel by pixel, and send the fused visible light and infrared images to the ResNet50 network embedded with CBR non-local attention to obtain the visible light features f and f respectively. vs and infrared characteristics f ir The first block of ResNet50 is specific to the two modalities to obtain modality-specific features, and the remaining blocks are shared to learn features shared by the modalities. Identity loss (IDloss), weighted regularization triplet loss (Weighted regularization Triplet loss) and visible-infrared shared loss (Visible-Infrared loss) are jointly used to optimize the network model.

[0016] (2) Random channel exchange enhancement

[0017] The main idea of ​​Random Channel Exchangeable Enhancement (CE) is to randomly select a color channel to replace other color channels in a visible light image, which can be expressed as:

[0018]

[0019] The calculation formula of the visible light-infrared cross-modal objective function using CE is:

[0020]

[0021] in represents an infrared pedestrian image, Represents a three-channel RGB visible light image or its visible light image processed by CE, y i and j Indicates the label information of each picture, f v and f rdenote the feature extraction networks for visible light and infrared images respectively, and l(.) denotes the optimization function, which can be identity loss, triplet loss, or their variants. Specifically, first, an integer is randomly selected from [0,1,2,3] to decide whether to perform CE operation or retain the original RGB image. The random exchange channel enhancement operation can be easily combined with other data enhancements (random erasing, random cropping, etc.). This enhancement can be understood as uniformly generating a three-channel visible light image by decomposing the color channels, which helps the model f learn the relationship between the single-channel infrared image and each color channel of the visible light.

[0022] (3) Spatial Transformation Network

[0023] The output features of the fifth residual block conv_5x in ResNet50 are used as the input of the STN (Spatial Transformer Network) module, and a set of affine transformation parameters A are linearly regressed. θ , the corresponding coordinate relationship of the image before and after affine transformation is established by the following formula:

[0024]

[0025] in are the target coordinates in the regular network that aligns the images, are the source coordinates of the sampling points in the input image, is the affine transformation matrix, θ 11 ,θ 12 Controls the size of the transformed image, θ 21 and θ 22 Controls the rotation of the transformed image, θ 13 ,θ 23 Controls the offset of the transformed image. Since the affine transformation matrix contains continuously differentiable decimals, and the target coordinates is discrete, the source coordinates is continuous, and a certain sampling strategy is required to generate aligned images. In this method, bilinear sampling is used to obtain the value of a specific pixel in the transformed image. The formula is as follows:

[0026]

[0027] X pq represents the pixel value of the source image at (p,q), X′ mn Represents the pixel value of the transformed image at (m,n), and H and W represent the height and width of the source image (target image).

[0028] (4) CBR non-local attention

[0029] CBR non-local attention includes four CBR components, which are composed of 1×1 convolutional layers, batch normalization layers, and nonlinear activation functions leaky relu. The input feature size of CBR non-local attention is N×C×H×W, where N represents batch size, C represents the number of channels, and H and W represent the height and width of the feature, respectively. Assume φ′(X j )=W φ X j , where W φ is the weight matrix to be learned, for example, using a 1×1 convolution in space. Similarly, θ'(X j )=W θ X j , g'(X j )=W g X j φ(X j ),θ(X j ) and g(X j ) is defined as follows:

[0030] θ(X j )=ε(σ(γ(W θ X j ))) (5)

[0031] φ(X j )=ε(σ(γ(W φ X j ))) (6)

[0032] g(X j )=ε(σ(γ(W g X j ))) (7)

[0033] Where γ(.) represents the batch normalization function, σ(.) represents the activation function, and ε(.) represents merging the H and W dimensions into HW, while the batch size and number of channels remain unchanged. CBR non-local attention can be expressed as:

[0034]

[0035] f(X i ,X j )=θ(X i ) T φ(X j ) (9)

[0036] Where i represents the index of the output position of the response to be calculated, and j represents the index of all possible positions enumerated. i ,X j) is used to calculate the relationship between i and all possible associated positions j. g(X j ) calculates the representation of the input signal at position j. Formula (8) calculates the response based on the relationship between different positions. The formula for the CBR non-local block is:

[0037] z i =W z y i +x i (10)

[0038] Where "+x i ” indicates residual connection.

[0039] (5) Joint loss function

[0040] Identity loss:

[0041] set up and They represent the input visible light image and infrared images The identity prediction probability is expressed as follows:

[0042]

[0043]

[0044] set up and Respectively represent the visible light image input when identity is i and infrared images If the label information and For a dataset containing N pedestrians, P pedestrians are randomly selected. For each pedestrian, K visible light images and K infrared images of the pedestrian are randomly selected. The ID loss formula is as follows:

[0045]

[0046] Weighted regularized triplet loss:

[0047] Assumptions is the visible light anchor sample in the training batch, is the infrared positive sample with the same identity, are infrared negative samples with different identities. Weighted regularized triplet loss function Defined as:

[0048]

[0049]

[0050]

[0051] Where p represents the positive sample set of anchor sample i, n is the negative sample set, and Represent the distance weights of positive samples and negative samples respectively. Similarly, The definition is as follows:

[0052]

[0053]

[0054]

[0055] The overall weighted regularized triplet loss is:

[0056]

[0057] Visible-infrared sharing loss:

[0058] Assume that the anchor features of visible light images and infrared images in a training batch are and by As an example, we calculate the average distance between it and other positive samples within and between modalities:

[0059]

[0060]

[0061] L vis Defined as:

[0062]

[0063] L vis Punishment D intra and D cross When the discriminant feature appears only in one modality, D intra and D cross The difference between them will increase, and this anomaly will be L vis Capture. In D intra and D cross In the bidirectional optimization process, unreliable features that only appear in one modality will be suppressed, and reliable features that appear in both modalities will be enhanced.

[0064] The total loss function is defined as follows:

[0065] L total =L id +λ 1 Lwrt +λ 2 L vis (twenty four)

[0066] where λ 1 and λ 2 is a hyperparameter for balancing the corresponding terms and is set to 1.

[0067] In order to verify the effectiveness of the cross-modal pedestrian re-identification method based on image alignment and channel enhancement described in the present invention, experiments were carried out in the RegDB and SYSU-MM01 datasets. The experimental platform of this article is: Ubuntu 20.04, NvidiaRTX 2080 GPU. The deep learning framework used is Pytorch1.7, and the accuracy evaluation indicators used are Rank-1, mAP (mean average precision) and mINP (mean Inverse Negative Penalty). The experimental results of the present invention are shown in Tables 1 and 2. The experimental accuracy is greatly improved compared with the benchmark network and other comparison algorithms.

[0068] Table 1 Comparison of results of different algorithms on RegDB dataset

[0069]

[0070]

[0071] Table 2 Comparison of results of different algorithms on the SYSU-MMO1 dataset

[0072]

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-modal person re-identification method based on image alignment and channel enhancement, characterized in that The following steps are involved: (1) Use Random Channel Exchangeable Enhancement (CE) to generate color-independent pedestrian images and learn the relationship between each color channel of single-channel infrared images and visible light images; (2) Use the Spatial Transformer Network (STN) to adaptively align the misaligned visible light and infrared pedestrian images, and fuse the aligned visible light and infrared pedestrian images with the original images at the pixel level; (3) Use the ResNet50 network embedded with CBR non-local attention to extract the features of visible light pedestrian images and infrared pedestrian images respectively; (4) The identity loss (ID loss), weighted regularization triplet loss (Weighted regularizationTriplet loss) and visible-infrared shared loss (Visible-Infrared loss) are used jointly to constrain the model and achieve the purpose of improving the accuracy of cross-modal pedestrian re-identification.

2. The method according to claim 1, characterized in that In step (1), in the visible light image, a color channel is randomly selected to replace other color channels, which can be expressed as: The calculation formula of the visible light-infrared cross-modal objective function using CE is: in represents an infrared pedestrian image, Represents a three-channel RGB visible light image or its visible light image processed by CE, y i and j Indicates the label information of each picture, f v and f r denote the feature extraction networks for visible light and infrared images respectively, and l(.) denotes the optimization function, which can be identity loss, triplet loss or their variants.

3. The method according to claim 1, characterized in that In step (2), the output features of the fifth residual block conv_5x in ResNet50 are used as the input of the STN (Spatial Transformer Network) module, and a set of affine transformation parameters A is linearly regressed. θ , the corresponding coordinate relationship of the image before and after affine transformation is established by the following formula: in are the target coordinates in the regular network that aligns the images, are the source coordinates of the sampling points in the input image, is the affine transformation matrix, θ 11 ,θ 12 Controls the size of the transformed image, θ 21 and θ 22 Controls the rotation of the transformed image, θ 13 ,θ 23 Controls the offset of the transformed image; due to the affine transformation matrix contains continuously differentiable decimals, and the target coordinates is discrete, the source coordinates It is continuous and requires a certain sampling strategy to generate aligned images. In this method, bilinear sampling is used to obtain the value of a specific pixel in the transformed image. The formula is as follows: X pq represents the pixel value of the source image at (p,q), X′ mn Represents the pixel value of the transformed image at (m,n), and H and W represent the height and width of the source image (target image).

4. The method according to claim 1, characterized in that: In step (3), the CBR non-local attention includes four CBR components, which are composed of a 1×1 convolution layer, a batch normalization layer and a nonlinear activation function leaky relu; the input feature size of the CBR non-local attention is N×C×H×W, where N represents batch size, C represents the number of channels, and H and W represent the height and width of the feature respectively; assuming φ′(X j )=W φ X j , where W φ is the weight matrix to be learned, for example, using a 1×1 convolution in space; similarly, θ'(X j )=W θ X j , g'(X j )=W g X j ;φ(X j ),θ(X j ) and g(X j ) is defined as follows: θ(X j )=ε(σ(γ(W θ X j ))) (5) φ(X j )=ε(σ(γ(W φ X j ))) (6) g(X j )=ε(σ(γ(W g X j ))) (7) Where γ(.) represents the batch normalization function, σ(.) represents the activation function, ε(.) represents merging the H and W dimensions into HW, and the batch size and number of channels remain unchanged; CBR non-local attention can be expressed as: f(X i ,X j )=θ(X i ) T φ(X j ) (9) Where i represents the index of the output position of the response to be calculated, and j represents the index of all possible positions enumerated; f(X i ,X j ) is used to calculate the relationship between i and all possible associated positions j; g(X j ) calculates the representation of the input signal at position j; Formula (8) calculates the response based on the relationship between different positions; The formula for the CBR non-local block is: z i =W z y i +x i (10) Where "+x i ” indicates residual connection.

5. The method according to claim 1, characterized in that The step (4) combines the loss function to optimize the model in the following way: set up and They represent the input visible light image and infrared images The identity prediction probability is expressed as follows: set up and Respectively represent the visible light image input when identity is i and infrared images If the label information and For a dataset containing N pedestrians, P pedestrians are randomly selected. For each pedestrian, K visible light images and K infrared images of the pedestrian are randomly selected. The ID loss formula is as follows: Assumptions is the visible light anchor sample in the training batch, is the infrared positive sample with the same identity, is an infrared negative sample with different identities; weighted regularized triplet loss function Defined as Where p represents the positive sample set of anchor sample i, n is the negative sample set, and W i p and W i n Represent the distance weights of positive samples and negative samples respectively; Similarly, The definition is as follows: The overall weighted regularized triplet loss is: Assume that the anchor features of visible light images and infrared images in a training batch are and by As an example, we calculate the average distance between it and other positive samples within and between modalities: L vis Defined as: L vis Punishment D intra and D cross When the discriminant feature appears in only one mode, D intra and D cross The difference between them will increase, and this anomaly will be L vis Capture; in D intra and D cross In the bidirectional optimization process, unreliable features that only appear in one mode will be suppressed, and reliable features that appear in both modes will be enhanced; The total loss function is defined as follows: L total =L id +λ1L wrt +λ2L vis (24) where λ1 and λ2 are hyperparameters that balance the corresponding terms and are set to 1.

Images