A cross-modal pedestrian re-identification method based on image alignment and channel enhancement

Abstract

The application discloses a cross-modal pedestrian re-identification method based on image alignment and channel enhancement. Mainly includes: (1) using random channel exchange enhancement to generate color-independent pedestrian images; (2) using a spatial transformation network to adaptively align and adjust the visible light and infrared pedestrian images with misalignment; (3) using a ResNet50 network embedded with CBR non-local attention to extract features of the visible light pedestrian images and the infrared pedestrian images respectively; (4) jointly using identity loss, weighted regularization triplet loss and visible-infrared shared loss to constrain the model. The application is mainly applied to the field of pedestrian re-identification and has broad application prospects in the fields of video monitoring and image retrieval.

Description

Technical Field

[0001] This invention relates to a cross-modal pedestrian re-identification method based on image alignment and channel enhancement, belonging to the field of computer vision and intelligent information technology. Background Technology

[0002] Pedestrian re-identification (Re-ID) was first proposed by Gheissari et al. in 2006 for cross-camera pedestrian image retrieval, aiming to retrieve a given pedestrian query image from image libraries captured by different cameras. However, most current Re-ID algorithms focus on searching visible pedestrian images captured by visible light cameras, and cannot handle low-light and low-illumination night scenes. Wu et al. proposed visible light-infrared cross-modal pedestrian re-identification (VI-ReID) in 2017. Compared with traditional pedestrian re-identification research, VI-ReID is more challenging because it involves significant modal differences and unknown environmental factors (such as occlusion and background interference, lighting changes, camera viewpoint differences, etc.).

[0003] To reduce visible-infrared modal differences, generating cross-modal images using Generative Adversarial Networks (GANs) is a popular approach, mitigating modal differences by aligning pixels. However, image generation typically incurs additional computational costs, and such algorithms heavily rely on training samples. Furthermore, some studies assume color information is uncorrelated and use grayscale images for cross-modal matching; while these methods eliminate color differences, they lose discriminative information in the color channels. Another approach measures similarity by mapping visible and infrared pedestrian images to a common feature space, typically using a dual-channel framework consisting of two feature-specific networks for extraction and a parameter-sharing network for mapping the extracted features to the common feature space. This method effectively extracts modality-specific features and reduces modality differences, but struggles to extract modality-shared features. Additionally, pedestrian spatial misalignment exists due to factors such as camera viewpoint and pedestrian pose variations. To address these issues, this invention proposes a cross-modal pedestrian re-identification method based on image alignment and channel enhancement, improving the performance of cross-modal pedestrian re-identification. Summary of the Invention

[0004] This invention proposes a cross-modal pedestrian re-identification method based on image alignment and channel enhancement. The aim is to design a dual-channel network, use a spatial transformation module to perform adaptive image alignment processing, use the classic ResNet50 to extract features from visible light and infrared pedestrian images, and jointly use multiple loss functions to optimize the model, thereby improving the performance of cross-modal pedestrian re-identification.

[0005] The present invention achieves the above objectives through the following technical solutions:

[0006] (1) Use Random Channel Exchangeable Enhancement (CE) to generate color-independent pedestrian images and learn the relationship between each color channel of single-channel infrared images and visible light images;

[0007] (2) Use Spatial Transformer Network (STN) to adaptively align and adjust the misaligned visible light and infrared pedestrian images, and then perform pixel-level fusion between the aligned visible light and infrared pedestrian images and the original images respectively.

[0008] (3) Then, the features of visible light pedestrian images and infrared pedestrian images are extracted using a ResNet50 network with CBR non-local attention embedded.

[0009] (4) The model is constrained by using ID loss, weighted regularization triplet loss and visible-infrared shared loss together to improve the accuracy of cross-modal pedestrian re-identification. Attached Figure Description

[0010] Figure 1 This is a framework diagram of the cross-modal person re-identification method based on image alignment and channel enhancement of the present invention;

[0011] Figure 2 This is a structural diagram of the STN spatial transformation module of the present invention;

[0012] Figure 3 This is a diagram of the CBR non-local attention structure of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0014] (1) Overall network framework

[0015] This network model framework is trained in an end-to-end manner. For example... Figure 1As shown, random channel commutative enhancement is first used to improve robustness to color changes. For pedestrian feature extraction, a ResNet50 pre-trained on ImageNet is used as the backbone network. A spatial transformation network is used to transform the original visible light and infrared images, fusing the original and transformed images pixel by pixel. The fused visible light and infrared images are then fed into a ResNet50 network embedded with CBR non-local attention to obtain visible light features f respectively. vs and infrared features f ir The first block of ResNet50 is specific to both modalities to obtain modality-specific features, while the remaining blocks are shared to learn modality-shared features. A joint identity loss (IDloss), weighted regularization triplet loss, and visible-infrared shared loss are used to optimize the network model.

[0016] (2) Enhanced random channel switching

[0017] The main idea of ​​Random Channel Exchangeable Enhancement (CE) is to randomly select one color channel in a visible light image to replace other color channels, which can be represented as:

[0018]

[0019] The formula for calculating the visible-infrared transmodal objective function using CE is as follows:

[0020]

[0021] in Represents infrared pedestrian images. This represents a three-channel RGB visible light image or its visible light image after CE processing, y i and y j This represents the tag information for each image, f v and f rLet represent the feature extraction networks for visible light and infrared images, respectively. l(.) represents the optimization function, which can be identity loss, triplet loss, or variations thereof. Specifically, an integer is first randomly selected from [0,1,2,3] to determine whether to perform a CE operation or preserve the original RGB image. The random channel swapping enhancement operation can be easily combined with other data augmentations (random erasure, random cropping, etc.). This enhancement can be understood as uniformly generating a three-channel visible light image by decomposing the color channels, which helps model f learn the relationship between a single-channel infrared image and each color channel of the visible light image.

[0022] (3) Spatial Transformation Network

[0023] The output features of the fifth residual block conv_5x in ResNet50 are used as the input of the STN (Spatial Transformer Network) module to linearly regress a set of affine transformation parameters A. θ The coordinate relationship between the images before and after the affine transformation is established using the following formula:

[0024]

[0025] in These are the target coordinates in the rule network for aligning images. These are the source coordinates of the sampling points in the input image. It is the affine transformation matrix, θ 11 θ 12 Controls the size of the transformed image, θ 21 and θ 22 Controlling the rotation of the transformed image, θ 13 θ 23 Control the offset of the transformed image. Due to the affine transformation matrix It contains continuously differentiable decimals, and the target coordinates It is discrete, source coordinates It is continuous and requires a certain sampling strategy to generate an aligned image. This method uses bilinear sampling to obtain the value of a specific pixel in the transformed image, as shown in the following formula:

[0026]

[0027] X pq X′ represents the pixel value at (p,q) in the source image. mn H represents the pixel value at (m,n) of the transformed image, and H and W represent the height and width of the source image (target image).

[0028] (4) CBR non-local attention

[0029] CBR non-local attention comprises four CBR components, consisting of 1×1 convolutional layers, batch normalization layers, and a leaky ReLU non-linear activation function. The input feature size for CBR non-local attention is N×C×H×W, where N represents the batch size, C represents the number of channels, and H and W represent the height and width of the feature, respectively. Assume φ′(X j ) = W φ X j W φ This is the weight matrix to be learned, implemented, for example, using a 1×1 convolution in space. Similarly, θ'(X j ) = W θ X j , g'(X j ) = W g X j φ(X) j ), θ(X j ) and g(X j The definition is as follows:

[0030] θ(X j )=ε(σ(γ(W θ X j ))) (5)

[0031] φ(X j )=ε(σ(γ(W φ X j ))) (6)

[0032] g(X j )=ε(σ(γ(W g X j ))) (7)

[0033] Where γ(.) represents the batch normalization function, σ(.) represents the activation function, and ε(.) represents merging the H and W dimensions into HW, while keeping the batch size and number of channels unchanged. CBR non-local attention can be represented as:

[0034]

[0035] f(X i ,X j )=θ(X i ) T φ(X j (9)

[0036] Where i represents the index of the output position of the response to be calculated, and j represents the index of all possible positions enumerated. f(X) i ,X j) is used to calculate the relationship between i and all possible associated positions j. g(X) j ) Calculate the representation of the input signal at position j. Formula (8) calculates the response based on the relationship between different positions. The formula for the CBR non-local block is:

[0037] z i =W z y i +x i (10)

[0038] Among them, "+x" i " indicates a residual connection.

[0039] (5) Joint loss function

[0040] Loss of identity:

[0041] set up and These respectively represent the input visible light images. and infrared images The probability of identity prediction is expressed as follows:

[0042]

[0043]

[0044] set up and These represent the visible light images input when identity is i. and infrared images The tag information, then and For a dataset containing N pedestrians, P pedestrians are randomly selected. For each pedestrian, K visible light images and K infrared images of that pedestrian are randomly selected. The ID loss formula is as follows:

[0045]

[0046] Weighted regularized triplet loss:

[0047] Assumption These are visible light anchor samples from the training batch. These are positive infrared samples with the same identity. These are infrared negative samples with different identities. Weighted regularized triplet loss function. Defined as:

[0048]

[0049]

[0050]

[0051] Where p represents the set of positive samples for anchor sample i, and n is the set of negative samples. and These represent the distance weights for positive and negative samples, respectively. Similarly, The definition is as follows:

[0052]

[0053]

[0054]

[0055] The overall weighted regularized triplet loss is:

[0056]

[0057] Visible-infrared shared loss:

[0058] Suppose the anchor features of the visible light images and infrared images in a training batch are respectively and by For example, we calculate its average distance from other positive samples both within and between modes:

[0059]

[0060]

[0061] L vis Defined as:

[0062]

[0063] L vis Punishment D intra and D cross The difference between them. When the discriminant feature appears in only one modality, D intra and D cross The differences between them will increase, and this anomaly will be L vis Captured. In D intra and D cross In the bidirectional optimization process, unreliable features that appear only in one mode will be suppressed, while reliable features that appear in both modes will be enhanced.

[0064] The total loss function is defined as follows:

[0065] L total =L id +λ1L wrt +λ2Lvis (twenty four)

[0066] Where λ1 and λ2 are the hyperparameters that balance the corresponding terms, and are set to 1.

[0067] To verify the effectiveness of the cross-modal person re-identification method based on image alignment and channel enhancement described in this invention, experiments were conducted on the RegDB and SYSU-MM01 datasets. The experimental platform used in this paper was Ubuntu 20.04 with an Nvidia RTX 2080 GPU. The deep learning framework used was PyTorch 1.7, and the accuracy evaluation metrics were Rank-1, mAP (mean average precision), and mINP (mean inverse negative penalty). The experimental results of this invention are shown in Tables 1 and 2. The experimental accuracy is significantly improved compared to the benchmark network and other comparative algorithms.

[0068] Table 1 Comparison of results of different algorithms on the RegDB dataset

[0069]

[0070]

[0071] Table 2 Comparison of results of different algorithms on the SYSU-MMO1 dataset

[0072]

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-modal person re-identification method based on image alignment and channel enhancement, characterized in that... Includes the following steps: (1) Use Random Channel Exchangeable Enhancement (CE) to generate color-independent pedestrian images and learn the relationship between each color channel of single-channel infrared and visible light images; (2) Use Spatial Transformer Network (STN) to adaptively align and adjust the misaligned visible light and infrared pedestrian images, and then perform pixel-level fusion between the aligned visible light and infrared pedestrian images and the original images respectively. (3) Use a ResNet50 network with CBR non-local attention to extract features from visible light pedestrian images and infrared pedestrian images respectively; (4) The model is constrained by using ID loss, weighted regularization triplet loss and visible-infrared shared loss together to improve the accuracy of cross-modal pedestrian re-identification. The CBR non-local attention in step (3) includes four CBR components, which are... The system consists of convolutional layers, batch normalization layers, and the non-linear activation function leaky ReLU; the input features are handled by CBR non-local attention. The size is ,in Indicates batch size. Indicates the number of channels. and These represent the height and width of the feature, respectively. , and The definition is as follows: （1） （2） （3） in This represents the batch normalization function. This represents the activation function. Indicates will and Dimensions merged into The batch size and number of channels remain unchanged; CBR non-local attention can be represented as: （4） （5） in The index representing the output position of the response to be calculated. The index representing all possible positions in the enumeration; Used for calculation and all possible associated locations The relationship between them; calculate The input signal at the location is represented; Formula (8) calculates the response based on the relationship between different locations; The formula for the CBR non-local block is: （6） in" "Indicates residual connection; Step (4) optimizes the model using the joint loss function, as follows: set up and These respectively represent the input visible light images. and infrared images The probability of identity prediction is expressed as follows: （7） （8） in and They represent the identities respectively. Visible light image input at time and infrared images The tag information, then and For those containing A dataset of individual pedestrians, randomly selected For each pedestrian, the pedestrian is randomly selected. Zhang Visible Light Images and For an infrared image, the ID loss formula is as follows: （9） Assumption These are visible light anchor samples from the training batch. These are positive infrared samples with the same identity. These are infrared negative samples with different identities; Weighted Regularized Triple Loss Function Defined as （10） （11） （12） in Represents anchor point sample The positive sample set, It is a negative sample set. and These represent the distance weights for positive and negative samples, respectively. Similarly, The definition is as follows: （13） （14） （15） The overall weighted regularized triplet loss is: （16） Suppose the anchor features of the visible light images and infrared images in a training batch are respectively and ,by For example, we calculate its average distance from other positive samples both within and between modes: （17） （18） Defined as: （19） When the discriminant feature appears in only one modality and The differences between them will increase, and this anomaly will be Capture; in and In the bidirectional optimization process, unreliable features that appear only in one mode will be suppressed, while reliable features that appear in both modes will be enhanced. The total loss function is defined as follows: （20） in and This is a hyperparameter that balances the corresponding terms; it is set to 1.

2. The method according to claim 1, characterized in that, Step (1), which involves randomly selecting a color channel in the visible light image to replace other color channels, can be represented as follows: （21） The formula for calculating the visible-infrared transmodal objective function using CE is as follows: （22） in Represents infrared pedestrian images. This refers to a three-channel RGB visible light image or a visible light image processed by CE. and This represents the tag information for each image. and These represent feature extraction networks for visible light and infrared images, respectively. The function represents the optimization function, which can be identity loss, triplet loss, or a variation thereof.

3. The method according to claim 1, characterized in that, Step (2) will convert the fifth residual block in ResNet50. The output features are used as input to the STN (Spatial Transformer Network) module to linearly regress a set of affine transformation parameters. The coordinate relationship between the images before and after the affine transformation is established using the following formula: （23） in It is an affine transformation matrix. It is an affine matrix The defined space transformation function, These are the target coordinates in the rule network for aligning images. These are the source coordinates of the sampling points in the input image. , Control the size of the transformed image. and Controlling the rotation of the transformed image, , Controlling the offset of the transformed image; due to the affine transformation matrix It contains continuously differentiable decimals, and the target coordinates It is discrete, source coordinates Since the image is continuous, a specific sampling strategy is needed to generate an aligned image. This method uses bilinear sampling to obtain the value of a specific pixel in the transformed image, as shown in the following formula: （24） Indicates the source image in Pixel value at that location, Indicates the transformed image in Pixel value at that location, and This represents the height and width of the source and target images.

Images