A cross-modal pedestrian re-identification method based on multi-dimensional information learning

Abstract

The application relates to a cross-modal pedestrian re-identification method based on multi-dimensional information learning, which considers visible light pedestrian images and infrared pedestrian images, introduces a multi-dimensional information mining module MIM for extracting global information and channel information, constructs a classification network to be trained for training, and obtains a cross-modal pedestrian re-identification test network; in application, different modes corresponding to visible light and infrared of the to-be-detected image and the candidate image library set are taken as inputs, pedestrian re-identification of the to-be-detected image is realized, random local cutting enhancement processing is introduced in training, view point changes and pedestrian body changes are effectively simulated, the robustness of the model to pedestrian postures and view point changes is improved, and the features obtained by the feature extractor are subjected to loss function constraint, periodic cyclic focus loss is applied to constrain different attention degrees of the model to difficult and easy samples in the training process, the model performance is more effectively improved, and better effects are realized on a cumulative matching feature curve and an average precision mean.

Description

Technical Field

[0001] This invention relates to a cross-modal person re-identification method based on multi-dimensional information learning, belonging to the field of computer vision technology. Background Technology

[0002] The goal of pedestrian re-identification is to match an image of a given person with images captured by other cameras. Pedestrian re-identification has been widely applied in fields such as intelligent video surveillance, and it has significant practical implications for mitigating crime and maintaining social security. Currently, research mainly focuses on the retrieval and matching of pedestrian images captured solely by visible light cameras. However, criminals tend to operate at night, in which case visible light cameras cannot capture pedestrian images with valid identification information, resulting in poor re-identification performance. With the development of intelligent devices, dual-modal cameras have been widely used in the public domain. Dual-modal cameras can automatically switch from visible light mode to infrared mode to capture valid pedestrian identification information. The visible-infrared cross-modal pedestrian re-identification task has received widespread attention due to its extensive application in recent years.

[0003] Visible-infrared cross-modal person re-identification aims to match the same person captured by different infrared and visible cameras, which is more challenging than the single-modal person re-identification problem. Visible-infrared cross-modal person re-identification not only faces intra-modal differences such as viewpoint changes, pose changes, and occlusion, but also significant inter-modal differences.

[0004] Current research mainly falls into two categories: feature-level and image-level. Feature-level methods attempt to project features from both modalities into a common space and design a loss function to minimize their differences, thereby improving model accuracy. However, due to the significant differences between the two modalities, learning an effective common space presents a considerable challenge. Image-level methods utilize generative adversarial networks (GANs) to generate information corresponding to a modality missing in another modality, enhancing modal-shared features and thus improving pedestrian re-identification accuracy. However, the generated images may introduce noise or lose detailed information, affecting subsequent recognition performance. Since visible light and infrared images are typically captured at different times, the angle and pose of pedestrian images vary. The aforementioned methods primarily reduce the differences between the two modalities but do not address the issue of angle and pose variations. Furthermore, in cross-modal pedestrian re-identification tasks using convolutional neural networks (CNNs) as feature extractors, CNNs capture local features of the input data, lacking the ability to capture global contextual information. Therefore, designing feature extractors to obtain global and channel information is crucial. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a cross-modal pedestrian re-identification method based on multi-dimensional information learning, which can solve the problems of intra-modal differences in visible-infrared cross-modal pedestrian re-identification and the shortcomings of feature extractors in capturing global and channel information, and effectively improve the accuracy of pedestrian re-identification.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: The present invention designs a cross-modal pedestrian re-identification method based on multi-dimensional information learning. Based on a candidate image library set composed of a preset number of visible light reference images and infrared reference images corresponding to each pedestrian label, the present invention executes the following steps A to C to obtain a cross-modal pedestrian re-identification test network, and then executes step i to realize pedestrian re-identification for the image to be detected containing pedestrians.

[0007] Step A. Collect a preset number of visible light pedestrian sample images and infrared pedestrian sample images corresponding to each pedestrian tag to form a pedestrian sample image set, and proceed to Step B;

[0008] Step B. Based on a preset classification network that includes two identical classification sub-networks and each classification sub-network contains at least one residual network in its feature extraction module, add a preset number of multi-dimensional information mining modules (MIMs) for extracting global and channel information to construct the classification network to be trained, and then proceed to step C.

[0009] Step C. Based on the pedestrian sample image set, with the subset of visible light pedestrian sample images and infrared reference images in the candidate image library as input, and the similarity between the pedestrian feature maps in the visible light pedestrian sample images and the pedestrian feature maps in each infrared reference image as output, the pedestrian labels corresponding to the pedestrians in the visible light pedestrian sample images are output; and with the subset of infrared pedestrian sample images and visible light reference images in the candidate image library as input, and the similarity between the pedestrian feature maps in the infrared pedestrian sample images and the pedestrian feature maps in each visible light reference image as output, the pedestrian labels corresponding to the pedestrians in the infrared pedestrian sample images are output, the classification network to be trained is trained to obtain the cross-modal pedestrian re-identification test network;

[0010] Step i. If the image to be detected is an infrared image, select the image to be detected and a subset of visible light reference images in the candidate image library as input, apply the cross-modal pedestrian re-identification test network to obtain the pedestrian labels corresponding to the pedestrians in the image to be detected, and realize pedestrian re-identification of the image to be detected.

[0011] If the image to be detected is an infrared image, then the subset consisting of the image to be detected and the visible light reference images in the candidate image library is selected as input. The cross-modal pedestrian re-identification test network is applied to obtain the pedestrian labels corresponding to the pedestrians in the image to be detected, and pedestrian re-identification is achieved for the image to be detected.

[0012] As a preferred technical solution of the present invention: In step B, each classification sub-network in the preset classification network includes, from the input end to the output end, a front-end convolutional layer Conv, a feature extraction module, and a generalized average pooling layer GEM connected in series; the number of multi-dimensional information mining modules (MIM) is equal to the number of all residual networks in the preset classification network, and each MIM corresponds one-to-one with each residual network.

[0013] The inputs of each Multidimensional Information Mining (MIM) module are connected to the outputs of their corresponding residual networks, forming feature extraction sub-blocks. The inputs of the residual networks constitute the inputs of the feature extraction sub-blocks, and the outputs of the MIM modules constitute the outputs of the feature extraction sub-blocks. The feature extraction sub-blocks corresponding to each residual network in the feature extraction module are sequentially connected from the input to the output. The input of the first feature extraction sub-block in sequence constitutes the input of the feature extraction module, and the output of the last feature extraction sub-block in sequence constitutes the output of the feature extraction module. This updates the feature extraction module, which in turn updates each classification sub-network. The inputs of two classification sub-networks constitute the two inputs of the classification network to be trained, and the outputs of the two classification sub-networks are connected to the two inputs of the classifier. The output of the classifier constitutes the output of the classification network to be trained, thus forming the classification network to be trained.

[0014] As a preferred embodiment of the present invention: each of the multi-dimensional information mining (MIM) modules includes a first convolutional layer (Conv1), a second convolutional layer (Conv2), a third convolutional layer (Conv3), a fourth convolutional layer (Conv4), a fifth convolutional layer (Conv5), a sixth convolutional layer (Conv6), a first multiplication module, a second multiplication module, a first addition module, an average pooling layer (AvgPool), a ReLU activation function, a sigmoid activation function, and a weight layer (Reweight). In the MIM structure, the input terminals of the first convolutional layer (Conv1), the second convolutional layer (Conv2), the third convolutional layer (Conv3), the average pooling layer (AvgPool), one of the input terminals of the first addition module, and one of the input terminals of the weight layer (Reweight) are connected together, and the connection terminals are... The input terminals of the Multidimensional Information Mining (MIM) module are formed. The output terminals of the second convolutional layer Conv2 and the third convolutional layer Conv3 are connected to the two input terminals of the first multiplication module. The output terminals of the first convolutional layer Conv1 and the first multiplication module are connected to the two input terminals of the second multiplication module. The output terminal of the second multiplication module is connected to the input terminal of the fourth convolutional layer Conv4. The output terminal of the average pooling layer AvgPool is connected in series with the fifth convolutional layer Conv5, the ReLU activation function, the sixth convolutional layer Conv6, and the sigmoid activation function, and then connected to the other input terminal of the weight layer Reweight. The output terminals of the fourth convolutional layer Conv4 and the weight layer Reweight are respectively connected to the other two input terminals of the first addition module. The output terminal of the first addition module constitutes the output terminal of the Multidimensional Information Mining (MIM) module.

[0015] As a preferred technical solution of the present invention: each multi-dimensional information mining module performs the following steps a to f for the output feature map of the residual network it is connected to, thereby updating the output feature map of the residual network it is connected to;

[0016] Step a. Output the feature x with pixel index l in the feature map x of the connected residual network respectively. l and the feature x at all pixel positions in the feature map x. g Process it according to the following formula:

[0017]

[0018] Where, x l W represents the feature at the l-th pixel position in feature map x. g W θ W ψ Let g(x) represent the weight matrices used for training in the first convolutional layer Conv1, the second convolutional layer Conv2, and the third convolutional layer Conv3, respectively. g), θ(x) l ), ψ(x) g The numbers ) represent the outputs of the first convolutional layer Conv1, the second convolutional layer Conv2, and the third convolutional layer Conv3, respectively.

[0019] Step b. Output the feature x with pixel index l in the feature map x of the connected residual network respectively. l According to the following formula:

[0020] F(x l ,x g )=θ(x l ) T ψ(x g )

[0021] Calculate the feature x of the pixel position l Compare each pixel position in the feature map x with the feature x of each other pixel position. g The correlation between F(x) l ,x g );

[0022] Step c. Output the feature x at each pixel position in the feature map x of the connected residual network. l The feature x is obtained by traversing the residual network and outputting the feature map x for all pixel positions. g According to the following formula:

[0023]

[0024] Calculate the nonlocal features y corresponding to each pixel position in the feature map x. l , where N represents the number of pixels in the output feature map of the connected residual network;

[0025] Step d. Iterate through the pixel positions in the output feature map x of the connected residual network, and denote X = {x} l The nonlocal feature representation of |l=1,2,...,N} is Y={y l |l=1,2,...,N}, according to the following formula:

[0026] G = W y Y+X

[0027] The feature x of the pixel position l Its corresponding nonlocal feature y l Perform linear addition to obtain the global feature G, where W y This represents the weight matrix used for training in the fourth convolutional layer, Conv4.

[0028] Step e. Follow the formula below:

[0029] C = sigmoid(W) h ReLu(W u x))x

[0030] The channel feature C is calculated, where W u W represents the weight matrix used for training in the fifth convolutional layer, Conv5. h This represents the weight matrix used for training in the sixth convolutional layer, Conv6.

[0031] Step f. According to the following formula:

[0032] x f =G+C

[0033] The update x of the output feature map x of the connected residual network is calculated. f .

[0034] As a preferred technical solution of the present invention: in step C, the weighted regularized triplet loss L between the outputs of the generalized average pooling layer (GEM) in the two classification subnetworks of the classification network to be trained is used. WRT To constrain the training process of the classification network to be trained; weighted regularized triplet loss L WRT as follows:

[0035]

[0036]

[0037] Where (i,j,k) represents the triplet in each training batch of each anchor sample, j and k correspond to different modes in the visible light reference image and the infrared reference image, respectively, or j and k correspond to the same mode in the visible light reference image and the infrared reference image, respectively; P i It is the set of positive samples corresponding to the anchor point, N i The anchor point corresponds to the negative sample set. This represents the Euclidean distance between the anchor point and its corresponding positive sample pair. This represents the Euclidean distance between the anchor point and its corresponding negative sample pair.

[0038] As a preferred technical solution of the present invention: the output of the generalized average pooling layer (GEM) in the two-class sub-network of the classification network to be trained is further connected to the input of the batch normalization layer (BN), and the periodic smoothing focusing loss L of the output of each batch normalization layer (BN) is used. CSF Combined with weighted regularized triplet loss L WRT To constrain the training process of the classification network to be trained; periodic smoothing focusing loss L CSF as follows:

[0039] L CSF=ηL hc +(1-η)L fc

[0040] Among them, L hc For easy sample loss term, L fc The loss term for hard samples, where η is a periodic smoothing factor, is expressed as follows:

[0041]

[0042] Among them, e i e represents the current round number. n For the total number of training rounds, f c It is a periodic factor;

[0043] Easy Sample Loss Term L hc :

[0044]

[0045] Hard sample loss term L fc :

[0046]

[0047] Wherein, given a label y i Input image x i x i Recognized as class y i The predicted probability is encoded using the softmax function, given by p(y i |x i ) indicates that C is the number of pedestrian labels, and α and β are training hyperparameters.

[0048] As a preferred technical solution of the present invention: In step A, for each of the collected visible light pedestrian sample images and each of the infrared pedestrian sample images, random local misalignment enhancement processing is performed according to the following steps I to II, the pedestrian sample images are updated, and then a pedestrian sample image set is formed.

[0049] Step I. Using a random area with an aspect ratio r s In the pedestrian sample image, a random local region is defined at a random location, and is defined according to the following formula:

[0050]

[0051] Obtain the length h of the random local region s and height w s , where s s It is the area of ​​a random local region, s s=a·S, where a is the proportion of the random local region to the area of ​​the original pedestrian sample image, and S is the total number of pixels in the original pedestrian sample image;

[0052] Step II. For each pixel position (m, n) in the random local region, apply the following formula:

[0053]

[0054] The pixel positions (m′, n′) after random local misslicing are obtained, and then random local misslicing enhancement processing is performed on the pedestrian sample image, where c s Let d = c be the randomly staggered radians. s ×h s It is the offset of the shearing.

[0055] The cross-modal person re-identification method based on multi-dimensional information learning described in this invention has the following technical advantages compared with existing technologies:

[0056] (1) The present invention designs a cross-modal pedestrian re-identification method based on multi-dimensional information learning. It considers visible light pedestrian images and infrared pedestrian images, introduces a multi-dimensional information mining module (MIM) for extracting global information and channel information, constructs a classification network to be trained, and obtains a cross-modal pedestrian re-identification test network. This enhances the expressive ability of feature learning and solves the problem of local perception in convolutional neural networks. In the application, the image to be detected and the candidate image library set correspond to different modes of visible light and infrared, respectively, as inputs to realize pedestrian re-identification of the image to be detected. Random local miscutting enhancement processing is introduced in the training to effectively simulate viewpoint changes and pedestrian limb changes, thereby improving the robustness of the model to pedestrian posture and viewpoint changes. Furthermore, by constraining the loss function of the features obtained by the feature extractor, the periodic cyclic focusing loss is applied to constrain the different attention of the model to easy and difficult samples during the training process. This improves the limitation of the previous identity loss that fixed the attention of all samples, and more effectively improves the model performance, especially when dealing with difficult-classified samples. This makes the scheme achieve better results in terms of cumulative matching feature curve and average precision. Attached Figure Description

[0057] Figure 1 This is a flowchart of the cross-modal person re-identification method based on multi-dimensional information learning designed in this invention;

[0058] Figure 2 This is the overall architecture of the cross-modal pedestrian re-identification network in the embodiment;

[0059] Figure 3 The flowchart shows the algorithm for random local shear enhancement. Detailed Implementation

[0060] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0061] This invention presents a cross-modal pedestrian re-identification method based on multi-dimensional information learning. It utilizes a candidate image library composed of a preset number of visible light reference images and infrared reference images corresponding to each pedestrian label, such as... Figure 1 As shown, perform steps A to C to obtain the cross-modal pedestrian re-identification test network.

[0062] Step A. Collect a preset number of visible light pedestrian sample images and infrared pedestrian sample images corresponding to each pedestrian tag to form a pedestrian sample image set, and then proceed to Step B.

[0063] In practical applications, preprocessing operations can be designed to be introduced for step A above. These operations could include resizing the input sample image to 3×288×144, randomly cropping and horizontally flipping the sample image to achieve initial preprocessing. Then, further processing can be performed on each of the acquired visible light pedestrian sample images and each of the infrared pedestrian sample images. Figure 3 As shown, perform steps I to II as follows: perform random local misalignment enhancement processing, update each pedestrian sample image, and then construct a pedestrian sample image set.

[0064] Step I. Using a random area with an aspect ratio r s In the pedestrian sample image, a random local region is defined at a random location, and is defined according to the following formula:

[0065]

[0066] Obtain the length h of the random local region s and height w s , where s s It is the area of ​​a random local region, s s = a·S, where a is the proportion of the random local region to the area of ​​the original pedestrian sample image, and S is the total number of pixels in the original pedestrian sample image.

[0067] Step II. For each pixel position (m, n) in the random local region, apply the following formula:

[0068]

[0069] The pixel positions (m′, n′) after random local misslicing are obtained, and then random local misslicing enhancement processing is performed on the pedestrian sample image, where c s Let d = c be the randomly staggered radians. s ×h s It is the offset of the shearing.

[0070] The introduction of the aforementioned random local shearing enhancement process goes beyond simply transforming the entire image; it performs shearing operations on local areas of the image. This effectively simulates changes in pedestrian limbs and viewpoints, allowing the model to better capture key information in pedestrian images and improve recognition accuracy for pedestrian images with different postures and perspectives. The random local shearing enhancement method can also effectively expand datasets, enrich the diversity of model learning, and enhance the model's robustness and generalization ability.

[0071] Step B. Based on a preset classification network that includes two identical classification sub-networks and each classification sub-network contains at least one residual network in its feature extraction module, add a preset number of multi-dimensional information mining modules (MIMs) for extracting global and channel information to construct the classification network to be trained, and then proceed to step C.

[0072] Step C. Based on the pedestrian sample image set, with the subset of visible light pedestrian sample images and infrared reference images in the candidate image library as input, and the similarity between the pedestrian feature maps in the visible light pedestrian sample images and the pedestrian feature maps in each infrared reference image as output, the pedestrian labels corresponding to the pedestrians in the visible light pedestrian sample images are output; and with the subset of infrared pedestrian sample images and visible light reference images in the candidate image library as input, and the similarity between the pedestrian feature maps in the infrared pedestrian sample images and the pedestrian feature maps in each visible light reference image as output, the pedestrian labels corresponding to the pedestrians in the infrared pedestrian sample images are output, the classification network to be trained is trained to obtain the cross-modal pedestrian re-identification test network.

[0073] Based on the execution of steps A to C above, a cross-modal pedestrian re-identification test network is obtained. Then, step i is executed to perform pedestrian re-identification on the image to be detected containing pedestrians.

[0074] Step i. If the image to be detected is an infrared image, select the image to be detected and a subset of visible light reference images in the candidate image library as input, apply the cross-modal pedestrian re-identification test network to obtain the pedestrian labels corresponding to the pedestrians in the image to be detected, and realize pedestrian re-identification of the image to be detected.

[0075] If the image to be detected is an infrared image, then the subset consisting of the image to be detected and the visible light reference images in the candidate image library is selected as input. The cross-modal pedestrian re-identification test network is applied to obtain the pedestrian labels corresponding to the pedestrians in the image to be detected, and pedestrian re-identification is achieved for the image to be detected.

[0076] In the actual implementation of the above technical solution, in step B above, such as Figure 2As shown, each classification sub-network in the preset classification network includes, from the input end to the output end, a front-end convolutional layer (Conv), a feature extraction module, and a generalized average pooling layer (GEM) connected in series. The number of multi-dimensional information mining modules (MIM) is equal to the number of all residual networks in the preset classification network, and each MIM corresponds one-to-one with each residual network.

[0077] The inputs of each Multidimensional Information Mining (MIM) module are connected to the outputs of their corresponding residual networks, forming feature extraction sub-blocks. The inputs of the residual networks constitute the inputs of the feature extraction sub-blocks, and the outputs of the MIM modules constitute the outputs of the feature extraction sub-blocks. The feature extraction sub-blocks corresponding to each residual network in the feature extraction module are sequentially connected from the input to the output. The input of the first feature extraction sub-block in sequence constitutes the input of the feature extraction module, and the output of the last feature extraction sub-block in sequence constitutes the output of the feature extraction module. This updates the feature extraction module, which in turn updates each classification sub-network. The inputs of two classification sub-networks constitute the two inputs of the classification network to be trained, and the outputs of the two classification sub-networks are connected to the two inputs of the classifier. The output of the classifier constitutes the output of the classification network to be trained, thus forming the classification network to be trained.

[0078] For the Multi-Dimensional Information Mining (MIM) module, in practical applications, such as Figure 2As shown, each Multi-Dimensional Information Mining (MIM) module in the specific design includes a first convolutional layer (Conv1), a second convolutional layer (Conv2), a third convolutional layer (Conv3), a fourth convolutional layer (Conv4), a fifth convolutional layer (Conv5), a sixth convolutional layer (Conv6), a first multiplication module, a second multiplication module, a first addition module, an average pooling layer (AvgPool), a ReLU activation function, a sigmoid activation function, and a weight layer (Reweight). In the MIM structure, the inputs of the first convolutional layer (Conv1), the second convolutional layer (Conv2), the third convolutional layer (Conv3), the average pooling layer (AvgPool), one input of the first addition module, and one input of the weight layer (Reweight) are connected, and these connections constitute the multi-dimensional... The input terminals of the information mining module MIM are as follows: the output terminals of the second convolutional layer Conv2 and the third convolutional layer Conv3 are connected to the two input terminals of the first multiplication module; the output terminals of the first convolutional layer Conv1 and the first multiplication module are connected to the two input terminals of the second multiplication module; the output terminal of the second multiplication module is connected to the input terminal of the fourth convolutional layer Conv4; the output terminal of the average pooling layer AvgPool is connected in series with the fifth convolutional layer Conv5, the ReLU activation function, the sixth convolutional layer Conv6, and the sigmoid activation function, and then connected to the other input terminal of the weight layer Reweight; the output terminals of the fourth convolutional layer Conv4 and the weight layer Reweight are respectively connected to the other two input terminals of the first addition module; the output terminals of the first addition module constitute the output terminals of the multidimensional information mining module MIM.

[0079] Corresponding to the above-mentioned multi-dimensional information mining module (MIM) structure, each multi-dimensional information mining module performs the following steps a to g on the output feature map of its connected residual network to update the output feature map of its connected residual network.

[0080] Step a. Output the feature x at each pixel position in the feature map x of the connected residual network. l and the feature x at all pixel positions in the feature map x. g Process it according to the following formula:

[0081]

[0082] Where, x l W represents the feature at the l-th pixel position in feature map x. g W θ W ψ Let g(x) represent the weight matrices used for training in the first convolutional layer Conv1, the second convolutional layer Conv2, and the third convolutional layer Conv3, respectively. g), θ(x) l ), ψ(x) g ) represent the outputs of the first convolutional layer Conv1, the second convolutional layer Conv2, and the third convolutional layer Conv3, respectively.

[0083] Step b. Output the feature x at each pixel position in the feature map x of the connected residual network. l According to the following formula:

[0084] F(x l ,x g )=θ(x l ) T ψ(x g )

[0085] Calculate the feature x of the pixel position l Compare each pixel position in the feature map x with the feature x of each other pixel position. g The correlation between F(x) l ,x g ).

[0086] Step c. Output the feature x at each pixel position in the feature map x of the connected residual network. l The feature x is obtained by traversing the residual network and outputting the feature map x for all pixel positions. g According to the following formula:

[0087]

[0088] Calculate the nonlocal features y corresponding to each pixel position in the feature map x. l , where N represents the number of pixels in the output feature map of the connected residual network.

[0089] Step d. Iterate through the pixel positions in the output feature map x of the connected residual network, and denote X = {x} l The nonlocal feature representation of |l=1,2,...,N} is Y={y l |l=1,2,...,N}, according to the following formula:

[0090] G = W y Y+X

[0091] The feature x of the pixel position l Its corresponding nonlocal feature y l Perform linear addition to obtain the global feature G, where W y This represents the weight matrix used for training in the fourth convolutional layer, Conv4.

[0092] In the other branch, x first obtains the average value of the input features through an average pooling layer (AvgPool), and then reduces the number of channels in the feature map through a 1×1 fifth convolutional layer (Con5). Next, a non-linear transformation is introduced using the ReLU activation function to make the model focus more on useful information and suppress useless information. Finally, channel weights are generated through a sixth convolutional layer (Conv6) and a sigmoid activation function to improve the model's expressive power; specifically, step e is performed as follows.

[0093] Step e. Follow the formula below:

[0094] C = sigmoid(W) h ReLu(W u x))x

[0095] The channel feature C is calculated, where W u W represents the weight matrix used for training in the fifth convolutional layer, Conv5. h This represents the weight matrix used for training in the sixth convolutional layer, Conv6.

[0096] Step f. According to the following formula:

[0097] x f =G+C

[0098] The update x of the output feature map x of the connected residual network is calculated. f .

[0099] The addition of the multi-dimensional information mining module, consisting of global and channel branches, results in features that capture not only the local information obtained by the convolutional neural network but also global and channel information, enabling more comprehensive feature learning. By combining local, global, and channel information, the network can better capture features of different dimensions in the image and the relationships between them, thereby improving its generalization ability to pedestrian features. By integrating various information layers, the network can more accurately understand pedestrian image features, thus improving the accuracy of cross-modal pedestrian re-identification.

[0100] Furthermore, in practical training applications for the classification network to be trained, in step C above, the weighted regularized triplet loss L between the outputs of the generalized average pooling layer (GEM) in the two classification sub-networks of the classification network to be trained is used. WRT To constrain the training process of the classification network to be trained; weighted regularized triplet loss L WRT as follows:

[0101]

[0102]

[0103] Where (i,j,k) represents the triplet in each training batch of each anchor sample, j and k correspond to different modes in the visible light reference image and the infrared reference image, respectively, or j and k correspond to the same mode in the visible light reference image and the infrared reference image, respectively; P i It is the set of positive samples corresponding to the anchor point, N i The anchor point corresponds to the negative sample set. This represents the Euclidean distance between the anchor point and its corresponding positive sample pair. This represents the Euclidean distance between the anchor point and its corresponding negative sample pair.

[0104] In practical design, not only is the weighted regularized triplet loss L applied, but also... WRT Furthermore, a periodic smoothing focusing loss L was incorporated into the design. CSF Specifically, in the training of the classification network to be trained, the output of the generalized average pooling layer (GEM) in the two classification sub-networks of the network to be trained is further connected to the input of the batch normalization layer (BN), and the periodic smoothing focusing loss L is used to smooth the output of each batch normalization layer (BN). CSF Combined with weighted regularized triplet loss L WRT To constrain the training process of the classification network to be trained; periodic smoothing focusing loss L CSF as follows:

[0105] L CSF =ηL hc +(1-η)L fc

[0106] Among them, L hc For easy sample loss term, L fc The loss term for hard samples, where η is a periodic smoothing factor, is expressed as follows:

[0107]

[0108] Among them, e i e represents the current round number. n For the total number of training rounds, f c It is a periodic factor.

[0109] Easy Sample Loss Term L hc :

[0110]

[0111] Hard sample loss term L fc :

[0112]

[0113] Wherein, given a label y i Input image xi x i Recognized as class y i The predicted probability is encoded using the softmax function, given by p(y i |x i ) indicates that C is the number of pedestrian labels, and α and β are training hyperparameters.

[0114] The primary function of periodic smoothing focus loss is to constrain the model's attention to different difficulty levels of samples during training. Specifically, it adjusts the model's focus on samples, making it pay more attention to easily classified samples in the early and late stages of training, and more attention to difficult-to-classify samples in the middle stages. This helps balance model weights and avoids the indiscriminate treatment of all samples seen with cross-entropy loss. By introducing periodic smoothing focus loss, the model can be guided to more effectively learn the differences between samples, increasing its attention to difficult-to-classify samples, thereby improving the model's generalization ability and performance. During training, this loss function helps the model better adapt to the feature differences between different categories, thus improving the model's performance on the entire dataset.

[0115] The above design scheme was applied to practical implementations, such as using the SYSU-MM01 dataset for visible-infrared cross-modal pedestrian re-identification. The SYSU-MM01 dataset consists of pedestrian images captured by four visible light cameras and two infrared cameras, containing 491 pedestrian images with different identities, including both indoor and outdoor scenes. 395 pedestrian images with different identities were used for model training, comprising 22,285 visible light images and 11,909 infrared images. The remaining 96 pedestrian images with different identities were used for the model testing phase, comprising 3,803 infrared images as the query set and 301 visible light images as the image library. The Cumulative Matching Feature (CMC) curve and mean average accuracy (mAP) were used as evaluation metrics for model performance.

[0116] In this embodiment, our model is implemented using the PyTorch library and trained on an NVIDIA 3090 GPU. We use a ResNet50 pre-trained on ImageNet as our feature extraction backbone, and all input images are resized to 3×288×144 through random cropping. For optimization, we use a stochastic gradient descent (SGD) optimizer for training, with an initial learning rate set to 0.1, which decays by a factor of 0.1 and 0.01 at 20 and 50 epochs, respectively. A warm-up strategy is used in the first 10 epochs, and the total number of training epochs is 80. In each mini-batch, we randomly select 4 visible light and 4 infrared images from 8 identities for training.

[0117] This embodiment selected 10 methods for comparison to evaluate the effectiveness of the present invention in visible-infrared cross-modal pedestrian re-identification. These methods can be divided into two categories: one is feature-level models, such as Zero-Padding, AGW, SPOT, MIMFE, PMT, GLSFF, and MSIF; the other is image-level models, such as AlignGAN and GECNet. CMC and mAP were used as quantitative indicators to evaluate the various models, and the quantitative evaluation results are shown in Table 1.

[0118] Table 1

[0119]

[0120] Note: The last line in bold indicates optimal performance.

[0121] The following conclusions can be drawn from Table 1:

[0122] (1) The Rank-1 accuracy and mAP values ​​of the present invention are superior to other re-identification methods on the SYSU-MM01 dataset, which shows the effectiveness and superiority of the present invention in the visible light-infrared cross-modal pedestrian re-identification task.

[0123] (2) Specifically, for the full search mode, this invention achieved a Rank-1 accuracy of 73.11% and an mAP of 69.69%. The Rank-1 accuracy surpassed the state-of-the-art (SOTA) model by 6.25%, and the mAP surpassed the SOTA model by 3.98%. For the indoor search mode, this invention achieved a Rank-1 accuracy of 79.23% and an mAP of 82.59%. The Rank-1 accuracy surpassed the SOTA model by 6.04%, and the mAP surpassed the SOTA model by 4.05%. This indicates that using random local cropping enhancement can change the perspective and pose of pedestrian images, enabling global and channel information mining, extracting richer pedestrian features, and improving the performance of VI-ReID.

[0124] This invention presents a cross-modal pedestrian re-identification method based on multi-dimensional information learning. Considering visible light pedestrian images and infrared pedestrian images, it introduces a multi-dimensional information mining module (MIM) for extracting global and channel information. A training network is constructed to obtain a cross-modal pedestrian re-identification test network, enhancing the expressive power of feature learning and solving the problem of local perceptualism inherent in convolutional neural networks. In application, the image to be detected and a candidate image library corresponding to different modalities of visible light and infrared light are used as inputs to achieve pedestrian re-identification of the image to be detected. This method achieves better results in terms of cumulative matching feature curves and average accuracy.

[0125] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A cross-modal person re-identification method based on multi-dimensional information learning, characterized in that: Based on a candidate image library set consisting of a preset number of visible light reference images and infrared reference images corresponding to each pedestrian label, the following steps A to C are performed to obtain a cross-modal pedestrian re-identification test network. Then, step i is performed to achieve pedestrian re-identification for the image to be detected containing pedestrians. Step A. Collect a preset number of visible light pedestrian sample images and infrared pedestrian sample images corresponding to each pedestrian tag to form a pedestrian sample image set, and proceed to Step B; Step B. Based on a preset classification network that includes two identical classification sub-networks and each classification sub-network contains at least one residual network in its feature extraction module, add a preset number of multi-dimensional information mining modules (MIMs) for extracting global and channel information to construct the classification network to be trained, and then proceed to step C. Step C. Based on the pedestrian sample image set, with the subset of visible light pedestrian sample images and infrared reference images in the candidate image library as input, and the similarity between the pedestrian feature maps in the visible light pedestrian sample images and the pedestrian feature maps in each infrared reference image as output, the pedestrian labels corresponding to the pedestrians in the visible light pedestrian sample images are output; and with the subset of infrared pedestrian sample images and visible light reference images in the candidate image library as input, and the similarity between the pedestrian feature maps in the infrared pedestrian sample images and the pedestrian feature maps in each visible light reference image as output, the pedestrian labels corresponding to the pedestrians in the infrared pedestrian sample images are output, the classification network to be trained is trained to obtain the cross-modal pedestrian re-identification test network; Step i. If the image to be detected is an infrared image, select the image to be detected and a subset of visible light reference images in the candidate image library as input, apply the cross-modal pedestrian re-identification test network to obtain the pedestrian labels corresponding to the pedestrians in the image to be detected, and realize pedestrian re-identification of the image to be detected. If the image to be detected is an infrared image, then the subset consisting of the image to be detected and the visible light reference images in the candidate image library is selected as input, and the cross-modal pedestrian re-identification test network is applied to obtain the pedestrian labels corresponding to the pedestrians in the image to be detected, and pedestrian re-identification is achieved in the image to be detected. In the classification network to be trained, each multi-dimensional information mining module performs the following steps a to f for the output feature map of its connected residual network, thereby updating the output feature map of its connected residual network. Step a. Output feature maps for each connected residual network. Features of each pixel location and feature maps Features of all pixel locations Process it according to the following formula: ； in, Representation of feature map The Middle Features at each pixel location , , These represent the weight matrices used for training in the first convolutional layer Conv1, the second convolutional layer Conv2, and the third convolutional layer Conv3, respectively. These represent the outputs of the first convolutional layer Conv1, the second convolutional layer Conv2, and the third convolutional layer Conv3, respectively. Step b. Output feature maps for each connected residual network. Features of each pixel location According to the following formula: ； Calculate the features of pixel location respectively with feature map Features of other pixel locations Correlation between ; Step c. Output feature maps for each connected residual network. Features of each pixel location Iterate through the connected residual networks to output feature maps. Features of all pixel locations According to the following formula: ； Calculate and obtain feature map Non-local features corresponding to each pixel position ,in, This indicates the number of pixels in the output feature map of the connected residual network; Step d. Traverse the connected residual networks and output feature maps. The position of each pixel in the middle is recorded. The corresponding nonlocal features are represented as According to the following formula: ； Features of pixel locations Its corresponding nonlocal features Perform linear addition to obtain global features. ,in, This represents the weight matrix used for training in the fourth convolutional layer, Conv4. Step e. Follow the formula below: ； Calculate and obtain channel features ,in, This represents the weight matrix used for training in the fifth convolutional layer, Conv5. This represents the weight matrix used for training in the sixth convolutional layer, Conv6. Step f. Use the following formula: ； Calculate and obtain the output feature map of the connected residual network. Update .

2. The cross-modal person re-identification method based on multi-dimensional information learning according to claim 1, characterized in that: In step B, each classification sub-network in the preset classification network includes, from the input end to the output end, a front-end convolutional layer Conv, a feature extraction module, and a generalized average pooling layer GEM connected in series; the number of multi-dimensional information mining modules (MIM) is equal to the number of all residual networks in the preset classification network, and each MIM corresponds one-to-one with each residual network. The inputs of each Multidimensional Information Mining (MIM) module are connected to the outputs of their corresponding residual networks, forming feature extraction sub-blocks. The inputs of the residual networks constitute the inputs of the feature extraction sub-blocks, and the outputs of the MIM modules constitute the outputs of the feature extraction sub-blocks. The feature extraction sub-blocks corresponding to each residual network in the feature extraction module are sequentially connected from the input to the output. The input of the first feature extraction sub-block in sequence constitutes the input of the feature extraction module, and the output of the last feature extraction sub-block in sequence constitutes the output of the feature extraction module. This updates the feature extraction module, which in turn updates each classification sub-network. The inputs of two classification sub-networks constitute the two inputs of the classification network to be trained, and the outputs of the two classification sub-networks are connected to the two inputs of the classifier. The output of the classifier constitutes the output of the classification network to be trained, thus forming the classification network to be trained.

3. The cross-modal person re-identification method based on multi-dimensional information learning according to claim 2, characterized in that: Each of the multi-dimensional information mining (MIM) modules includes a first convolutional layer (Conv1), a second convolutional layer (Conv2), a third convolutional layer (Conv3), a fourth convolutional layer (Conv4), a fifth convolutional layer (Conv5), a sixth convolutional layer (Conv6), a first multiplication module, a second multiplication module, a first addition module, an average pooling layer (AvgPool), a ReLU activation function, a sigmoid activation function, and a weight layer (Reweight). In the MIM structure, the inputs of the first convolutional layer (Conv1), the second convolutional layer (Conv2), the third convolutional layer (Conv3), the average pooling layer (AvgPool), one input of the first addition module, and one input of the weight layer (Reweight) are connected, and these connections constitute the multi-dimensional information. The input terminals of the mining module MIM are as follows: the output terminals of the second convolutional layer Conv2 and the third convolutional layer Conv3 are connected to the two input terminals of the first multiplication module; the output terminals of the first convolutional layer Conv1 and the first multiplication module are connected to the two input terminals of the second multiplication module; the output terminal of the second multiplication module is connected to the input terminal of the fourth convolutional layer Conv4; the output terminal of the average pooling layer AvgPool is connected in series with the fifth convolutional layer Conv5, the ReLU activation function, the sixth convolutional layer Conv6, and the sigmoid activation function, and then connected to the other input terminal of the weight layer Reweight; the output terminals of the fourth convolutional layer Conv4 and the weight layer Reweight are respectively connected to the other two input terminals of the first addition module; the output terminals of the first addition module constitute the output terminals of the multidimensional information mining module MIM.

4. The cross-modal person re-identification method based on multi-dimensional information learning according to claim 1, characterized in that: In step C, the weighted regularized triplet loss is used between the outputs of the generalized average pooling layers (GEM) in the two classification subnetworks of the classification network to be trained. To constrain the training process of the classification network to be trained; Weighted Regularized Triple Loss as follows: ； ， ； in, This represents the triplet in each training batch for each anchor sample. and These correspond to different modes in the visible light reference image and the infrared reference image, respectively. and These correspond to the same modes in the visible light reference image and the infrared reference image, respectively. It is the set of positive samples corresponding to the anchor point. The anchor point corresponds to the negative sample set. This represents the Euclidean distance between the anchor point and its corresponding positive sample pair. This represents the Euclidean distance between the anchor point and its corresponding negative sample pair.

5. The cross-modal person re-identification method based on multi-dimensional information learning according to claim 4, characterized in that: Based on the output of the Generalized Average Pooling (GEM) layer in the two-class subnetwork of the classification network to be trained, the input of the Batch Normalization (BN) layer is further connected to the output of each BN layer, and the periodic smoothing of the focusing loss is achieved by the output of each BN layer. Combined with weighted regularized triplet loss To constrain the training process of the classification network to be trained; periodic smoothing focusing loss. as follows: ； in, For easy sample loss terms, Difficult sample loss term, It is a periodic smoothing factor, expressed as follows: ； in, This is the current round number. To the total number of training rounds, It is a periodic factor; Easy Sample Loss Term : ； Hard Sample Loss Term : ； Wherein, given a label Input image , Recognized as a class The predicted probability is encoded using the softmax function, by express, The number of pedestrian tags, , These are the hyperparameters used in training.

6. The cross-modal person re-identification method based on multi-dimensional information learning according to claim 1, characterized in that: In step A, for each of the collected visible light pedestrian sample images and each of the collected infrared pedestrian sample images, the following steps are further performed. To the steps Random local misalignment enhancement processing is performed to update each pedestrian sample image, and then a pedestrian sample image set is constructed. step Using random area aspect ratio In the pedestrian sample image, a random local region is defined at a random location, and is defined according to the following formula: ； Get the length of a random local region and height ,in, It is the area of ​​a random local region. , It is the proportion of a random local region to the area of ​​the original pedestrian sample image. It represents the total number of pixels in the original pedestrian sample image; step For each pixel location in a random local region... According to the following formula: ； Obtain the pixel position after random local misalignment This leads to random local misalignment enhancement processing of pedestrian sample images, whereby... For randomly staggered radians, It is the offset of the shearing.

Images