Cross-modal pedestrian re-identification method based on multi-feature hierarchical fusion learning

Abstract

The invention belongs to the technical field of image processing, and discloses a cross-modal pedestrian re-identification method based on multi-feature hierarchical fusion learning. Comprising the steps of dividing a data set into a natural light set and an infrared set; setting a modal feature enhancement module, and performing modal enhancement on the original pedestrian image; a four-flow Resnet 50 backbone network model is constructed; taking the generated enhanced image as the input of a backbone network, and extracting a feature map; setting a feature fusion module based on a time sequence, and performing hierarchical time sequence fusion on the feature map; jointly optimizing the loss training model; and the query image and the image in the image library are matched after being subjected to model feature extraction. The modal difference is effectively relieved, and the re-identification accuracy is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a cross-modal pedestrian re-identification method based on multi-feature hierarchical fusion learning. Background Art

[0002] Given a query image taken by a natural light or infrared camera, the goal of visible-infrared person re-identification (VI-ReID) is to retrieve pedestrians with the same identity in the infrared or natural light database. With the popularity of surveillance cameras and the introduction of related datasets, the VI-ReID task has received a lot of attention from researchers.

[0003] As an image retrieval task, VI-ReID faces two challenges: data bias in the metric function and the incompatibility between metric and representation learning.

[0004] In order to narrow the gap within the class and the difference between modalities, current methods can be divided into two categories: pixel-based and feature-based. The pixel-based method [AlignGan, JSIA] aims to generate fake images with the corresponding modality coloring style through a generative model, and the fake images are used to assist image retrieval. However, due to the large difference in modalities, style transfer requires a complex network structure, and this method has not achieved good performance.

[0005] The current mainstream approach is to train the model to learn modal coexistence features by designing a metric function and limiting the feature distance between modalities. MMN, AGW, and FBP-AL are based on the classic triple strategy, which limits the relative distance between modalities of the same type and between classes to alleviate modal differences. However, these methods only optimize one pair of samples at a time, lack robustness, and are prone to sub-optimal results. HCT and MPNAet proposed to constrain the distance from the sample point to the modal center, which can alleviate the above problems to a certain extent. However, the modal center is based on the feature mean of the current batch, and it still cannot avoid the data bias caused by small batches. In addition, the metric function evaluates features from the perspective of feature distance, while representation learning evaluates features from the perspective of identity expression. This difference will affect model training. Summary of the invention

[0006] The purpose of the present invention is to provide a cross-modal pedestrian re-identification method based on multi-feature hierarchical fusion learning, which performs hierarchical fusion of features from a temporal perspective, helps the model learn richer features, and effectively alleviates modal differences.

[0007] To achieve the above object, the technical solution adopted by the present invention is:

[0008] A cross-modal pedestrian re-identification method based on multi-feature hierarchical fusion learning, the cross-modal pedestrian re-identification method based on multi-feature hierarchical fusion learning comprising:

[0009] Step S1: a visible light modality image / infrared modality image is given as a query image, and an image library containing infrared modality images / visible light modality images is given;

[0010] Step S2, loading a multi-feature hierarchical fusion learning network, wherein the multi-feature hierarchical fusion learning network includes a modal feature enhancement module, a four-stream Resnet-50 backbone network model, and a time-series-based feature fusion module;

[0011] Step S3, performing two modality enhancements on the visible light modality image and the infrared modality image respectively by the modality feature enhancement module to obtain a visible light first enhanced modality image, a visible light second enhanced modality image, an infrared first enhanced modality image, and an infrared second enhanced modality image;

[0012] Step S4, inputting the visible light first enhanced modality image, the visible light second enhanced modality image, the infrared first enhanced modality image, and the infrared second enhanced modality image into a four-stream Resnet-50 backbone network model, and outputting feature maps C2, C3, C4, and C5 of different sizes;

[0013] Step S5: input the feature maps C2, C3, C4 and C5 of different sizes into the time-series-based feature fusion module for hierarchical time-series fusion. The time-series-based feature fusion module operates as follows:

[0014] The feature maps C3 and C4 are unified into the same size as the feature map C2 through 1×1 convolution and upsampling, respectively, and are denoted as feature f 3 and f 4 , and record the feature map C2 as feature f 2 , the feature map C5 is unified into the same size as the feature map C2 through 1×1 convolution and upsampling, and is recorded as feature f ori ;

[0015] First, the feature f t Passing into LSTM 2 The network is processed, t∈[2,T] is the level, and the highest level T=4, and then the output result is fused with the original feature to obtain the fused feature It is expressed as follows:

[0016]

[0017] Then the obtained features Perform attention processing to obtain features It is expressed as follows:

[0018]

[0019] The final features With feature f ori Further integration can be expressed as follows:

[0020]

[0021] Among them, SE t represents the attention processing operation for the t-th level feature, represents the fusion feature of the tth level, f out represents the fusion features finally output by the time-series-based feature fusion module, and represents the first enhanced modality fusion feature of visible light, represents the visible light second enhanced modality fusion feature, represents the infrared first enhanced modality fusion feature, represents the infrared second enhanced modality fusion feature;

[0022] Step S6: Fusion features based on visible light And the fusion features corresponding to infrared Calculate the matching degree between the query image and the images in the image library, and extract the top-k images with the highest matching degree in the image library as the re-identification results.

[0023] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution, but are merely further supplements or preferences. Under the premise that there are no technical or logical contradictions, each optional method can be combined with the above-mentioned overall solution separately, and multiple optional methods can also be combined.

[0024] Preferably, the two modality enhancements are performed on the visible light modality image and the infrared modality image respectively to obtain a visible light first enhanced modality image, a visible light second enhanced modality image, an infrared first enhanced modality image, and an infrared second enhanced modality image, including:

[0025] For the visible light modality image X vis The first enhancement method is to first enhance the original visible light modality image X vis The three channels x r ,x g ,x b Randomly weighted to obtain the degenerate graph X ds , and then compared with the original visible light modality image X vis Weighted fusion is performed to obtain the visible light first enhanced modality image It is expressed as follows:

[0026] X ds=α1×x r +α2×x g +α3×x b

[0027]

[0028] Among them, x r ,x g ,x b They are the visible light modality images X vis R channel, G channel and B channel, α1, α2, α3 are random weighted weights, belonging to [0,1], and α1+α2+α3=1, β1, β2 are weighted fusion weights, belonging to [0,1], and β1+β2=1;

[0029] For the visible light modality image X vis The second enhancement method: In the visible light modality image X vis A channel is randomly selected from the three channels, and the information of the other two channels is added to the randomly selected channel with a preset coefficient, which is expressed by the following formula:

[0030]

[0031] in, is the visible light second enhanced modality image, RandSelect is the random selection operation, is the image X in the visible light modality vis Randomly select one of the three channels of is the visible light modality image X vis The other two channels except the selected channel among the three channels (x r ,x g ,x b ) is the visible light modality image X vis The three channels of γ1, γ2, and γ3 are preset coefficients, belonging to [0,1], and γ1+γ2+γ3=1, 0<γ2+γ3≤0.15;

[0032] For infrared modality image X ir The first enhancement method: channel x of the infrared modality image ir The pixel value is multiplied and expressed as follows:

[0033]

[0034] in, is the infrared first enhanced modality image, δ is the multiplication factor, which belongs to [0.01, 0.5];

[0035] For infrared modality image Xir The second enhancement method is to randomly flip and / or randomly erase the image, which can be expressed as follows:

[0036]

[0037] in, is the infrared second enhanced modality image, RandErase represents the random erasing function, and RanDFlip represents the random flipping function.

[0038] Preferably, the four-stream Resnet-50 backbone network model includes four branches for processing visible light first enhanced modality images, visible light second enhanced modality images, infrared first enhanced modality images, and infrared second enhanced modality images, respectively. Each branch includes a subnetwork with independent parameters, denoted as E vis1 、E vis2 、E ir1 and E ir2 , all implemented by the first layer of the Resnet-50 network;

[0039] The four-stream Resnet-50 backbone network model also contains a sub-network E with shared parameters. S , implemented by the remaining layers of the Resnet-50 network except the first layer.

[0040] Preferably, the visible light first enhanced modality image, the visible light second enhanced modality image, the infrared first enhanced modality image and the infrared second enhanced modality image are input into a four-stream Resnet-50 backbone network model, and feature maps C2, C3, C4 and C5 of different sizes are output, including:

[0041] Visible light first enhanced modality image Enter subnetwork E vis1 ;

[0042] Visible light second enhancement mode image Enter subnetwork E vis2 ;

[0043] Infrared first enhanced modality image Enter subnetwork E ir1 ;

[0044] Infrared second enhanced modality image Enter subnetwork E ir2 ;

[0045] Take subnetwork E vis1 、E vis2 、E ir1 and E ir2 The outputs of are concatenated to obtain feature map C1;

[0046] Input feature map C1 into sub-network E S , get the sub-network E S The output feature maps of each layer are C2, C3, C4 and C5.

[0047] Preferably, the training process of the multi-feature hierarchical fusion learning network is as follows:

[0048] Obtain a training data set divided into a natural light set and an infrared set, wherein the natural light set contains visible light modality images, and the infrared set contains infrared modality images;

[0049] The natural light set and the infrared set are input into the multi-feature hierarchical fusion learning network, and the joint optimization loss is calculated according to the output of the multi-feature hierarchical fusion learning network for training and updating.

[0050] Preferably, the calculating the joint optimization loss comprises:

[0051]

[0052] in, To jointly optimize the loss, represents the heterocentric cluster loss, Indicates id loss, represents the triplet loss between modalities, represents the four-element center cluster loss;

[0053] The id loss The loss is calculated for the features of the visible light first enhanced modality image and the infrared first enhanced modality image obtained by the four-stream Resnet-50 backbone network model. The calculation formula is as follows:

[0054]

[0055] Among them, P is the number of categories in the training data set, a represents the ath category, and q a Represents the actual label distribution probability of the a-th category, p a represents the distribution probability of the a-th category prediction, ε is a constant, and y represents the category label of the pedestrian;

[0056] The heterocentric cluster loss The calculation formula is as follows:

[0057]

[0058] Among them, Q is the number of batches in the training phase and The number of images in each class, f() is the output before the fully connected layer of the four-stream Resnet-50 backbone network model, mean[AB] is the mean operation of the difference between object A and object B, represents the i-th visible light first enhanced modality image in the batch during the training phase, represents the i-th infrared first enhanced modality image in the batch during the training phase;

[0059] The triplet loss between the modalities The calculation formula is expressed as:

[0060]

[0061] in, Indicates VIS 1 and IR 1 The triplet loss between modalities, Indicates that the input comes from VIS 1 and IR 1 Positive sample pairs of modalities and from VIS 1 The triplet loss calculated for the negative sample pairs of the modality, Indicates that the input comes from VIS 1 and IR 1 Positive sample pairs of modality and from IR 1 The triplet loss calculated by the negative sample pair of the modality, VIS 1 Mode indicates the visible light first enhanced mode corresponding to the visible light first enhanced mode image, IR 1 The mode indicates the infrared first enhanced mode corresponding to the infrared first enhanced mode image. and The available formula is:

[0062]

[0063] Where M represents the number of pedestrians randomly selected in each iteration, b represents the bth pedestrian, c represents the cth pedestrian, ξ is the interval parameter, Indicates VIS 1 The bth pedestrian and IR in the image 1 The Euclidean distance between the cth pedestrians in the image, Indicates VIS 1 The bth pedestrian and IR in the image 1 The Euclidean distance between the kth pedestrians in the image, Representation IR 1 The bth pedestrian and VIS in the image 1 The Euclidean distance between the cth pedestrians in the image, Representation IR 1 The bth pedestrian and VIS in the image1 The Euclidean distance between the kth pedestrians in the image, and b and c are the same, indicating the same pedestrian; k is different from b and c, indicating different pedestrians, y b Indicates the category label of the bth pedestrian in the current mode, y c Represents y b Category labels of the same person in different modalities, y k Represents y b and c Category labels of different pedestrians, [z] + =max(z,0), z is the function input;

[0064] The four-element center cluster loss The calculation formula is expressed as:

[0065]

[0066] in, Indicates VIS 1 and IR 1 The center cluster loss between modalities, Indicates VIS 1 and IR 2 The center cluster loss between modalities, Indicates VIS 2 and IR 1 The center cluster loss between modalities, Indicates VIS 2 and IR 2 Inter-modal center cluster loss, VIS 2 Mode represents the visible light second enhanced mode corresponding to the visible light second enhanced mode image, IR 2 The mode represents the infrared second enhanced mode corresponding to the infrared second enhanced mode image;

[0067] V 1 or V 2 Denote it as V, and I 1 or I 2 Recorded as I, the center cluster loss It is expressed as follows:

[0068]

[0069] Among them, d represents the d-th category, g represents the g-th category, ρ is the preset parameter, represents the average center position of the VIS fusion features corresponding to the pedestrians in the dth category, represents the average center position of the IR fusion features corresponding to the pedestrians in the dth category, and VIS is VIS 1 or VIS 2, IR is IR 1 or IR 2 , represents the average center position of the VIS or IR fusion features corresponding to the pedestrian of the g-th category, and ∥AB∥2 represents the calculation of the Euclidean distance between object A and object B.

[0070] The present invention provides a cross-modal pedestrian re-identification method based on multi-feature hierarchical fusion learning, which has the following beneficial effects compared with the prior art:

[0071] (1) In order to extract diverse features, the method of the present invention enriches the network that can be learned by the model by extracting features from each enhanced image from the perspective of image enhancement.

[0072] (2) The method of the present invention considers the cross-modal pedestrian re-identification problem from the perspective of time series fusion for the first time. Specifically, in the face of the diverse features obtained, this method designs a time series-based method to layer-by-layer fuse the learned features, effectively alleviating the modality differences and improving the re-identification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 A flowchart of a cross-modal person re-identification method based on multi-feature hierarchical fusion learning of the present invention;

[0074] Figure 2 It is a schematic diagram of the structure of the multi-feature hierarchical fusion learning network of the present invention;

[0075] Figure 3 Schematic diagram of the operation of the time-series-based feature fusion module of the present invention. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0077] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0078] In order to solve the defects of the prior art, this embodiment proposes a cross-modal pedestrian re-identification method based on multi-feature hierarchical fusion learning. Since the method involves a network module, for ease of understanding, this embodiment uses the training process plus the reasoning process to explain the whole.

[0079] like Figure 1 As shown, a cross-modal pedestrian re-identification method based on multi-feature hierarchical fusion learning in this embodiment includes the following steps:

[0080] (1) Divide the training data set into natural light set and infrared set.

[0081] In this example, the training data set D used in the training process is vis ,X ir}, divided into natural light set and infrared set This facilitates the subsequent processing of the two-stream network. M is the total number of images in the natural light set, i is the i-th visible light modality image, M is the total number of images in the infrared set, and j is the j-th infrared modality image.

[0082] (2) Set up a modal feature enhancement module to perform modal enhancement on the original pedestrian image.

[0083] like Figure 2 As shown, in this example, the visible light modality image and the infrared modality image are respectively enhanced in two modalities, so as to obtain four enhanced modality images, namely, a visible light first enhanced modality image, a visible light second enhanced modality image, an infrared first enhanced modality image, and an infrared second enhanced modality image.

[0084] For the visible light modality image X vis The first enhancement method is to use a spectrum jitter enhancement method to change the brightness, color and other information of the original image so that the model can better alleviate the difference between the modalities. Different from the traditional spectrum jitter enhancement method that directly weights the image with the black image, the original visible light modality image X is first vis The three channels x r ,x g ,x b Randomly weighted to obtain the degenerate graph X ds , and then compared with the original visible light modality image X vis Weighted fusion is performed to obtain the visible light first enhanced modality image It is expressed as follows:

[0085] X ds =α1×x r +α2×x g +α3×x b

[0086]

[0087] Among them, x r ,x g ,x b They are the visible light modality images X vis R channel, G channel and B channel, α1, α2, α3 are random weighted weights, belonging to [0,1], and α1+α2+α3=1, β1, β2 are weighted fusion weights, belonging to [0,1], and β1+β2=1.

[0088] For the visible light modality image X vis The second enhancement method: In the visible light modality image X vis A channel is randomly selected from the three channels to assist the model. In order to further improve the adaptability of the model in various environments, this example adds the information of the other two channels with a smaller coefficient to the randomly selected channel as auxiliary information to improve the robustness of the model, which is expressed as the following formula:

[0089]

[0090] in, is the visible light second enhanced modality image, RandSelect is the random selection operation, is the image X in the visible light modality vis Randomly select one of the three channels of is the visible light modality image X vis The other two channels except the selected channel among the three channels (x r ,x g ,x b ) is the visible light modality image X vis The three channels of γ1, γ2, and γ3 are preset coefficients, belonging to [0, 1], and γ1+γ2+γ3=1, 0<γ2+γ3≤0.15. In this embodiment, γ2+γ3=0.15.

[0091] For infrared modality image X ir The first enhancement method: channel x of the infrared modality image ir Multiplying the pixel value can effectively enhance the contrast of the image, so that the features can be more prominent and cleaned, which can be expressed by the following formula:

[0092]

[0093] in, is the infrared first enhanced modality image, δ is the multiplication factor, which belongs to [0.01, 0.5].

[0094] For infrared modality image X ir The second enhancement method is to not perform additional processing on the channel but simply randomly flip and / or randomly erase the image, which can be expressed as follows:

[0095]

[0096] in, is the infrared second enhanced modality image, RandErase represents a random erasing function, which includes whether to perform a random erasing selection, and RandFlip represents a random flipping function, which includes whether to perform a random flipping selection.

[0097] (3) A four-stream Resnet-50 backbone network model is set as the backbone network (backbone network) of the designed multi-feature hierarchical fusion learning network, which contains four branches to extract and Each branch contains a sub-network with independent parameters, denoted by E vis1 、E vis2 、E ir1 、E ir2 , which is implemented by the first layer of the Resnet-50 network. And a sub-network E with shared parameters S , which is implemented by the residual layers of the Resnet-50 network. vis1 、E vis2 、E ir1 、E ir2 The independent parameters are used to accommodate modal differences. S The shared parameters are designed to extract modality-independent features of pedestrians.

[0098] (4) The generated enhanced image is used as the input of the backbone network to extract the feature map.

[0099] Visible light first enhanced modality image Enter subnetwork E vis1 ; The visible light second enhanced modality image Enter subnetwork E vis2 ; Infrared first enhanced modality image Enter subnetwork E ir1 ; Infrared second enhanced modality image Enter subnetwork E ir2 ; Take subnetwork E vis1 、E vis2 、E ir1 and E ir2 The output of is concatenated to obtain the feature map C1 = {C1 vis1 ,C1vis2 ,C1 ir1 ,C1 ir2}, the feature map C1 contains four enhanced modes, C1 vis1 Represents the feature map C1, C1 corresponding to the first enhanced mode of visible light vis2 Represents the feature map C1, C1 corresponding to the second enhanced mode of visible light ir1 Indicates the feature map C1, C1 corresponding to the first enhanced mode of infrared ir2 Represents the feature map C1 corresponding to the second enhanced mode of infrared; input the feature map C1 into the sub-network E S , get the sub-network E S The output feature maps of each layer are C2, C3, C4 and C5.

[0100] Similarly, feature maps C2, C3, C4 and C5 also contain four enhanced modes, which can be divided into feature map C2 corresponding to the first enhanced mode of visible light vis1 、C3 vis1 、C4 vis1 、C5 vis1 , the feature map C2 corresponding to the second enhanced mode of visible light vis2 、C3 vis2 、C4 vis2 、C5 vis2 , the feature map C2 corresponding to the first enhanced mode of infrared ir1 、C3 ir1 、C4 ir1 、C5 ir1 , the feature map C2 corresponding to the second enhanced mode of infrared ir2 、C3 ir2 、C4 ir2 、C5 ir2 .

[0101] (5) Set up a time-series-based feature fusion module to perform hierarchical time-series fusion on the feature maps.

[0102] like Figure 3 As shown, the feature maps C3 and C4 are respectively unified into the same size as the feature map C2 through 1×1 convolution and upsampling, and are recorded as feature f 3 and f 4 , and record the feature map C2 as feature f 2 , the feature map C5 is unified into the same size as the feature map C2 through 1×1 convolution and upsampling, and is recorded as feature f ori .

[0103] First, the feature f t Passing into LSTM 2The network is processed, t∈[2,T] is the level, and the highest level T=4, and then the output result is fused with the original feature to obtain the fused feature It is expressed as follows:

[0104]

[0105] Then the obtained features Perform attention processing to obtain features It is expressed as follows:

[0106]

[0107] The final features With feature f ori Further integration can be expressed as follows:

[0108]

[0109] Among them, SE t represents the attention processing operation for the t-th level feature, represents the fusion feature of the tth level, f out represents the fusion features finally output by the time-series-based feature fusion module, and represents the first enhanced modality fusion feature of visible light, represents the visible light second enhanced modality fusion feature, represents the infrared first enhanced modality fusion feature, Represents the infrared second enhanced modality fusion feature.

[0110] (6) Jointly optimize the loss to train a multi-feature hierarchical fusion learning network, which includes a modal feature enhancement module, a four-stream Resnet-50 backbone network model, and a time-based feature fusion module.

[0111] This embodiment performs joint optimization on the obtained losses, and the overall loss is composed of the losses of four stages:

[0112]

[0113] in, To jointly optimize the loss, represents the heterocentric cluster loss, Indicates id loss, represents the triplet loss between modalities, represents the four-element center cluster loss;

[0114] The id loss After the enhanced image The loss is calculated by the features obtained through the backbone network. The calculation formula is as follows:

[0115]

[0116] Among them, P is the number of categories in the training data set, a represents the ath category, and q a Represents the actual label distribution probability of the a-th category, p a represents the distribution probability of the a-th category prediction, ε is a small constant, and y represents the category label of the pedestrian. In the id loss, only the loss calculation is performed on the visible light first enhanced mode image and the infrared first enhanced mode image. In the specific operation, based on Corresponding C2 vis1 、C3 vis1 、C4 vis1 、C5 vis1 ,as well as Corresponding C2 ir1 、C3 ir1 、C4 ir1 、C5 ir1 , the feature maps of the same modality are concatenated and processed by the softmax function to obtain p a .

[0117] Heterocentric cluster loss For the input image The loss is calculated by the features obtained through the backbone network. The calculation formula is as follows:

[0118]

[0119] Among them, Q is the number of batches in the training phase and The number of images in each class, f() is the output before the fully connected layer of the four-stream Resnet-50 backbone network model, mean[AB] is the mean operation of the difference between object A and object B, represents the i-th visible light first enhanced modality image in the batch during the training phase, represents the i-th infrared first enhanced modality image in the batch during the training phase;

[0120] Inter-modality triplet loss For the input image The features obtained by multi-feature joint progressive learning network processing (i.e. fusion feature f out ) is used to calculate the loss, and the calculation formula is expressed as:

[0121]

[0122] in, Indicates VIS 1 and IR 1 The triplet loss between modalities, Indicates that the input comes from VIS 1 and IR 1 Positive sample pairs of modalities and from VIS 1 The triplet loss calculated for the negative sample pairs of the modality, Indicates that the input comes from VIS 1 and IR 1 Positive sample pairs of modality and from IR 1 The triplet loss calculated by the negative sample pair of the modality, VIS 1 The modality indicates the first enhanced modality of visible light corresponding to the first enhanced modality image of visible light, that is, the enhanced modality obtained after the visible light modality is enhanced by the first method, IR 1 The mode indicates the infrared first enhanced mode corresponding to the infrared first enhanced mode image, that is, the enhanced mode obtained after the infrared mode is enhanced by the first method. and The available formula is:

[0123]

[0124] Where M represents the number of pedestrians randomly selected in each iteration, b represents the bth pedestrian, c represents the cth pedestrian, ξ is the interval parameter, Indicates VIS 1 The bth pedestrian and IR in the image 1 The Euclidean distance between the cth pedestrians in the image, Indicates VIS 1 The bth pedestrian and IR in the image 1 The Euclidean distance between the kth pedestrians in the image, Representation IR 1 The bth pedestrian and VIS in the image 1 The Euclidean distance between the cth pedestrians in the image, Representation IR 1 The bth pedestrian and VIS in the image 1 The Euclidean distance between the kth pedestrians in the image, and b and c are the same, indicating the same pedestrian; k is different from b and c, indicating different pedestrians, y b Indicates the category label of the bth pedestrian in the current mode, y c Indicates that b Category labels of the same person in different modalities, y k Indicates that b and c Category labels of different pedestrians, [z] +=max(z,0), z is the function input.

[0125] Four-element center cluster loss For the input image The features obtained by multi-feature joint progressive learning network processing (i.e. fusion features ) The average center position is calculated based on the pedestrian position information in the image. The calculation formula is:

[0126]

[0127] in, Indicates VIS 1 and IR 1 The center cluster loss between modalities, Indicates VIS 1 and IR 2 The center cluster loss between modalities, Indicates VIS 2 and IR 1 The center cluster loss between modalities, Indicates VIS 2 and IR 2 Inter-modal center cluster loss, VIS 2 Mode represents the visible light second enhanced mode corresponding to the visible light second enhanced mode image, IR 2 The modality indicates the infrared second enhanced modality corresponding to the infrared second enhanced modality image.

[0128] V 1 or V 2 V, I 1 or I 2 Recorded as I, the center cluster loss It is expressed as follows:

[0129]

[0130] Among them, d represents the d-th category, g represents the g-th category, ρ is the preset parameter, represents the average center position of the VIS fusion features corresponding to the pedestrians in the dth category, represents the average center position of the IR fusion features corresponding to the pedestrians in the dth category, and VIS is VIS 1 or VIS 2 , IR is IR 1 or ID 2 , It represents the average center position of the VIS or IR fusion features corresponding to the pedestrians in the g-th category. In actual calculation, n takes V and I as input into the formula for calculation. ∥AB∥2 represents the Euclidean distance between objects A and B.

[0131]

[0132] in, is the average center position of the first enhanced modal fusion feature of visible light, that is, V is V 1 , VIS is VIS 1 c calculated at V , when you need to calculate the c corresponding to a specific pedestrian V (For example ), the first enhanced modal fusion feature of visible light containing the corresponding pedestrian is taken for calculation. K represents the number of images under each pedestrian category. Represents the visible light first enhanced modality fusion feature corresponding to the j-th image.

[0133] in, is the average center position of the fusion feature of the second enhanced modality of visible light, that is, V is V 2 , VIS is VIS 2 c calculated at V , when you need to calculate the c corresponding to a specific pedestrian V (For example ), the second enhanced modal fusion feature of the visible light containing the corresponding pedestrian is taken for calculation. Represents the visible light second enhanced modality fusion feature corresponding to the j-th image.

[0134] in, is the average center position of the infrared first enhanced modality fusion feature, that is, I is I 1 , IR is IR 1 c calculated at I , when you need to calculate the c corresponding to a specific pedestrian I (For example ), the infrared first enhanced modal fusion feature containing the corresponding pedestrian is taken for calculation. Represents the infrared first enhanced modality fusion feature corresponding to the j-th image.

[0135] in, is the average center position of the infrared second enhanced modality fusion feature, that is, I is I 2 , IR is IR 2 c calculated at I , when you need to calculate the c corresponding to a specific pedestrian I (For example ), the infrared second enhanced modality fusion feature containing the corresponding pedestrian is taken for calculation. Represents the infrared second enhanced modality fusion feature corresponding to the j-th image.

[0136] (7) The query image and the images in the gallery are matched after their features are extracted by the model.

[0137] In the inference phase, given a query image in natural light (infrared) mode and an infrared (natural light) image library, this example uses the image as the input of the model and projects these images into the same feature space. Based on the fusion features corresponding to visible light And the fusion features corresponding to infrared Calculate the matching degree between the query image and the images in the image library, and extract the top-k images with the highest matching degree in the image library as the re-identification results. Splicing, Splicing, based on the existing feature matching method, measures the similarity between the two modalities after splicing as the matching degree between the two modalities.

[0138] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0139] The above-mentioned embodiments only express several implementation modes of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A cross-modal person re-identification method based on multi-feature hierarchical fusion learning, characterized in that: The cross-modal person re-identification method based on multi-feature hierarchical fusion learning includes: Step S1: a visible light modality image / infrared modality image is given as a query image, and an image library containing infrared modality images / visible light modality images is given; Step S2, loading a multi-feature hierarchical fusion learning network, wherein the multi-feature hierarchical fusion learning network includes a modal feature enhancement module, a four-stream Resnet-50 backbone network model, and a time-series-based feature fusion module; Step S3, performing two modality enhancements on the visible light modality image and the infrared modality image respectively by the modality feature enhancement module to obtain a visible light first enhanced modality image, a visible light second enhanced modality image, an infrared first enhanced modality image, and an infrared second enhanced modality image; Step S4, inputting the visible light first enhanced modality image, the visible light second enhanced modality image, the infrared first enhanced modality image, and the infrared second enhanced modality image into a four-stream Resnet-50 backbone network model, and outputting feature maps C2, C3, C4, and C5 of different sizes; Step S5: input the feature maps C2, C3, C4 and C5 of different sizes into the time-series-based feature fusion module for hierarchical time-series fusion. The time-series-based feature fusion module operates as follows: The feature maps C3 and C4 are unified into the same size as the feature map C2 through 1×1 convolution and upsampling, respectively, and are denoted as feature f 3 and f 4 , and record the feature map C2 as feature f 2 , the feature map C5 is unified into the same size as the feature map C2 through 1×1 convolution and upsampling, and is recorded as feature f ori ; First, the feature f t Passing into LSTM 2 The network is processed, t∈[2,T] is the level, and the highest level T=4, and then the output result is fused with the original feature to obtain the fused feature It is expressed as follows: Then the obtained features Perform attention processing to obtain features It is expressed as follows: The final features With feature f ori Further integration can be expressed as follows: Among them, SE t represents the attention processing operation for the t-th level feature, represents the fusion feature of the tth level, f out represents the fusion feature finally output by the time-series-based feature fusion module, and f out = represents the first enhanced modality fusion feature of visible light, represents the visible light second enhanced modality fusion feature, represents the infrared first enhanced modality fusion feature, represents the infrared second enhanced modality fusion feature; Step S6: Fusion features based on visible light And the fusion features corresponding to infrared Calculate the matching degree between the query image and the images in the image library, and extract the top-k images with the highest matching degree in the image library as the re-identification results.

2. The cross-modal person re-identification method based on multi-feature hierarchical fusion learning according to claim 1 is characterized in that: The method of performing two modality enhancements on the visible light modality image and the infrared modality image to obtain a first visible light enhanced modality image, a second visible light enhanced modality image, a first infrared enhanced modality image, and a second infrared enhanced modality image comprises: For the visible light modality image X vis The first enhancement method is to first enhance the original visible light modality image X vis The three channels x r ,x g ,x b Perform random weighting to obtain the degenerate graph X ds , and then compared with the original visible light modality image X vis Weighted fusion is performed to obtain the visible light first enhanced modality image It is expressed as follows: x ds =α1×x r +α2×x g +α3×x b Among them, x r ,x g ,x b They are the visible light modality images X vis R channel, G channel and B channel, α1, α2, α3 are random weighted weights, belonging to [0,1], and α1+α2+α3=1, β1, β2 are weighted fusion weights, belonging to [0,1], and β1+β2=1; For the visible light modality image X vis The second enhancement method: In the visible light modality image X vis A channel is randomly selected from the three channels, and the information of the other two channels is added to the randomly selected channel with a preset coefficient, which is expressed by the following formula: in, is the visible light second enhanced modality image, RandSelect is the random selection operation, is the image X in the visible light modality vis Randomly select one of the three channels of is the visible light modality image X vis The other two channels except the selected channel among the three channels (x r ,x g ,x b ) is the visible light modality image X vis The three channels of γ1, γ2, and γ3 are preset coefficients, belonging to [0,1], and γ1+γ2+γ3=1, 0<γ2+γ3≤0.15; For the infrared modality image X ir The first enhancement method: channel x of the infrared modality image ir The pixel value is multiplied and expressed as follows: in, is the infrared first enhanced modality image, δ is the multiplication factor, which belongs to [0.01, 0.5]; For the infrared modality image X ir The second enhancement method is to randomly flip and / or randomly erase the image, which can be expressed as follows: in, is the infrared second enhanced modality image, RandErase represents the random erasing function, and RandFlip represents the random flipping function.

3. The cross-modal person re-identification method based on multi-feature hierarchical fusion learning according to claim 1 is characterized in that: The four-stream Resnet-50 backbone network model includes four branches for processing visible light first enhanced modality images, visible light second enhanced modality images, infrared first enhanced modality images, and infrared second enhanced modality images, respectively. Each branch contains a subnetwork with independent parameters, denoted as E vis1 、E vis2 、E ir1 and E ir2 , all implemented by the first layer of the Resnet-50 network; The four-stream Resnet-50 backbone network model also contains a sub-network E with shared parameters. S , implemented by the remaining layers of the Resnet-50 network except the first layer.

4. The cross-modal person re-identification method based on multi-feature hierarchical fusion learning according to claim 3 is characterized in that: The method inputs the visible light first enhanced modality image, the visible light second enhanced modality image, the infrared first enhanced modality image and the infrared second enhanced modality image into a four-stream Resnet-50 backbone network model, and outputs feature maps C2, C3, C4 and C5 of different sizes, including: The first enhanced visible light modality image Enter subnetwork E vis1 ; Visible light second enhancement mode image Enter subnetwork E vis2 ; Infrared first enhanced modality image Enter subnetwork E ir1 ; Infrared second enhanced modality image Enter subnetwork E ir2 ; Take subnetwork E vis1 、E vis2 、E ir1 and E ir2 The outputs of are concatenated to obtain feature map C1; Input feature map C1 into sub-network E S , get the sub-network E S The output feature maps of each layer are C2, C3, C4 and C5.

5. The cross-modal person re-identification method based on multi-feature hierarchical fusion learning according to claim 1 is characterized in that: The training process of the multi-feature hierarchical fusion learning network is as follows: Obtain a training data set divided into a natural light set and an infrared set, wherein the natural light set contains visible light modality images, and the infrared set contains infrared modality images; The natural light set and the infrared set are input into the multi-feature hierarchical fusion learning network, and the joint optimization loss is calculated according to the output of the multi-feature hierarchical fusion learning network for training and updating.

6. The cross-modal person re-identification method based on multi-feature hierarchical fusion learning according to claim 5 is characterized in that: The calculation of the joint optimization loss includes: in, To jointly optimize the loss, represents the heterocentric cluster loss, Indicates id loss, represents the triplet loss between modalities, represents the four-element center cluster loss; The id loss The loss is calculated for the features of the visible light first enhanced modality image and the infrared first enhanced modality image obtained by the four-stream Resnet-50 backbone network model. The calculation formula is as follows: Among them, P is the number of categories in the training data set, a represents the ath category, and q a Represents the actual label distribution probability of the a-th category, p a represents the distribution probability of the a-th category prediction, ε is a constant, and y represents the category label of the pedestrian; The heterocentric cluster loss The calculation formula is as follows: Among them, Q is the number of batches in the training phase and The number of images in each class, f() is the output before the fully connected layer of the four-stream Resnet-50 backbone network model, mean[AB] is the mean operation of the difference between object A and object B, represents the i-th visible light first enhanced modality image in the batch during the training phase, represents the i-th infrared first enhanced modality image in the batch during the training phase; The triplet loss between the modalities The calculation formula is expressed as: in, Indicates VIS 1 and IR 1 The triplet loss between modalities, Indicates that the input comes from VIS 1 and IR 1 Positive sample pairs of modalities and from VIS 1 The triplet loss calculated for the negative sample pairs of the modality, Indicates that the input comes from VIS 1 and IR 1 Positive sample pairs of modality and from IR 1 The triplet loss calculated by the negative sample pair of the modality, VIS 1 Mode indicates the visible light first enhanced mode corresponding to the visible light first enhanced mode image, IR 1 The mode indicates the infrared first enhanced mode corresponding to the infrared first enhanced mode image. and The available formula is: Where M represents the number of pedestrians randomly selected in each iteration, b represents the bth pedestrian, c represents the cth pedestrian, ξ is the interval parameter, Indicates VIS 1 The bth pedestrian and IR in the image 1 The Euclidean distance between the cth pedestrians in the image, Indicates VIS 1 The bth pedestrian and IR in the image 1 The Euclidean distance between the kth pedestrians in the image, Representation IR 1 The bth pedestrian and VIS in the image 1 The Euclidean distance between the cth pedestrians in the image, Representation IR 1 The bth pedestrian and VIS in the image 1 The Euclidean distance between the kth pedestrians in the image, and b and c are the same, indicating the same pedestrian; k is different from b and c, indicating different pedestrians, y b Indicates the category label of the bth pedestrian in the current mode, y c Represents y b Category labels of the same person in different modalities, y k Represents y b and c Category labels of different pedestrians, [z] + =max(z,0), z is the function input; The four-element center cluster loss The calculation formula is expressed as: in, Indicates VIS 1 and IR 1 The center cluster loss between modalities, Indicates VIS 1 and IR 2 The center cluster loss between modalities, Indicates VIS 2 and IR 1 The center cluster loss between modalities, Indicates VID 2 and IR 2 Inter-modal center cluster loss, VIS 2 Mode represents the visible light second enhanced mode corresponding to the visible light second enhanced mode image, IR 2 The mode represents the infrared second enhanced mode corresponding to the infrared second enhanced mode image; V 1 or V 2 V, I 1 or I 2 Recorded as I, the center cluster loss It is expressed as follows: Among them, d represents the d-th category, g represents the g-th category, ρ is the preset parameter, represents the average center position of the VIS fusion features corresponding to the pedestrians in the dth category, represents the average center position of the IR fusion features corresponding to the pedestrians in the dth category, and VIS is VIS 1 or VIS 2 , IR is IR 1 or IR 2 , represents the average center position of the VIS or IR fusion features corresponding to the pedestrians in the g-th category, and ||AB||2 represents the calculation of the Euclidean distance between objects A and B.

Images