A cross-modal person re-identification method based on multi-feature layered fusion learning
Through the multi-feature hierarchical fusion learning method, combined with modal feature enhancement and temporal feature fusion, the problems of modal difference and insufficient robustness in cross-modal pedestrian re-identification are solved, and the re-identification accuracy and model adaptability are improved.
Patent Information
- Application Number
- CN202411101372.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2025-09-23
- Estimated Expiration
- 2044-08-12
AI Technical Summary
Existing cross-modal person re-identification methods suffer from data bias and representation learning mismatch problems in the metric function, resulting in modal differences and insufficient robustness, affecting the re-identification accuracy.
A multi-feature hierarchical fusion learning method is adopted. Through the modal feature enhancement module and the four-stream Resnet-50 backbone network model, combined with the time-based feature fusion module, image enhancement and feature extraction are performed. The LSTM network is used for hierarchical time fusion, and the model is trained by joint optimization loss to alleviate modal differences.
It effectively alleviates modal differences, improves the accuracy of cross-modal person re-identification, and enhances the robustness and adaptability of the model.
Smart Images
Figure CN119251862B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of image processing technology, and specifically relates to a cross-modal pedestrian re-identification method based on multi-feature hierarchical fusion learning. Background Art
[0002] Given a query image captured by a natural light or infrared camera, the goal of visible-infrared person re-identification (VI-ReID) is to retrieve pedestrians with the same identity in the infrared or natural light database. With the popularity of surveillance cameras and the introduction of related datasets, the VI-ReID task has attracted a lot of attention from researchers.
[0003] As an image retrieval task, VI-ReID faces two challenges: data bias in the metric function and the incompatibility between metric and representation learning.
[0004] To narrow intra-class gaps and modality differences, current methods can be categorized as pixel-based and feature-based. Pixel-based methods [AlignGan, JSIA] aim to generate fake images with the corresponding modality's coloring style through generative models, which are then used to assist in image retrieval. However, due to the significant differences in modalities, style transfer requires complex network structures, and this approach has not achieved good performance.
[0005] The current mainstream approach is to train the model to learn modal coexistence features by designing a metric function and limiting the feature distance between modalities. MMN, AGW, and FBP-AL are based on the classic triple strategy, which limits the relative distance between modalities of the same type and between classes to alleviate modal differences. However, these methods only optimize one pair of samples at a time, lack robustness, and are prone to sub-optimal results. HCT and MPNAet proposed constraining the distance between sample points and the modal center to alleviate the above problems to a certain extent. However, the modal center is based on the feature mean of the current batch, and it still cannot avoid the data bias brought by small batches. In addition, the metric function evaluates features from the perspective of feature distance, while representation learning evaluates features from the perspective of identity expression. This difference will affect model training. Summary of the Invention
[0006] The purpose of the present invention is to provide a cross-modal pedestrian re-identification method based on multi-feature hierarchical fusion learning, which performs hierarchical fusion of features from a temporal perspective, helps the model learn richer features, and effectively alleviates modal differences.
[0007] To achieve the above object, the technical solution adopted by the present invention is:
[0008] A cross-modal person re-identification method based on multi-feature hierarchical fusion learning, the cross-modal person re-identification method based on multi-feature hierarchical fusion learning comprising:
[0009] Step S1: Given a visible light modality image / infrared modality image as a query image and an image library containing infrared modality images / visible light modality images;
[0010] Step S2: loading a multi-feature hierarchical fusion learning network, wherein the multi-feature hierarchical fusion learning network includes a modal feature enhancement module, a four-stream Resnet-50 backbone network model, and a time-series-based feature fusion module;
[0011] Step S3: performing two modal enhancements on the visible light modal image and the infrared modal image respectively by the modal feature enhancement module to obtain a first visible light enhanced modal image, a second visible light enhanced modal image, a first infrared enhanced modal image, and a second infrared enhanced modal image;
[0012] Step S4: input the visible light first enhanced modality image, the visible light second enhanced modality image, the infrared first enhanced modality image, and the infrared second enhanced modality image into a four-stream Resnet-50 backbone network model, and output feature maps C2, C3, C4, and C5 of different sizes;
[0013] Step S5: Input the feature maps C2, C3, C4, and C5 of different sizes into the time-series-based feature fusion module for hierarchical time-series fusion. The time-series-based feature fusion module operates as follows:
[0014] The feature maps C3 and C4 are respectively unified into the same size as the feature map C2 through 1×1 convolution and upsampling, and are denoted as feature f 3 and f 4 , and record the feature map C2 as feature f 2 , the feature map C5 is unified into the same size as the feature map C2 through 1×1 convolution and upsampling, and is recorded as feature f ori ;
[0015] First, the feature f t Passing into LSTM 2 The network is processed, t∈[2,T] is the level, and the highest level T=4, and then the output result is fused with the original feature to obtain the fused feature It is expressed as follows:
[0016]
[0017] Then the obtained features Perform attention processing to obtain features It is expressed as follows:
[0018]
[0019] The final features With feature f ori Further integration can be expressed as follows:
[0020]
[0021] Among them, SE t represents the attention processing operation for the t-th level feature, represents the fusion feature of the tth level, f out represents the fusion feature finally output by the time-series-based feature fusion module, and represents the first enhanced modality fusion feature of visible light, represents the visible light second enhanced modality fusion feature, represents the infrared first enhanced modality fusion feature, represents the infrared second enhanced modality fusion feature;
[0022] Step S6: Fusion features based on visible light And the fusion features corresponding to infrared Calculate the matching degree between the query image and the images in the image library, and extract the top-k images with the highest matching degree in the image library as the re-identification results.
[0023] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution. They are merely further supplements or optimizations. Under the premise that there are no technical or logical contradictions, each optional method can be combined separately for the above-mentioned overall solution, or multiple optional methods can be combined.
[0024] Preferably, the performing two modal enhancements on the visible light modality image and the infrared modality image respectively to obtain a first visible light enhanced modality image, a second visible light enhanced modality image, a first infrared enhanced modality image, and a second infrared enhanced modality image comprises:
[0025] For the visible light modality image X vis The first enhancement method is to first enhance the original visible light modality image X vis The three channels x r ,x g ,x b Perform random weighting to obtain the degenerate graph X ds , and then compared with the original visible light modality image X vis Perform weighted fusion to obtain the first enhanced modality image of visible light It is expressed as follows:
[0026] X ds=α1×x r +α2×x g +α3×x b
[0027]
[0028] Among them, x r ,x g ,x b They are the visible light modality images X vis The R channel, G channel and B channel of α1, α2, α3 are random weighted weights, belonging to [0, 1], and α1+α2+α3=1, β1 and β2 are weighted fusion weights, belonging to [0, 1], and β1+β2=1;
[0029] For the visible light modality image X vis The second enhancement method: in the visible light modality image X vis Randomly select one channel from the three channels and add the information of the other two channels to the randomly selected channel with a preset coefficient, which is expressed as follows:
[0030]
[0031] in, is the visible light second enhanced modality image, RandSelect is the random selection operation, is the image X in the visible light modality vis Randomly select one of the three channels of is the visible light modality image X vis The other two channels except the selected channel among the three channels (x r ,x g ,x b ) is the visible light modality image X vis The three channels of γ1, γ2, and γ3 are preset coefficients, belonging to [0,1], and γ1+γ2+γ3=1, 0<γ2+γ3≤0.15;
[0032] For the infrared modality image X ir The first enhancement method: channel x of the infrared modality image ir The pixel value is multiplied and expressed as follows:
[0033]
[0034] in, is the infrared first enhanced modality image, δ is the multiplication coefficient, which belongs to [0.01, 0.5];
[0035] For the infrared modality image Xir The second enhancement method is to randomly flip and / or randomly erase the image, which can be expressed as follows:
[0036]
[0037] in, is the infrared second enhanced modality image, RandErase represents the random erasing function, and RanDFlip represents the random flipping function.
[0038] Preferably, the four-stream Resnet-50 backbone network model includes four branches for processing visible light first enhanced modality images, visible light second enhanced modality images, infrared first enhanced modality images, and infrared second enhanced modality images, respectively. Each branch contains a subnetwork with independent parameters, denoted as E vis1 、E vis2 、E ir1 and E ir2 , are all implemented by the first layer of the Resnet-50 network;
[0039] The four-stream Resnet-50 backbone network model also contains a sub-network E with shared parameters. S , implemented by the remaining layers of the Resnet-50 network except the first layer.
[0040] Preferably, the method of inputting the visible light first enhanced modality image, the visible light second enhanced modality image, the infrared first enhanced modality image, and the infrared second enhanced modality image into a four-stream Resnet-50 backbone network model and outputting feature maps C2, C3, C4, and C5 of different sizes includes:
[0041] Visible light first enhanced modality image Input subnetwork E vis1 ;
[0042] The visible light second enhanced mode image Input subnetwork E vis2 ;
[0043] Infrared first enhanced modality image Input subnetwork E ir1 ;
[0044] Infrared second enhanced modality image Input subnetwork E ir2 ;
[0045] Take subnetwork E vis1 、E vis2 、E ir1 and E ir2 The output of is spliced to obtain the feature map C1;
[0046] Input the feature map C1 into the sub-network E S , get the sub-network E S The feature maps C2, C3, C4 and C5 output by each layer.
[0047] Preferably, the training process of the multi-feature hierarchical fusion learning network is as follows:
[0048] Obtaining a training data set divided into a natural light set and an infrared set, wherein the natural light set contains visible light modality images, and the infrared set contains infrared modality images;
[0049] The natural light set and the infrared set are input into the multi-feature hierarchical fusion learning network, and the joint optimization loss is calculated according to the output of the multi-feature hierarchical fusion learning network to perform training updates.
[0050] Preferably, the calculating the joint optimization loss includes:
[0051]
[0052] in, To jointly optimize the loss, represents the heterocentric cluster loss, Indicates id loss, represents the triplet loss between modalities, represents the four-element center cluster loss;
[0053] The id is lost The loss is calculated for the features obtained by passing the four-stream Resnet-50 backbone network model of the visible light first enhanced modality image and the infrared first enhanced modality image. The calculation formula is as follows:
[0054]
[0055] Among them, P is the number of categories in the training data set, a represents the ath category, q a Indicates the actual label distribution probability of the a-th category, p a represents the distribution probability of the a-th category prediction, ε is a constant, and y represents the category label of the pedestrian;
[0056] The heterocentric cluster loss The calculation formula is as follows:
[0057]
[0058] Among them, Q is the number of batches in the training phase and The number of images in each class, f() is the output before the fully connected layer of the four-stream Resnet-50 backbone network model, mean[AB] is the mean operation of the difference between object A and object B, represents the i-th visible light first enhanced modality image in the batch during the training phase, represents the i-th infrared first enhanced modality image in the batch during the training phase;
[0059] The triplet loss between the modalities The calculation formula is expressed as:
[0060]
[0061] in, Indicates VIS 1 and IR 1 Inter-modality triplet loss, Indicates that the input comes from VIS 1 and IR 1 Positive sample pairs of modalities and from VIS 1 The triplet loss calculated from the negative sample pairs of the modality, Indicates that the input comes from VIS 1 and IR 1 Positive sample pairs of modalities and from IR 1 The triplet loss calculated by the negative sample of the modality, VIS 1 Mode represents the visible light first enhanced mode corresponding to the visible light first enhanced mode image, IR 1 Mode represents the infrared first enhanced mode corresponding to the infrared first enhanced mode image. and The available formula is:
[0062]
[0063] Where M represents the number of pedestrians randomly selected in each iteration, b represents the bth pedestrian, c represents the cth pedestrian, ξ is the interval parameter, Indicates VIS 1 The bth pedestrian and IR in the image 1 The Euclidean distance between the c-th pedestrians in the image, Indicates VIS 1 The bth pedestrian and IR in the image 1 The Euclidean distance between the k-th pedestrians in the image, Represents IR 1 The bth pedestrian and VIS in the image 1 The Euclidean distance between the c-th pedestrians in the image, Represents IR 1 The bth pedestrian and VIS in the image1 The Euclidean distance between the kth pedestrians in the image, and b and c are the same, indicating the same pedestrian; k is different from b and c, indicating different pedestrians, y b Indicates the category label of the b-th pedestrian in the current mode, y c Indicates that y b Category labels of the same person in different modalities, y k Indicates that y b and y c Category labels of different pedestrians, [z] + =max(z,0), z is the function input;
[0064] The four-element center cluster loss The calculation formula is expressed as:
[0065]
[0066] in, Indicates VIS 1 and IR 1 Center cluster loss between modalities, Indicates VIS 1 and IR 2 Center cluster loss between modalities, Indicates VIS 2 and IR 1 Center cluster loss between modalities, Indicates VIS 2 and IR 2 Inter-modal center cluster loss, VIS 2 Mode represents the visible light second enhanced mode corresponding to the visible light second enhanced mode image, IR 2 The mode represents the infrared second enhanced mode corresponding to the infrared second enhanced mode image;
[0067] V 1 or V 2 Denote it as V, and I 1 or I 2 Recorded as I, the center cluster loss It is expressed as follows:
[0068]
[0069] Among them, d represents the d-th category, g represents the g-th category, and ρ is a preset parameter. represents the average center position of the VIS fusion features corresponding to the pedestrians in the d-th category, represents the average center position of the IR fusion features corresponding to the pedestrian of the d-th category, and VIS is VIS 1 or VIS 2, IR is IR 1 or IR 2 , represents the average center position of the VIS or IR fusion features corresponding to the pedestrian of the g-th category, and AB2 represents the Euclidean distance between objects A and B.
[0070] The present invention provides a cross-modal person re-identification method based on multi-feature layered fusion learning, which has the following advantages over the existing technology:
[0071] (1) In order to extract diverse features, the method of the present invention enriches the network that the model can learn by extracting features from each enhanced image from the perspective of image enhancement.
[0072] (2) The proposed method considers the cross-modal person re-identification problem from the perspective of temporal fusion for the first time. Specifically, in the face of the diverse features obtained, this method designs a temporal-based method to layer-wise fuse the learned features, effectively alleviating modal differences and improving re-identification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0073] Figure 1 This is a flow chart of a cross-modal person re-identification method based on multi-feature hierarchical fusion learning of the present invention;
[0074] Figure 2 Schematic diagram of the structure of the multi-feature hierarchical fusion learning network of the present invention;
[0075] Figure 3 Schematic diagram of the operation of the time-series-based feature fusion module of the present invention. DETAILED DESCRIPTION
[0076] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.
[0077] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.
[0078] To address the shortcomings of existing technologies, this embodiment proposes a cross-modal person re-identification method based on multi-feature layered fusion learning. Since this method involves network modules, for ease of understanding, this embodiment uses a training process followed by an inference process to explain the overall process.
[0079] like Figure 1 As shown, a cross-modal person re-identification method based on multi-feature layered fusion learning in this embodiment includes the following steps:
[0080] (1) The training dataset is divided into natural light set and infrared set.
[0081] In this example, the training data set D used in the training process is vis ,X ir}, divided into natural light set and infrared sets This facilitates subsequent two-stream network processing. M is the total number of images in the natural light set, i is the i-th visible light modality image, M is the total number of images in the infrared set, and j is the j-th infrared modality image.
[0082] (2) Set up a modal feature enhancement module to perform modal enhancement on the original pedestrian image.
[0083] like Figure 2 As shown, in this example, the visible light modal image and the infrared modal image are respectively enhanced in two modal ways, thereby obtaining four enhanced modal images, namely, a visible light first enhanced modal image, a visible light second enhanced modal image, an infrared first enhanced modal image, and an infrared second enhanced modal image.
[0084] For the visible light modality image X vis The first enhancement method is to use a spectrum jitter enhancement method to change the brightness, color and other information of the original image so that the model can better alleviate the difference between the modalities. Different from the traditional spectrum jitter enhancement method that directly weights the image with the black image, the original visible light modality image X is first vis The three channels x r ,x g ,x b Perform random weighting to obtain the degenerate graph X ds , and then compared with the original visible light modality image X vis Perform weighted fusion to obtain the first enhanced modality image of visible light It is expressed as follows:
[0085] X ds =α1×x r +α2×x g +α3×x b
[0086]
[0087] Among them, x r ,x g ,x b They are the visible light modality images X vis The R channel, G channel and B channel, α1, α2, α3 are random weighted weights, belonging to [0, 1], and α1+α2+α3=1, β1, β2 are weighted fusion weights, belonging to [0, 1], and β1+β2=1.
[0088] For the visible light modality image X vis The second enhancement method: in the visible light modality image X vis A channel is randomly selected from the three channels to assist the model. To further improve the model's adaptability in various environments, this example adds information from the other two channels to the randomly selected channel with a smaller coefficient as auxiliary information to improve the model's robustness, as expressed in the following formula:
[0089]
[0090] in, is the visible light second enhanced modality image, RandSelect is the random selection operation, is the image X in the visible light modality vis Randomly select one of the three channels of is the visible light modality image X vis The other two channels except the selected channel among the three channels (x r ,x g ,x b ) is the visible light modality image X vis γ1, γ2, and γ3 are preset coefficients, belonging to [0, 1], and γ1+γ2+γ3=1, 0<γ2+γ3≤0.15. In this embodiment, γ2+γ3=0.15.
[0091] For the infrared modality image X ir The first enhancement method: channel x of the infrared modality image ir Multiplying the pixel values can effectively enhance the contrast of the image, making the features more prominent and clean, which can be expressed by the following formula:
[0092]
[0093] in, is the infrared first enhanced modality image, δ is the multiplication coefficient, which is in the range of [0.01, 0.5].
[0094] For the infrared modality image X ir The second enhancement method is to randomly flip and / or randomly erase the image without performing additional channel processing, which can be expressed as follows:
[0095]
[0096] in, is the infrared second enhanced modality image, RandErase represents a random erasing function, which includes whether to perform random erasing selection, and RandFlip represents a random flipping function, which includes whether to perform random flipping selection.
[0097] (3) Set a four-stream Resnet-50 backbone network model as the backbone network (backbone network) of the designed multi-feature layered fusion learning network, which contains four branches to extract and Each branch contains a sub-network with independent parameters, denoted as E vis1 、E vis2 、E ir1 、E ir2 , which is implemented by the first layer of the Resnet-50 network. And a sub-network E with shared parameters S , which is implemented by the residual layers of the Resnet-50 network. E vis1 、E vis2 、E ir1 、E ir2 The independent parameters are used to accommodate modal differences. S The shared parameters are designed to extract modality-independent features of pedestrians.
[0098] (4) The generated enhanced image is used as the input of the backbone network to extract the feature map.
[0099] Visible light first enhanced modality image Input subnetwork E vis1 ; The visible light second enhanced mode image Input subnetwork E vis2 ; The infrared first enhanced modality image Input subnetwork E ir1 ; Infrared second enhanced modality image Input subnetwork E ir2 ; Take subnetwork E vis1 、E vis2 、E ir1 and E ir2 The output of is spliced to obtain the feature map C1 = {C1 vis1 ,C1vis2 ,C1 ir1 ,C1 ir2}, the feature map C1 contains four enhanced modes, C1 vis1 Represents the feature map C1, C1 corresponding to the first enhanced mode of visible light vis2 Represents the feature map C1, C1 corresponding to the second enhanced mode of visible light ir1 Indicates the feature map C1, C1 corresponding to the first enhanced mode of infrared ir2 Represents the feature map C1 corresponding to the second enhanced mode of infrared; input the feature map C1 into the sub-network E S , get the sub-network E S The feature maps C2, C3, C4 and C5 output by each layer.
[0100] Similarly, the feature maps C2, C3, C4 and C5 also contain four enhanced modes, which can be divided into the feature map C2 corresponding to the first enhanced mode of visible light vis1 、C3 vis1 、C4 vis1 、C5 vis1 , the feature map C2 corresponding to the second enhanced mode of visible light vis2 、C3 vis2 、C4 vis2 、C5 vis2 , the feature map C2 corresponding to the first enhanced mode of infrared ir1 、C3 ir1 、C4 ir1 、C5 ir1 , the feature map C2 corresponding to the second enhanced mode of infrared ir2 、C3 ir2 、C4 ir2 、C5 ir2 .
[0101] (5) Set up a time-series-based feature fusion module to perform hierarchical time-series fusion on the feature maps.
[0102] like Figure 3 As shown, the feature maps C3 and C4 are respectively unified into the same size as the feature map C2 through 1×1 convolution and upsampling, and are recorded as feature f 3 and f 4 , and record the feature map C2 as feature f 2 , the feature map C5 is unified into the same size as the feature map C2 through 1×1 convolution and upsampling, and is recorded as feature f ori .
[0103] First, the feature f t Passing into LSTM 2The network is processed, t∈[2,T] is the level, and the highest level T=4, and then the output result is fused with the original feature to obtain the fused feature It is expressed as follows:
[0104]
[0105] Then the obtained features Perform attention processing to obtain features It is expressed as follows:
[0106]
[0107] The final features With feature f ori Further integration can be expressed as follows:
[0108]
[0109] Among them, SE t represents the attention processing operation for the t-th level feature, represents the fusion feature of the tth level, f out represents the fusion feature finally output by the time-series-based feature fusion module, and represents the first enhanced modality fusion feature of visible light, represents the visible light second enhanced modality fusion feature, represents the infrared first enhanced modality fusion feature, Represents the infrared second enhanced modality fusion feature.
[0110] (6) Jointly optimize the loss to train a multi-feature hierarchical fusion learning network, which includes a modal feature enhancement module, a four-stream Resnet-50 backbone network model, and a time-series-based feature fusion module.
[0111] This embodiment performs joint optimization on the obtained losses. The overall loss is composed of the losses of four stages:
[0112]
[0113] in, To jointly optimize the loss, represents the heterocentric cluster loss, Indicates id loss, represents the triplet loss between modalities, represents the four-element center cluster loss;
[0114] The id is lost For the enhanced image The loss is calculated based on the features obtained through the backbone network. The calculation formula is as follows:
[0115]
[0116] Among them, P is the number of categories in the training data set, a represents the ath category, q a Indicates the actual label distribution probability of the a-th category, p a Indicates the distribution probability of the a-th category prediction, ε is a small constant, and y represents the category label of the pedestrian. In the id loss, only the loss calculation is performed on the visible light first enhanced mode image and the infrared first enhanced mode image. In the specific operation, based on Corresponding C2 vis1 、C3 vis1 、C4 vis1 、C5 vis1 ,as well as Corresponding C2 ir1 、C3 ir1 、C4 ir1 、C5 ir1 , the feature maps of the same modality are concatenated and processed by the softmax function to obtain p a .
[0117] Heterocentric cluster loss For the input image The loss is calculated based on the features obtained through the backbone network. The calculation formula is as follows:
[0118]
[0119] Among them, Q is the number of batches in the training phase and The number of images in each class, f() is the output before the fully connected layer of the four-stream Resnet-50 backbone network model, mean[AB] is the mean operation of the difference between object A and object B, represents the i-th visible light first enhanced modality image in the batch during the training phase, represents the i-th infrared first enhanced modality image in the batch during the training phase;
[0120] Inter-modal triplet loss For the input image The features obtained by multi-feature joint progressive learning network processing (i.e. fusion features f out ) to calculate the loss, the calculation formula is expressed as:
[0121]
[0122] in, Indicates VIS 1 and IR 1 Inter-modality triplet loss, Indicates that the input comes from VIS 1 and IR 1 Positive sample pairs of modalities and from VIS 1 The triplet loss calculated from the negative sample pairs of the modality, Indicates that the input comes from VIS 1 and IR 1 Positive sample pairs of modalities and from IR 1 The triplet loss calculated by the negative sample of the modality, VIS 1 The modality indicates the first enhanced modality of visible light corresponding to the first enhanced modality image of visible light, that is, the enhanced modality obtained after the visible light modality is enhanced by the first method, IR 1 The mode indicates the infrared first enhanced mode corresponding to the infrared first enhanced mode image, that is, the enhanced mode obtained after the infrared mode is enhanced by the first method. and The available formula is:
[0123]
[0124] Where M represents the number of pedestrians randomly selected in each iteration, b represents the bth pedestrian, c represents the cth pedestrian, ξ is the interval parameter, Indicates VIS 1 The bth pedestrian and IR in the image 1 The Euclidean distance between the c-th pedestrians in the image, Indicates VIS 1 The bth pedestrian and IR in the image 1 The Euclidean distance between the k-th pedestrians in the image, Represents IR 1 The bth pedestrian and VIS in the image 1 The Euclidean distance between the c-th pedestrians in the image, Represents IR 1 The bth pedestrian and VIS in the image 1 The Euclidean distance between the kth pedestrians in the image, and b and c are the same, indicating the same pedestrian; k is different from b and c, indicating different pedestrians, y b Indicates the category label of the b-th pedestrian in the current mode, y c Indicates that y b Category labels of the same person in different modalities, y k Indicates that y b and y c Category labels of different pedestrians, [z] +=max(z,0), where z is the function input.
[0125] Four-element center cluster loss For the input image The features obtained by multi-feature joint progressive learning network processing (i.e. fusion features ) is used to calculate the average center position of the pedestrian position information in the image. The calculation formula is expressed as:
[0126]
[0127] in, Indicates VIS 1 and IR 1 Center cluster loss between modalities, Indicates VIS 1 and IR 2 Center cluster loss between modalities, Indicates VIS 2 and IR 1 Center cluster loss between modalities, Indicates VIS 2 and IR 2 Inter-modal center cluster loss, VIS 2 Mode represents the visible light second enhanced mode corresponding to the visible light second enhanced mode image, IR 2 The modality indicates the infrared second enhanced modality corresponding to the infrared second enhanced modality image.
[0128] V 1 or V 2 Denote it as V, and I 1 or I 2 Recorded as I, the center cluster loss It is expressed as follows:
[0129]
[0130] Among them, d represents the d-th category, g represents the g-th category, and ρ is a preset parameter. represents the average center position of the VIS fusion features corresponding to the pedestrians in the d-th category, represents the average center position of the IR fusion features corresponding to the pedestrian of the d-th category, and VIS is VIS 1 or VIS 2 , IR is IR 1 or ID 2 , It represents the average center position of the VIS or IR fusion features corresponding to the pedestrians in the g-th category. In actual calculation, n is substituted into the formula with V and I respectively for calculation. ∥AB∥2 represents the Euclidean distance between objects A and B.
[0131]
[0132] in, is the average center position of the first enhanced modal fusion feature of visible light, that is, V is V 1 , VIS is VIS 1 c calculated when V , when you need to calculate the c corresponding to a specific pedestrian V (For example ), the first enhanced modal fusion feature of visible light containing the corresponding pedestrian is taken for calculation, K represents the number of images under each pedestrian category, Represents the visible light first enhanced modality fusion feature corresponding to the j-th image.
[0133] in, is the average center position of the second enhanced modality fusion feature of visible light, that is, V is V 2 , VIS is VIS 2 c calculated when V , when you need to calculate the c corresponding to a specific pedestrian V (For example ), the second enhanced modal fusion feature of visible light containing the corresponding pedestrian is taken for calculation. Represents the visible light second enhanced modality fusion feature corresponding to the j-th image.
[0134] in, is the average center position of the infrared first enhanced modality fusion feature, that is, I is I 1 , IR is IR 1 c calculated when I , when you need to calculate the c corresponding to a specific pedestrian I (For example ), the infrared first enhanced modal fusion feature containing the corresponding pedestrian is taken for calculation. Represents the infrared first enhanced modality fusion feature corresponding to the j-th image.
[0135] in, is the average center position of the infrared second enhanced modality fusion feature, that is, I is I 2 , IR is IR 2 c calculated when I , when you need to calculate the c corresponding to a specific pedestrian I (For example ), the infrared second enhanced modal fusion feature containing the corresponding pedestrian is taken for calculation. Represents the infrared second enhanced modality fusion feature corresponding to the j-th image.
[0136] (7) The query image and the images in the gallery are matched after their features are extracted by the model.
[0137] In the inference phase, given a query image in natural light (infrared) mode and an infrared (natural light) image library, this example uses the image as the input of the model and projects these images into the same feature space. Based on the fusion features corresponding to visible light And the fusion features corresponding to infrared Calculate the matching degree between the query image and the images in the image library, and extract the top-k images with the highest matching degree in the image library as the re-identification results. Splicing, Splicing, based on the existing feature matching method, measures the similarity between the two modalities after splicing as the matching degree between the two modalities.
[0138] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0139] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.
Claims
1. A cross-modal person re-identification method based on multi-feature layered fusion learning, characterized by: The cross-modal person re-identification method based on multi-feature layered fusion learning includes: Step S1: Given a visible light modality image / infrared modality image as a query image and an image library containing infrared modality images / visible light modality images; Step S2: loading a multi-feature hierarchical fusion learning network, wherein the multi-feature hierarchical fusion learning network includes a modal feature enhancement module, a four-stream Resnet-50 backbone network model, and a time-series-based feature fusion module; Step S3: performing two modal enhancements on the visible light modal image and the infrared modal image respectively by the modal feature enhancement module to obtain a first visible light enhanced modal image, a second visible light enhanced modal image, a first infrared enhanced modal image, and a second infrared enhanced modal image; Step S4: input the visible light first enhanced modality image, the visible light second enhanced modality image, the infrared first enhanced modality image, and the infrared second enhanced modality image into a four-stream Resnet-50 backbone network model, and output feature maps C2, C3, C4, and C5 of different sizes; Step S5: Input the feature maps C2, C3, C4, and C5 of different sizes into the time-series-based feature fusion module for hierarchical time-series fusion. The time-series-based feature fusion module operates as follows: The feature maps C3 and C4 are respectively unified into the same size as the feature map C2 through 1×1 convolution and upsampling, and are denoted as feature f 3 and f 4 , and record the feature map C2 as feature f 2 , the feature map C5 is unified into the same size as the feature map C2 through 1×1 convolution and upsampling, and is recorded as feature f ori ; First, the feature f t Passing into LSTM 2 The network is processed, t∈[2,T] is the level, and the highest level T=4, and then the output result is fused with the original feature to obtain the fused feature It is expressed as follows: Then the obtained features Perform attention processing to obtain features It is expressed as follows: The final features With feature f ori Further integration can be expressed as follows: Among them, SE t represents the attention processing operation for the t-th level feature, represents the fusion feature of the tth level, f out represents the fusion feature finally output by the time-series-based feature fusion module, and f out = represents the first enhanced modality fusion feature of visible light, represents the visible light second enhanced modality fusion feature, represents the infrared first enhanced modality fusion feature, represents the infrared second enhanced modality fusion feature; Step S6: Fusion features based on visible light And the fusion features corresponding to infrared Calculate the matching degree between the query image and the images in the image library, and extract the top-k images with the highest matching degree in the image library as the re-identification results.
2. The cross-modal person re-identification method based on multi-feature layered fusion learning according to claim 1 is characterized in that: The method of performing two modal enhancements on the visible light modal image and the infrared modal image to obtain a first visible light enhanced modal image, a second visible light enhanced modal image, a first infrared enhanced modal image, and a second infrared enhanced modal image comprises: For the visible light modality image X vis The first enhancement method is to first enhance the original visible light modality image X vis The three channels x r ,x g ,x b Perform random weighting to obtain the degenerate graph X ds , and then compared with the original visible light modality image X vis Perform weighted fusion to obtain the first enhanced modality image of visible light It is expressed as follows: x ds =α1×x r +α2×x g +α3×x b Among them, x r ,x g ,x b They are the visible light modality images X vis The R channel, G channel and B channel of α1, α2, α3 are random weighted weights, belonging to [0, 1], and α1+α2+α3=1, β1 and β2 are weighted fusion weights, belonging to [0, 1], and β1+β2=1; For the visible light modality image X vis The second enhancement method: in the visible light modality image X vis Randomly select one channel from the three channels and add the information of the other two channels to the randomly selected channel with a preset coefficient, which is expressed as follows: in, is the visible light second enhanced modality image, RandSelect is the random selection operation, is the image X in the visible light modality vis Randomly select one of the three channels of is the visible light modality image X vis The other two channels except the selected channel among the three channels (x r ,x g ,x b ) is the visible light modality image X vis The three channels of γ1, γ2, and γ3 are preset coefficients, belonging to [0,1], and γ1+γ2+γ3=1, 0<γ2+γ3≤0.15; For the infrared modality image X ir The first enhancement method: channel x of the infrared modality image ir The pixel value is multiplied and expressed as follows: in, is the infrared first enhanced modality image, δ is the multiplication coefficient, which belongs to [0.01, 0.5]; For the infrared modality image X ir The second enhancement method is to randomly flip and / or randomly erase the image, which can be expressed as follows: in, is the infrared second enhanced modality image, RandErase represents the random erasing function, and RandFlip represents the random flipping function.
3. The cross-modal person re-identification method based on multi-feature layered fusion learning according to claim 1 is characterized in that: The four-stream Resnet-50 backbone network model includes four branches that process visible light first enhanced modality images, visible light second enhanced modality images, infrared first enhanced modality images, and infrared second enhanced modality images, respectively. Each branch contains a subnetwork with independent parameters, denoted as E vis1 、E vis2 、E ir1 and E ir2 , are all implemented by the first layer of the Resnet-50 network; The four-stream Resnet-50 backbone network model also contains a sub-network E with shared parameters. S , implemented by the remaining layers of the Resnet-50 network except the first layer.
4. The cross-modal person re-identification method based on multi-feature layered fusion learning according to claim 3 is characterized in that: The method inputs the visible light first enhanced modality image, the visible light second enhanced modality image, the infrared first enhanced modality image, and the infrared second enhanced modality image into a four-stream Resnet-50 backbone network model, and outputs feature maps C2, C3, C4, and C5 of different sizes, including: Visible light first enhanced modality image Input subnetwork E vis1 ; The visible light second enhanced mode image Input subnetwork E vis2 ; Infrared first enhanced modality image Input subnetwork E ir1 ; Infrared second enhanced modality image Input subnetwork E ir2 ; Take subnetwork E vis1 、E vis2 、E ir1 and E ir2 The output of is spliced to obtain the feature map C1; Input the feature map C1 into the sub-network E S , get the sub-network E S The feature maps C2, C3, C4 and C5 output by each layer.
5. The cross-modal person re-identification method based on multi-feature layered fusion learning according to claim 1 is characterized in that: The training process of the multi-feature hierarchical fusion learning network is as follows: Obtaining a training data set divided into a natural light set and an infrared set, wherein the natural light set contains visible light modality images, and the infrared set contains infrared modality images; The natural light set and the infrared set are input into the multi-feature hierarchical fusion learning network, and the joint optimization loss is calculated according to the output of the multi-feature hierarchical fusion learning network to perform training updates.
6. The cross-modal person re-identification method based on multi-feature layered fusion learning according to claim 5 is characterized in that: The calculation of the joint optimization loss includes: in, To jointly optimize the loss, represents the heterocentric cluster loss, Indicates id loss, represents the triplet loss between modalities, represents the four-element center cluster loss; The id is lost The loss is calculated for the features obtained by passing the four-stream Resnet-50 backbone network model of the visible light first enhanced modality image and the infrared first enhanced modality image. The calculation formula is as follows: Among them, P is the number of categories in the training data set, a represents the ath category, q a Indicates the actual label distribution probability of the a-th category, p a represents the distribution probability of the a-th category prediction, ε is a constant, and y represents the category label of the pedestrian; The heterocentric cluster loss The calculation formula is as follows: Among them, Q is the number of batches in the training phase and The number of images in each class, f() is the output before the fully connected layer of the four-stream Resnet-50 backbone network model, mean[AB] is the mean operation of the difference between object A and object B, represents the i-th visible light first enhanced modality image in the batch during the training phase, represents the i-th infrared first enhanced modality image in the batch during the training phase; The triplet loss between the modalities The calculation formula is expressed as: in, Indicates VIS 1 and IR 1 Inter-modality triplet loss, Indicates that the input comes from VIS 1 and IR 1 Positive sample pairs of modalities and from VIS 1 The triplet loss calculated from the negative sample pairs of the modality, Indicates that the input comes from VIS 1 and IR 1 Positive sample pairs of modalities and from IR 1 The triplet loss calculated by the negative sample of the modality, VIS 1 Mode represents the visible light first enhanced mode corresponding to the visible light first enhanced mode image, IR 1 Mode represents the infrared first enhanced mode corresponding to the infrared first enhanced mode image. and The available formula is: Where M represents the number of pedestrians randomly selected in each iteration, b represents the bth pedestrian, c represents the cth pedestrian, ξ is the interval parameter, Indicates VIS 1 The bth pedestrian and IR in the image 1 The Euclidean distance between the c-th pedestrians in the image, Indicates VIS 1 The bth pedestrian and IR in the image 1 The Euclidean distance between the k-th pedestrians in the image, Represents IR 1 The bth pedestrian and VIS in the image 1 The Euclidean distance between the c-th pedestrians in the image, Represents IR 1 The bth pedestrian and VIS in the image 1 The Euclidean distance between the kth pedestrians in the image, and b and c are the same, indicating the same pedestrian; k is different from b and c, indicating different pedestrians, y b Indicates the category label of the b-th pedestrian in the current mode, y c Indicates that y b Category labels of the same person in different modalities, y k Indicates that y b and y c Category labels of different pedestrians, [z] + =max(z,0), z is the function input; The four-element center cluster loss The calculation formula is expressed as: in, Indicates VIS 1 and IR 1 Center cluster loss between modalities, Indicates VIS 1 and IR 2 Center cluster loss between modalities, Indicates VIS 2 and IR 1 Center cluster loss between modalities, Indicates VID 2 and IR 2 Inter-modal center cluster loss, VIS 2 Mode represents the visible light second enhanced mode corresponding to the visible light second enhanced mode image, IR 2 The mode represents the infrared second enhanced mode corresponding to the infrared second enhanced mode image; V 1 or V 2 Denote it as V, and I 1 or I 2 Recorded as I, the center cluster loss It is expressed as follows: Among them, d represents the d-th category, g represents the g-th category, and ρ is a preset parameter. represents the average center position of the VIS fusion features corresponding to the pedestrians in the d-th category, represents the average center position of the IR fusion features corresponding to the pedestrian of the d-th category, and VIS is VIS 1 or VIS 2 , IR is IR 1 or IR 2 , represents the average center position of the VIS or IR fusion features corresponding to the pedestrian of the g-th category, and ||AB||2 represents the Euclidean distance between objects A and B.