Three-modal pedestrian re-identification method based on instance normalization style fusion module

Abstract

The application discloses a three-modal pedestrian re-identification method based on an instance normalization style fusion module and belongs to the technical field of pedestrian re-identification. The application extracts style information of a shallow network by using instance normalization, extracts a focus by combining a channel attention mechanism of an excitation squeeze mode, and fuses domain-invariant style information of two modes to form an input of a third branch. Then, the input is sent to a parameter-shared three-branch network for common feature extraction. In the aspect of feature space constraint, a novel three-modal triplet loss function is further proposed, which strengthens the inter-class constraint by limiting the modalities of positive samples, negative samples and anchor points. Experimental results prove that the application can effectively improve the precision of the pedestrian re-identification method by enhancing the feature style information and strengthening the modal constraint.

Description

TECHNICAL FIELD

[0001] The application relates to a three-modal pedestrian re-identification method based on an instance normalization style fusion module and belongs to the technical field of pedestrian re-identification. BACKGROUND

[0002] Infrared and visible light pedestrian re-identification needs to process images of two modalities, and thus needs to face not only the differences between different modalities but also the differences between modalities. Starting from the fundamental problem of cross-modal pedestrian re-identification, the visible differences in the data set become the difficulty and key point of the challenge. Different imaging methods of infrared and visible light have different forms on the same identity of pedestrians. Even if the posture changes of pedestrians caused by cross-camera shooting are not considered, the styles of the two kinds of images are almost different, and the data distribution also has obvious deviation. Therefore, in the learning process, the computer needs to accurately extract the common features under the two modalities to help establish a suitable calculation model. The main research methods of cross-modal pedestrian re-identification include a single-flow or multi-flow neural network-based method and another method mainly using a generative adversarial network.

[0003] There are many more fine-grained learning methods in the existing pedestrian re-identification methods. The team of Tay (Tay C-P, Roy S, Yap K-H. Aanet: Attribute attention network for person re-identifications [C]. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, 2019: 7134-7143.) combines global and local information and attribute information, and the proposed method can adaptively learn the multi-task loss weight. Hao et al. (Hao Y, Wang N, Gao X, et al. Dual-alignment feature embedding for cross-modality person re-identification [C]. Proceedings of the 27th ACM International Conference on Multimedia, Bloomington, 2019: 57-65.) designed a dual-alignment feature embedding method to extract discriminant modality invariant features, namely spatial alignment and pattern alignment. And partial level feature extraction is used to extract fine-grained camera invariant information. By introducing a distribution loss function and a correlation loss function, the embedding features of the visible and infrared modalities are aligned.

[0004] However, the above method mainly integrates identity classification and human auxiliary information into a framework to consider human details, often loses the unique subtle differences of the shallow modal, cannot effectively extract the invariant information between the two domains, and mostly depends on additional labels and prior artificial analysis information, treats these information equally and thus cannot adaptively learn in the cross-modal environment, so a lot of necessary information is missing, resulting in low accuracy of pedestrian re-identification. SUMMARY

[0005] In order to improve the accuracy of the pedestrian re-identification method under the infrared and visible light cross-modal, the application provides a three-modal pedestrian re-identification method based on an instance normalization style fusion module, and the technical solution is as follows:

[0006] The first object of the application is to provide a cross-modal pedestrian re-identification method, comprising: constructing a pedestrian re-identification double-flow network based on a ResNet50 network, inputting cross-modal images into the pedestrian re-identification double-flow network respectively, and outputting a recognition result;

[0007] The training process of the pedestrian re-identification double-flow network comprises:

[0008] Step one: constructing a set of infrared images to be trained and a set of visible light images, and initializing the pedestrian re-identification double-flow network;

[0009] Step two: extracting visible light modal specific features x rgb and infrared modal specific features x ir from the first convolution block Conv1 of the ResNet50 network respectively through the infrared images and the visible light images to be trained;

[0010] Step three: performing instance normalization on the visible light modal specific features x rgb and the infrared modal specific features x ir respectively, and calculating the feature difference before and after normalization respectively;

[0011] Step four: performing squeezing excitation operation on the feature difference respectively to obtain domain invariant style features under visible light modal and infrared modal, and obtaining the third modal feature by averaging the domain invariant style features of the two modalities;

[0012] Step five: extracting three-modal shared features from through the remaining part of the pedestrian re-identification double-flow network;

[0013] Step six: taking the pooling features of the three modalities as the input of the three-modal triplet loss, and the calculation method is as follows:

[0014]

[0015] where P represents the number of selected pedestrian parts in each batch, K represents the number of photos of each pedestrian, p represents a threshold value of the loss function, f represents a feature after global average pooling, wherein the subscripts m1 and m2 represent three modalities, i.e., infrared ir, visible light rgb, and generated mixed modalities c, the superscripts i and j represent the identity of the pedestrian, and a represents an anchor point;

[0016] Finally, classification and identity loss constraints are performed by a classifier:

[0017]

[0018] where N represents the number of batches; p i represents the probability that the pedestrian with identity i is correctly identified;

[0019] Step seven: training the pedestrian re-identification double-flow network based on the common constraint of the three-modal triplet loss and the identity loss.

[0020] Optionally, the instance normalization of step three includes:

[0021]

[0022]

[0023]

[0024] where μ ic and σ ic are the average value and variance of each channel across the spatial dimension, γ and β are affine parameters of the normalized feature, x represents the input feature, H represents the height of the image, W represents the width of the image, x ichw represents an element located at the cth channel of the ith feature map with spatial coordinates (h, w), and ε represents a constant greater than 0 to ensure that the denominator is not equal to zero.

[0025] Optionally, the squeezing excitation operation of step four includes:

[0026]

[0027] where X m represents the difference between the normalized and unnormalized features, i and j represent the horizontal and vertical coordinates of the pixel points of the features on the spatial plane, and the feature T m obtained after the squeezing operation is reduced in size to C×1×1, and C represents the number of channels.

[0028] Then sent into two full connection layers, respectively, on the channel dimension reduction and dimension operation, to adaptively calibrate the channel features, the first full connection layer will be compressed channel features, after the Relu activation, the second full connection layer will restore it to the original dimension, finally through the Sigmoid activation function to obtain the channel response weight;

[0029] Two full connection layer weights are shared to ensure that the channel response is consistent, the specific formula is as follows:

[0030] a m =σ(W1(W0(T m )))

[0031] Wherein, represents the first full connection operation, then represents the second full connection layer, and gamma indicates the proportion of channel number reduction, and sigma indicates the Sigmoid activation operation;

[0032] The obtained channel response weight a m is multiplied back to X m and the feature after instance normalization is added back Obtain the style feature with channel attention That is, the domain invariant style features of two modalities;

[0033] Finally, the domain invariant style features of two modalities are averaged to obtain the third modal feature

[0034] Optionally, the total loss function of step seven is: L total =L id +L T-Tri .

[0035] Optionally, the preprocessing of the training set image in step one includes adjusting the image size, zero padding and horizontal flip random cropping.

[0036] Optionally, the pedestrian re-identification double flow network in step one is initialized with pre-trained ImageNet parameters, and the convolution layer step of the last layer is changed from 2 to 1.

[0037] Optionally, the optimizer of the model training of the pedestrian re-identification double flow network adopts the stochastic gradient descent method.

[0038] Optionally, the momentum of the stochastic gradient descent method is set to 0.9, and the weight decay is set to 0.0005.

[0039] A second object of the present application is to provide a cross-modal pedestrian re-identification device, comprising an image acquisition device, a memory and a processor, characterized in that the memory stores instructions executed by the processor, when the instructions are executed by the processor, the cross-modal pedestrian re-identification device implements the cross-modal pedestrian re-identification method according to any one of the above.

[0040] A third object of the present application is to provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the cross-modal pedestrian re-identification method according to any one of the above.

[0041] The present application has the following advantages:

[0042] The present application uses an instance normalization style fusion module to solve the problem of style difference in cross-modal feature extraction by a shallow network. The instance normalization can better preserve individual difference features between images than batch normalization. The shallow style information of infrared and visible light is extracted in the network without sharing parameters, and the adaptive channel attention is used to further obtain the modal invariant feature information after fusion style, which is used as the input of the third branch to assist the learning of the other two branches. In addition, to solve the problem that the original ordinary triplets for double modal are slightly insufficient due to the generation of the fusion modal, the present application also proposes a triple modal triplet loss for the three-branch structure. The pooling features learned by the three branches are further constrained within and between classes, and the ability to constrain feature differences in the feature space is stronger than that of the ordinary triplet loss.

[0043] Experimental results show that the recognition network trained based on the loss function proposed in the present application has higher precision, effectively improving the accuracy of the cross-modal pedestrian re-identification method. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0045] Figure 1 Structure diagram of the triple modal pedestrian re-identification method based on the instance normalization style fusion module.

[0046] Figure 2 Structure diagram of the instance normalization style fusion module.

[0047] Figure 3Figures for visualization experimental results on SYSU-MM01 dataset, where (a) is the case that the query image is an infrared image; (b) is the case that the query image is an infrared image. DETAILED DESCRIPTION

[0048] In order to make the objects, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0049] Embodiment one:

[0050] The embodiment provides a cross-modal pedestrian re-identification method, referring to Figure 1 The method comprises the following steps: constructing a pedestrian re-identification double-flow network based on a ResNet50 network, inputting cross-modal images into the pedestrian re-identification double-flow network respectively, and outputting a recognition result.

[0051] The training process of the pedestrian re-identification double-flow network comprises the following steps:

[0052] Step one: constructing a set of infrared images to be trained and a set of visible light images, and initializing the pedestrian re-identification double-flow network;

[0053] Step two: extracting visible light modal unique features x rgb and infrared modal unique features x ir from the first convolution block Conv1 of the ResNet50 network respectively from the infrared images to be trained and the visible light images;

[0054] Step three: performing instance normalization on the visible light modal unique features x rgb and the infrared modal unique features x ir respectively, and calculating the feature difference before and after normalization respectively;

[0055] Step four: performing squeeze-and-excitation operation on the feature difference respectively to obtain domain-invariant style features under the visible light modal and the infrared modal, and obtaining a third modal feature by averaging the domain-invariant style features of the two modalities;

[0056] Step five: extracting a three-modal shared feature from by the remaining part of the pedestrian re-identification double-flow network;

[0057] Step six: taking the pooling features of the three modalities as the input of the three-modal triplet loss, and the calculation method is as follows:

[0058]

[0059] Wherein, P represents the number of selected pedestrian body parts per batch, K represents the number of photos of each pedestrian, p represents the threshold value of the loss function, f represents the feature after global average pooling, wherein the subscripts m1, m2∈{rgb, ir, c} represent three modalities, that is, infrared ir, visible light rgb and generated mixed modal c, the superscripts i and j represent the identity of the pedestrian, and a represents the anchor point;

[0060] Finally, the classifier is classified and the identity loss is constrained:

[0061]

[0062] Wherein, N represents the number of batches; p i represents the probability that the pedestrian with identity i is correctly identified;

[0063] Step seven: based on the common constraint of the three modalities triplet loss and the identity loss, the training of the pedestrian re-identification double-flow network is completed.

[0064] Embodiment two:

[0065] The embodiment provides a three-modal pedestrian re-identification method based on an instance normalization style fusion module. For the classification task of pedestrian images, improvements are made on the existing double-flow network framework.

[0066] The method of the embodiment comprises:

[0067] Step A.1, select an infrared image set and a visible light image set of a data set to be trained, specifically comprising:

[0068] In the embodiment, SYSU-MM01 is selected as the training data set, the training batch size is set to 32, 16 infrared images and 16 visible light images are selected, that is, four different pedestrians are selected per batch, and four photos of each pedestrian are selected. The size of the input image is adjusted to 384x128. For data enhancement, random cropping with zero padding and horizontal flipping is adopted.

[0069] Step B.1, set the pre-training network.

[0070] ResNet50 is used as the backbone network, the pre-trained ImageNet parameters are used to initialize the double-flow network, and the convolution layer step of the last layer is changed from 2 to 1. The double-flow network comprises five stages, the structure of the first convolution block is relatively simple, and the structures of the last four convolution blocks are relatively similar. In order to enhance the representation of modal specific features, the network parameters of the first convolution block and the first Layer convolution block of the two branches are different. The remaining convolution blocks share parameters.

[0071] Step C.1, set the learning rate and the optimizer.

[0072] The optimizer used for model training is stochastic gradient descent (SGD), with momentum set to 0.9 and weight decay set to 0.0005. The initial learning rate during training is set to 0.01.

[0073] Step D.1: Perform repeated iterative training of the network, setting the training epochs to 80, saving the model every 20 epochs, and finally outputting the best model up to the current iteration number. This includes the following steps:

[0074] (1) Input RGB dataset D rgb and the Infrared dataset D ir .

[0075] (2) In dataset D rgb and D ir The modality-specific features x are extracted using the first convolutional block Conv1 of ResNet50. rgb and x ir .

[0076] (3) Obtained through instance normalization and Subtract the original feature from the current feature:

[0077]

[0078]

[0079] (4) Feature X rgb and X ir The features are fed into the excitation module to obtain channel-weighted features and then added back to the instance-normalized features:

[0080]

[0081]

[0082] (5) Fusing features from two modalities and Obtain domain-invariant style information, i.e., third-modal features.

[0083] (6) From the remaining phase of the network Modality-sharing features are extracted from them.

[0084] (7) Calculate the triplet loss and identity loss for the three modalities. The triplet loss threshold is set to 0.3.

[0085]

[0086]

[0087] The result under the joint constraint of the two loss functions is as follows:

[0088] L total = L id + L T-Tri

[0089] (9) Complete back propagation.

[0090] (10) Optimize network parameters.

[0091] Step E.1, test the dataset under the global search mode.

[0092] The step E.1 includes:

[0093] (1) Input the test batch image under the global search mode, and respectively pass through the ResNet50 backbone network part for forward propagation.

[0094] (2) Perform feature matching on the query image and the library image.

[0095] (3) Use the standard cumulative matching characteristic (CMC), Rank-1 (R1), Rank-10 (R10), Rank-20 (R20) value and average precision (mAP) as the evaluation standard. Rank-r represents the probability that r pictures with the highest confidence in the search have correct results. The abscissa of CMC is the rank-r precision, which measures the matching probability that the true value result appears in the first n retrieval tasks. mAP measures the retrieval performance when multiple matching images appear in the library.

[0096] Step F.1, test the dataset under the indoor search mode.

[0097] The step F.1 includes:

[0098] (1) Input the test batch image under the indoor search mode, and respectively pass through the ResNet50 backbone network part for forward propagation.

[0099] (2) Perform feature matching on the query image and the library image.

[0100] (3) Use the standard cumulative matching characteristic (CMC), Rank-1 (R1), Rank-10 (R10), Rank-20 (R20) value and average precision (mAP) as the evaluation standard. Rank-r represents the probability that r pictures with the highest confidence in the search have correct results. The abscissa of CMC is the rank-r precision, which measures the matching probability that the true value result appears in the first n retrieval tasks. mAP measures the retrieval performance when multiple matching images appear in the library.

[0101] Example 3:

[0102] This embodiment provides a trimodal pedestrian re-identification method based on an instance normalization style fusion module. The method improves upon the existing two-stream network framework for pedestrian image classification tasks.

[0103] The method in this embodiment includes:

[0104] Step A.1: Select the infrared image set and visible light image set to be trained from a certain dataset.

[0105] RegDB was chosen as the training dataset, with a batch size of 32. Sixteen infrared images and 16 visible light images were selected, meaning four different pedestrians were chosen per batch, and four images were selected for each pedestrian. The input image size was adjusted to 384×128. For data augmentation, random cropping with zero padding and horizontal flipping was used.

[0106] Step B.1: Set up the pre-trained network.

[0107] Using ResNet50 as the backbone, the two-stream network is initialized with pre-trained ImageNet parameters, and the stride of the last convolutional layer is changed from 2 to 1. The two-stream network consists of five stages. The first convolutional block has a relatively simple structure, while the structures of the following four convolutional blocks are more similar. To enhance the representation of modality-specific features, the network parameters of the first convolutional block and the first layer convolutional block of each branch are different. The remaining convolutional blocks share parameters.

[0108] Step C.1: Set the learning rate and optimizer.

[0109] The optimizer used for model training is stochastic gradient descent (SGD), with momentum set to 0.9 and weight decay set to 0.0005. The initial learning rate during training is set to 0.01.

[0110] Step D.1: Perform repeated iterative training of the network, setting the training epochs to 80. Save the model every 20 epochs and finally output the best model up to the current iteration number. The training process includes the following steps:

[0111] (1) Input RGB dataset D rgb and the Infrared dataset D ir .

[0112] (2) In dataset D rgb and D ir The modality-specific features x are extracted using the first convolutional block Conv1 of ResNet50. rgb and x ir .

[0113] (3) Obtain by instance normalization and Subtract the original features after the instance normalization:

[0114]

[0115]

[0116] (4) Send the features X rgb and X ir to the squeeze excitation module respectively to obtain the channel weighted features and add back the instance normalized features:

[0117]

[0118]

[0119] (5) Fuse the features of the two modalities and to obtain the domain invariant style information, i.e., the third modality feature X

[0120] (6) Extract the modality shared feature from X through the remaining stage of the network.

[0121] (7) Calculate the triplet loss and identity loss of the three modalities, and set the triplet loss threshold to 0.3.

[0122]

[0123]

[0124] The result under the joint constraint of the two loss functions is as follows:

[0125] L total = L id + L T-Tri

[0126] (8) Complete backpropagation.

[0127] (9) Optimize the network parameters.

[0128] Step E.1, perform infrared-visible light image testing, including the following steps:

[0129] (1) Take the infrared image as the query image and the visible light image as the library image to input the test batch, and respectively pass through the ResNet50 backbone network part for forward propagation.

[0130] (2) Perform feature matching on the query image and the library image.

[0131] (3) The Cumulative Match Characteristic (CMC), Rank-1 (R1), Rank-10 (R10), Rank-20 (R20) values and mean Average Precision (mAP) are used as evaluation criteria. Rank-r represents the probability that the r-th highest confidence image in the search has the correct result. The horizontal coordinate of CMC is the rank-r accuracy, which measures the matching probability of the true value result appearing in the first n retrieval tasks. mAP measures the retrieval performance when multiple matching images appear in the gallery.

[0132] Step F.1, visible-infrared image test is performed.

[0133] The step F.1 comprises:

[0134] (1) The visible light image is used as the query image, and the infrared image is used as the gallery image to input the test batch, and the ResNet50 backbone network part is used for forward propagation.

[0135] (2) Feature matching is performed on the query image and the gallery image.

[0136] (3) The Cumulative Match Characteristic (CMC), Rank-1 (R1), Rank-10 (R10), Rank-20 (R20) values and mean Average Precision (mAP) are used as evaluation criteria. Rank-r represents the probability that the r-th highest confidence image in the search has the correct result. The horizontal coordinate of CMC is the rank-r accuracy, which measures the matching probability of the true value result appearing in the first n retrieval tasks. mAP measures the retrieval performance when multiple matching images appear in the gallery.

[0137] (1) The experimental results of the present application are as follows, Table 1 and Table 2 respectively show the comparison results of the pedestrian re-identification method based on the instance normalization style fusion (INSFT) module of the present application and other existing methods on different data sets.

[0138] Table 1 Comparison results with other methods on SYSU-MM01 dataset (%)

[0139]

[0140]

[0141] (Best results are bold, second underlined)

[0142] On the SYSU-MM01 dataset, Table 1 presents the experimental performance of the global search mode and the indoor search mode, using Rank-1, Rank10, Rank20 and mAP to evaluate the network effect.

[0143] Zero-Pad (Wu A, Zheng W, Yu H, et al. RGB-infrared cross-modality person re-identification [C]. Proceedings of the IEEE international Conference on Computer Vision, Hawaii: IEEE, 2017: 5380-5389.) uses a single-stream network for classification, uses deep zero padding to convert infrared and visible light images into dual channels, although the performance is not good, but it is the first method for cross-modal person re-identification, and the INSFT method is far superior to its accuracy. AGW as a cross-modal benchmark method, it fuses non-local attention blocks in the backbone network, which lays the foundation for cross-modal person re-identification, and proposes an important backbone network, the INSFT method exceeds the AGW method (Ye M, Shen J, Lin G, et al. Deep learning for person re-identification: A survey and outlook [J]. IEEE Transactions on Pattern Analysis Machine Intelligence, 2021, 44(6): 2872-2893.) 11.39% in Rank-1, and 8.59% in mAP. TAANet (Huang P, Zhu S, Wang D, et al. Cross-modality person re-identication with triple-attentive feature aggregation [J]. Multimedia Tools Applications, 2022, 81(3): 4455-4473.) proposes a new hybrid attention aggregation learning method to mine the context relationship between modal and cross-modal image levels to promote the feature learning process of the infrared visible light person re-identification task. The experimental comparison on the SYSU-MM01 dataset also shows that the Rank-1 / mAP of the method of the present application is 2.86% / 1.91% higher than that of TAANet in the global search mode, and 0.72% / 0.6% higher in the indoor search mode.

[0144] Table 2 Comparison results with other methods on RegDB dataset (%)

[0145]

[0146]

[0147] (The best result is bold, the second is underlined)

[0148] The experiments with visible light as the query image and with infrared as the query image were carried out on the RegDB dataset respectively, and it can be seen from Table 2 that the universal results of the present application are better than the experimental results on the SYSU-MM01 dataset, and the experimental results with visible light as the query image are slightly higher than those with infrared as the query image. GECNet (Zhong X, Lu T, Huang W, et al. Grayscale Enhancement Colorization Network for Visible-Infrared Person Re-Identification [J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(3): 1418-1430.) learns the corresponding relationship between single-channel infrared images and three-channel visible light images by generating an intermediate grayscale image as auxiliary information, and colors the single-channel infrared image, but the coloring process uses GAN, which introduces more noise, thereby affecting the classification ability of the network. The INSFT method proposed in the present application better avoids this problem, and in the case of visible light as the query image, the experimental results are higher than GECNet, and in the case of infrared as the query image, except that the Rank-20 index is slightly lower, other indexes are better than it. In the Rank-1 and mAP, 87.87 / 85.04% and 85.53 / 83.20% are respectively reached.

[0149] (2) Ablation experiment results:

[0150] To verify the effectiveness of the instance normalization style fusion module of the present application, module ablation and module comparison experiments were carried out under the global search mode of the SYSU-MM01 dataset. As shown in Table 3, wherein B represents the baseline method, that is, a method comprising only the backbone network of ResNet50 and the ordinary triplet loss and identity loss, wherein the parameters of Conv1 and Layer1 of ResNet50 are not shared, and the parameters of the convolution blocks of the remaining stages are shared, 2Branch represents the generation without fusion of the modalities, 3Branch represents the averaging of the two modality features after Conv1 to establish a third branch, INSF represents the instance normalization style fusion module of the present application, 1_2 in the parentheses represents that the insertion position of the module is between the first Conv1 convolution block and Layer1, and 2_3 in the parentheses represents that the insertion position of the module is between Layer1 and Layer2. The experiments prove that the INSF module proposed by the present application is effective and suitable for insertion in the shallow network, because the image contains more rich detail information in the shallow network, and with the increase of the depth of network training, it tends to semantic information, and the effect of instance normalization in the shallow layer is more obvious, so the INSF module can extract the domain invariant style information of the two modalities in the shallower layer.

[0151] Table 3 Ablation experiment results of the instance normalization style fusion module (%)

[0152]

[0153] In addition, experiments were also conducted on the structure of INSF, and different attention strategies used by the following three kinds of INSF were discussed, and in the experiments of the following table, three modality triplet loss was used as a constraint. The experimental results are shown in Table 4.

[0154] Table 4 Experimental results of the instance normalization style fusion module under different strategies (%)

[0155]

[0156] Wherein SE represents an extrusion excitation operation, CAM is a spatial attention mechanism proposed by the article (Woo S, Park J, Lee J-Y, et al. Cbam: Convolutional block attention module [C]. Proceedings of the European Conference on Computer Vision, Munich: Springer, 2018: 3-19.), first obtains a spatial attention map through maximum pooling and global average pooling, then obtains the weight of the attention area of the picture through 7*7 convolution and once Sigmoid activation, and w / o ATT is the result of completely canceling the attention mechanism, only the normalized features of the instance are used as the style features of the third branch. It can be obtained from the experimental results that the extrusion excitation operation can effectively extract domain-invariant information.

[0157] Part of the steps in the embodiments of the present application can be realized by software, and the corresponding software program can be stored in a readable storage medium, such as an optical disc or a hard disk.

[0158] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A cross-modal pedestrian re-identification method, characterized in that, The method includes: constructing a pedestrian re-identification dual-stream network based on the ResNet50 network, inputting cross-modal images into the pedestrian re-identification dual-stream network respectively, and outputting the recognition result; The training process of the pedestrian re-identification dual-stream network includes: Step 1: Construct the infrared image set and the visible light image set to be trained, and initialize the pedestrian re-identification dual-stream network; Step 2: Extract the visible light mode-specific features x from the infrared image and the visible light image to be trained by passing them through the first convolutional block Conv1 of the ResNet50 network. rgb and infrared modal characteristics x ir ; Step 3: Analyze the unique features x of the visible light modes respectively. rgb and infrared modal characteristics x ir Perform instance normalization and calculate the feature difference before and after normalization respectively; Step 4: Perform a squeezing excitation operation on the feature difference to obtain domain-invariant style features in the visible light mode and infrared mode, and average the domain-invariant style features of the two modes to obtain the third mode feature. Step 5: Using the remaining portion of the pedestrian re-identification dual-stream network, from Extracting shared features across the three modes; Step Six: Use the pooling features of the three modalities as input to the trimodal triplet loss, calculated as follows: Where P represents the number of pedestrian identities selected in each batch, K represents the number of photos of each pedestrian, ρ represents the threshold of the loss function, f represents the features after global average pooling, and the subscripts m1, m2∈{rgb,ir,c} represent three modes, namely infrared ir, visible light rgb and the generated mixed mode c, respectively. The superscripts i and j represent the pedestrian identities, and a represents the anchor point. Finally, the classification and identity loss are constrained by the classifier: Where N represents the batch size; p i This represents the probability that pedestrian i is correctly identified; Step 7: Train the pedestrian re-identification dual-stream network based on the joint constraints of the trimodal triplet loss and identity loss.

2. The cross-modal pedestrian re-identification method according to claim 1, characterized in that, The instance normalization in step three includes: Where, μ ic and σ ic These are the mean and variance across the spatial dimensions for each channel. γ and β are affine parameters of the features after scaling and bias normalization operations. x represents the input features, H represents the image height, and W represents the image width. ichw ε represents the element located in the c-th channel of the i-th feature map with spatial coordinates (h, w), and ε represents a constant greater than 0 to ensure that the denominator is not equal to zero.

3. The cross-modal pedestrian re-identification method according to claim 1, characterized in that, The extrusion excitation operation in step four includes: Among them, X m The feature difference before and after normalization is represented by i and j, which represent the x and y coordinates of the pixel points of the feature in the spatial plane, respectively. The feature T obtained after the squeezing operation is... m The size is reduced to C×1×1, where C represents the number of channels; Then, the data is fed into two fully connected layers to perform dimensionality reduction and dimensionality increase operations on the channels, thereby adaptively calibrating the feature responses on the channels. The first fully connected layer compresses the channels of the features and performs ReLU activation. The second fully connected layer restores them to their original dimensions. Finally, the channel response weights are obtained after passing through the Sigmoid activation function. The two fully connected layers share weights to ensure consistent channel responses. The specific calculation formula is shown below: a m =σ(W1(W0(T m ))) in, This represents the first fully connected operation. This represents the second fully connected layer, γ refers to the percentage reduction in the number of channels, and σ represents the Sigmoid activation operation; The obtained channel corresponding weight a m Multiply back X m And add back the features after instance normalization. Style features obtained through channel attention That is, the domain-invariant style features of the two modalities; Finally, the average of the domain-invariant style features of the two modalities is used to obtain the third modal feature.

4. The cross-modal pedestrian re-identification method according to claim 1, characterized in that, The total loss function in step seven is: L total =L id +L T-Tri .

5. The cross-modal pedestrian re-identification method according to claim 1, characterized in that, The preprocessing of the training set images in step one includes: resizing the images, random cropping with zero padding and horizontal flipping.

6. The cross-modal pedestrian re-identification method according to claim 1, characterized in that, Step one initializes the pedestrian re-identification dual-stream network with pre-trained ImageNet parameters and changes the stride of the last convolutional layer from 2 to 1.

7. The cross-modal pedestrian re-identification method according to claim 1, characterized in that, The optimizer for training the pedestrian re-identification dual-stream network model employs stochastic gradient descent.

8. The cross-modal pedestrian re-identification method according to claim 7, characterized in that, The momentum of the stochastic gradient descent method is set to 0.9, and the weight decay is set to 0.0005.

9. A cross-modal pedestrian re-identification device, comprising an image acquisition device, a memory, and a processor, characterized in that, The memory stores instructions that are executed by the processor. When the instructions are executed by the processor, the cross-modal pedestrian re-identification device causes the cross-modal pedestrian re-identification method according to any one of claims 1 to 8 to be implemented.

10. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the cross-modal pedestrian re-identification method according to any one of 1 to 8.

Images