Cross-modal pedestrian re-identification method and system

Abstract

The invention relates to a cross-modal pedestrian re-identification method and system, and the method comprises the steps: randomly selecting a plurality of pedestrians from an infrared visible light data set in each training batch, and enabling each pedestrian to comprise four visible light images and four infrared images; preprocessing the visible light image and the infrared image; inputting the preprocessed visible light image and infrared image into a deep learning model, and extracting visible light features, infrared features and mixed modal features through the deep learning model; total loss is constructed, whether training of the deep learning model is finished is judged through the total loss, and if the total loss is lower than a preset threshold value, training is finished; if the total loss is higher than a preset threshold value, continuing training until the total loss is lower than the preset threshold value; and identifying a pedestrian from the visible light image or the infrared image to be identified by using the trained deep learning model. According to the invention, cross-modal pedestrian re-identification can be effectively carried out through the constructed deep learning model.

Description

Technical Field

[0001] The present invention relates to the technical field of pedestrian re-identification, and in particular to a cross-modal pedestrian re-identification method and system. Background Art

[0002] In recent years, person re-identification has become an important task in the field of computer vision and video surveillance. By comparing pedestrian images from different cameras, it can automatically identify and match the same individuals. It has been widely used in intelligent monitoring, video retrieval, intelligent transportation and other fields. However, traditional pedestrian re-identification methods usually rely on single-modality visual feature extraction and show obvious limitations in cross-modal and low-light scenarios. For example, there are significant modal differences between infrared and visible light images, making feature alignment and cross-modal matching very challenging.

[0003] To solve this problem, the infrared-visible light cross-modal pedestrian re-identification method based on deep learning shows superior performance and can better extract and align cross-modal features, thereby significantly improving recognition accuracy. Infrared-visible light cross-modal pedestrian re-identification achieves cross-modal pedestrian recognition by utilizing the correlation between visible light and infrared images, which is of great significance in all-weather monitoring scenarios, especially at night or in low light conditions. Therefore, it has gradually become an important research direction in the field of computer vision.

[0004] As an important research direction in the field of computer vision, infrared-visible cross-modal person re-identification aims to reduce the modal differences between infrared image and visible light image features to achieve cross-modal pedestrian matching and recognition. However, infrared-visible cross-modal person re-identification faces some challenges, the main difficulties of which can be summarized as the following two points: huge modal differences and less cross-modal training data.

[0005] 1. Huge modal differences

[0006] There are significant modal differences in visual features between infrared images and visible light images. Visible light images are based on natural light reflection, which can provide rich color information, texture details and contour features. Infrared images are based on the thermal radiation of objects, which do not contain color information and mainly reflect the temperature distribution of objects. This significant difference in modality makes the appearance style features of the same person in infrared images and visible light images very different, making it difficult to directly match them using traditional feature extraction methods.

[0007] Therefore, in the implementation of cross-modal person re-identification, these heterogeneous visual features need to be aligned to narrow the modality gap. There are currently two common methods. One is the feature-level method, which attempts to project the visible light and infrared features extracted by the backbone network into the same embedding space, and perform matching retrieval in the projected new feature space, thereby reducing the impact of modality differences. However, due to the large modality difference, it is difficult for these methods to directly project cross-modal images into a feature space that can align the features of infrared and visible light images. Even if feature mapping is achieved, many beneficial shared features will be lost in the conversion process between the original features and the new features. The other is the image-level method, specifically using a generative adversarial network to generate infrared images and visible light images into images of the same modality for feature extraction, or using a generative adversarial network to fill in the missing modality, such as converting infrared (or visible light) images into visible light (or infrared) images, and finally performing feature fusion with the existing modality and performing image matching in the fused feature space. However, due to the large modality difference and the lack of infrared-visible light image pairs, it is difficult to generate cross-modal images, and they are usually accompanied by large noise.

[0008] 2. Less cross-modal training data

[0009] Another major difficulty is the scarcity of cross-modal training data. Compared with the single-modal visible light pedestrian re-identification task, the infrared-visible light cross-modal dataset needs to contain paired datasets of both infrared and visible light images, but the acquisition cost of such cross-modal datasets is high and the data annotation process is complicated. In actual monitoring scenarios, especially at night, it is easier to obtain infrared images, but the corresponding visible light images are often missing, which leads to a shortage of infrared and visible light paired data.

[0010] The problem of insufficient data directly affects the generalization ability of deep learning models. Deep learning models usually require a large amount of labeled data for training in order to learn effective cross-modal features. Especially for deep learning models whose backbone networks are Transformer and its variants, since they do not have spatial or temporal inductive biases and have large parameters, they need more data to give full play to their structural advantages, establish their global dependencies, and increase the generalization of the model. However, due to the scarcity of cross-modal datasets, the model is prone to overfitting during training, resulting in reduced recognition performance in practical applications. In order to alleviate this problem, in infrared-visible light cross-modal pedestrian re-identification, one type of method uses data enhancement techniques such as channel erasing, channel exchange, random erasing, grayscale enhancement, etc. to add new samples. These methods are only based on the original image for data enhancement, and the newly added samples lack diversity. The second type of method uses generative adversarial networks to synthesize new samples or generates more samples of the same modality by mixing local features of pedestrians. However, these methods are all to enhance the data within the modality, and it is still difficult to completely solve the challenges brought by data scarcity.

[0011] In summary, the existing technology cannot effectively solve the large modal differences between infrared image and visible light image features, and there is little cross-modal training data, resulting in poor cross-modal pedestrian matching and recognition effects. Summary of the invention

[0012] To this end, the technical problem to be solved by the present invention is to overcome the problem in the prior art that there are large modal differences between the features of infrared images and visible light images, and there is less cross-modal training data, resulting in poor cross-modal pedestrian matching and recognition effects.

[0013] In order to solve the above technical problems, the present invention provides a cross-modal pedestrian re-identification method, comprising:

[0014] Step S1: In each training batch, a number of pedestrians are randomly selected from the infrared and visible light dataset, and each pedestrian includes 4 visible light images and 4 infrared images;

[0015] Step S2: preprocessing the visible light image and the infrared image;

[0016] Step S3: inputting the preprocessed visible light image and infrared image into a deep learning model, and extracting visible light features, infrared features, and mixed modal features through the deep learning model;

[0017] Step S4: construct a total loss, and judge whether the deep learning model training is completed by the total loss. If the total loss is lower than a preset threshold, the training is completed; if the total loss is higher than the preset threshold, the training is continued until the total loss is lower than the preset threshold, wherein,

[0018] The total loss includes a local maximum average difference loss constructed based on the visible light feature and the infrared feature, an inter-modal divergence loss constructed based on the visible light feature and the infrared feature, a circle loss constructed based on the visible light feature, the infrared feature and the mixed modal feature, and a cross entropy loss constructed based on the visible light feature, the infrared feature and the mixed modal feature;

[0019] Step S5: Use the trained deep learning model to identify pedestrians from the visible light image or infrared image to be identified.

[0020] In one embodiment of the present invention, the formula of the local maximum average difference loss constructed based on the visible light feature and the infrared feature in step S4 is:

[0021]

[0022] Where H represents the regenerated Hilbert space, φ(·) represents the regenerated Hilbert space transformation mapping, and N id Represents the total number of categories of pedestrian identities, N V represents the number of visible light images in the training batch, f i V represents the features of the i-th visible light image, represents f i V The probability weight of belonging to category k, N I represents the number of infrared images in the training batch, represents the features of the jth infrared image, express The probability weight of belonging to category k, ||ab|| 2 Represents the square of the distance between a and b.

[0023] In one embodiment of the present invention, the formula of the inter-modal divergence loss constructed based on the visible light feature and the infrared feature in step S4 is:

[0024]

[0025] Among them, N V 、N I Represent the number of visible light images and infrared images in the training batch, respectively, and C V , C I Represent visible light image classifier and infrared image classifier respectively, represents the feature of the jth infrared image, f i V Represents the features of the i-th visible light image.

[0026] In one embodiment of the present invention, the formula of the circle loss constructed based on the visible light feature, infrared feature and mixed modal feature in step S4 is:

[0027]

[0028] in, s p =w y f m / (||w y ||||f m ||),m∈{V,I,H}, s p Respectively represent the intra-class similarity score and the inter-class similarity score, w j , w y Represent the classification weight of non-target and target, respectively, w j , w y Respectively represent w j and w y The transpose of m represents the image feature belonging to modality m, where m is the modality to which the image feature belongs, V, I, and H represent visible light modality, infrared modality, and mixed modality, respectively, and ||·|| represents the L2 norm. Represent the first and second weight coefficients of the circular loss, Δ n , Δ p , γ are the first, second and third hyperparameters respectively.

[0029] In one embodiment of the present invention, the cross entropy loss constructed based on the visible light feature, infrared feature and mixed modal feature in step S4 is specifically:

[0030] The visible light features, infrared features and mixed modal features are passed through the classification layer to obtain their respective category scores, and the cross entropy loss is constructed based on their respective category scores. The formula is:

[0031]

[0032] Among them, N V , N I , N H Respectively represent the number of visible light images, infrared images and mixed modality images in the training batch, N id The total number of categories representing pedestrian identities, y i,k Indicates the true category score that the i-th image belongs to the k-th pedestrian, It represents the predicted category score that the i-th image belongs to the k-th pedestrian, m is the modality to which the image feature belongs, V, I, and H represent the visible light modality, infrared modality, and mixed modality, respectively.

[0033] In one embodiment of the present invention, the deep learning model in step S3 includes a first stage, a mixed modality generation module, a second stage, a third stage and a fourth stage connected in sequence, each stage includes at least one Swin Transformer block that appears in pairs, and each Swin Transformer block that appears in pairs includes a first sub-block and a second sub-block that are connected in sequence, wherein:

[0034] The first sub-block includes a first normalization layer LN, a window-based multi-head self-attention layer W-MSA, a second normalization layer LN, and a first multi-layer perceptron MLP connected in sequence, and the input of the first normalization layer LN and the output of the window-based multi-head self-attention layer W-MSA are added by matrix elements as the input of the second normalization layer LN, and the input of the second normalization layer LN and the first multi-layer perceptron MLP are added by matrix elements as the input of the second sub-block;

[0035] The second sub-block includes a third normalization layer LN, a multi-head self-attention layer SW-MSA based on a shift window, a fourth normalization layer LN, and a second multi-layer perceptron MLP connected in sequence, and the input of the third normalization layer LN and the output of the multi-head self-attention layer SW-MSA based on a shift window are added as the input of the fourth normalization layer LN after matrix elements are added, and the input of the fourth normalization layer LN and the second multi-layer perceptron MLP are added as the output of the paired SwinTransformer block;

[0036] In the first stage, a Linear Embedding module is set before the first Swin Transformer block. The Linear Embedding module is used to convert the image into patches required for the Swin Transformer block operation. In the second, third, and fourth stages, a Patching Merging module is set before the first Swin Transformer block. The Patching Merging module is used to reduce the number of patches and increase the dimension of the patches to form a hierarchical representation of features.

[0037] In one embodiment of the present invention, the visible light image features and the infrared image features outputted in the first stage are used by the mixed modality generation module to obtain mixed modality image features, including:

[0038] Suppose the feature of each visible light image output in the first stage is x V , infrared image feature is x I ;

[0039] x VThrough the first dilated convolution with dilation rates of 1, 2, and 4, the results are added and averaged, and then through a fully connected layer and a ReLU activation function, the generated features of the visible light part are obtained;

[0040] x I The generated features of the infrared part are obtained through the second dilated convolution with dilation rates of 1, 2, and 4, and through a fully connected layer and a ReLU activation function;

[0041] In order to extract the common scene-level context information of infrared images and visible light images and generate features for background features, x V and x I The common background features are obtained by performing the dot product of the two pixels through the same adaptive average pooling and 1x1 convolution. Then, the features are adjusted to the same size as the image features output in the first stage through bilinear interpolation, and the global background generation features are obtained through the ReLU activation function.

[0042] Finally, the generated features of the visible light part, the generated features of the infrared part and the generated features of the global background are added and averaged to obtain the final mixed-modal image features;

[0043] In order to keep the final mixed modal features consistent, the modal consistency loss is introduced into the total loss, and the formula is:

[0044]

[0045] Among them, l 2 (·) represents L2 regularization, f t H represents the tth image of the generated mixed modality, f s H represents the sth image of the generated mixed modality, ||·|| 2 represents the Euclidean distance, and n is the number of mixed modes of the same pedestrian.

[0046] In one embodiment of the present invention, the hybrid modality generation module further includes: performing adversarial learning to generate the visible light image feature x output in the first stage. V The mixed modality image features generated by the mixed modality generation module and the infrared image features x output by the first stage I Feature alignment with the mixed modality image features generated by the mixed modality generation module, including:

[0047] Narrowing the visible light image feature x through the generator and discriminator V The mixed modality image features and infrared image features x generated by the mixed modality generation module IThe modality difference between the mixed modality image features generated by the mixed modality generation module, where

[0048] The generator consists of three cascaded fully connected layers. There is a ReLU activation function between the fully connected layers to increase the nonlinearity of the generator. The visible light image feature x output in the first stage V and infrared image feature x I After adding a compensating embedding vector with the same length and width as the original input image and dimension 1, the forged mixed modal features are obtained after passing through their respective generators;

[0049] The discriminator is a Patch Gan discriminator, which is based on the visible light image feature x V Or infrared image feature x I The forged mixed modal features and the mixed modal image features generated by the mixed modal generation module are input into the discriminator, and the adversarial loss about the discriminator is constructed in the total loss function, and the adversarial loss is used to judge whether the forged mixed modal features are the same as the mixed modal image features generated by the mixed modal generation module, so as to achieve feature alignment. The adversarial loss includes the discriminator loss, the generator loss and the joint adversarial loss, and the formula is:

[0050]

[0051] L GAN =L D +L G ;

[0052] Among them, L D is the discriminator loss, L G is the generator loss, L GAN To jointly combat the loss, N V is the number of visible light images in the training batch, N I is the number of infrared images in the training batch, N H is the number of mixed-modality images in the training batch, D(·) is the discriminator, is the i-th visible light image, is the jth infrared image, is the kth mixed modality image, y V is the label of the visible light image, y I is the label of the infrared image, y H is the mixed-modal label, G V-H is the visible light-mixed mode generator, G I-H It is an infrared-mixed mode generator.

[0053] In one embodiment of the present invention, a component detection and exchange module is further provided after the last Swin Transformer block of the third stage, and the component detection and exchange module includes:

[0054] The global image features G corresponding to the visible light, mixed, and infrared modes output by the last Swin Transformer block in the third stage V , G H , G I , and sending the features output by the first and second stages to the body part detector Component Detector to obtain an initial component local feature composed of several human body local features. The initial component local feature will pass through a component prediction network Predictor Network, which is composed of a fully connected layer and a Sigmoid function, and is used to predict the score of each human body local feature in the initial component local feature. The human body local feature in the initial component local feature is replaced according to the predicted score: if the human body local feature score of the mixed modality is smaller than the human body local feature scores of the other two modalities, the human body local feature with the largest score in the other two modalities is replaced with the human body local feature of the mixed modality, so as to replace the overlapping and redundant human body local features in the mixed modality and obtain the component local feature P. V , P H , P I ,in,

[0055] The body part detector Component Detector passes the features output by the first and second stages and the features output by the last Swin Transformer block in the third stage through a 1×1 convolutional neural network and a Sigmoid function to obtain attention masks at different levels, adjusts the attention masks to the same size and multiplies them by pixel to obtain the final mask, and then respectively compares the final mask with the global image features G corresponding to the visible light, mixed, and infrared modes output by the last Swin Transformer block in the third stage. V , G H , G I Multiply them, and finally concatenate them in the dimension to obtain the local features of the initial components;

[0056] The component feature is constructed in the total loss function subject to the component identity loss, and the formula is:

[0057]

[0058] Where N represents the total number of images in the training batch, N c Represents the number of local features of the human body, Nid is the total number of categories of pedestrian identities, y i,k Indicates the true category score of the i-th image belonging to the k-th pedestrian, z i,j,k Indicates the predicted category score that the jth human local feature in the i-th image belongs to the k-th pedestrian.

[0059] In order to solve the above technical problems, the present invention provides a cross-modal person re-identification system, comprising:

[0060] Selection module: used to randomly select several pedestrians from the infrared and visible light dataset in each training batch. Each pedestrian includes 4 visible light images and 4 infrared images.

[0061] Preprocessing module: used to preprocess visible light images and infrared images;

[0062] Feature extraction module: used to input the preprocessed visible light image and infrared image into the deep learning model, and extract visible light features, infrared features and mixed modal features through the deep learning model;

[0063] Construction module: used to construct the total loss, and judge whether the training of the deep learning model is completed by the total loss. If the total loss is lower than the preset threshold, the training is completed; if the total loss is higher than the preset threshold, the training is continued until the total loss is lower than the preset threshold, wherein,

[0064] The total loss includes a local maximum average difference loss constructed based on the visible light feature and the infrared feature, an inter-modal divergence loss constructed based on the visible light feature and the infrared feature, a circle loss constructed based on the visible light feature, the infrared feature and the mixed modal feature, and a cross entropy loss constructed based on the visible light feature, the infrared feature and the mixed modal feature;

[0065] Recognition module: used to use the trained deep learning model to identify pedestrians from the visible light image or infrared image to be identified.

[0066] To solve the above technical problems, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above cross-modal pedestrian re-identification method when executing the computer program.

[0067] In order to solve the above technical problems, the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above cross-modal pedestrian re-identification method are implemented.

[0068] The above technical solution of the present invention has the following advantages compared with the prior art:

[0069] The deep learning model constructed by the present invention involves a novel three-modal branch structure, which can effectively increase the number of training samples and facilitate subsequent pedestrian re-identification by learning the generation of mixed modalities from visible light images and infrared images;

[0070] The present invention introduces a mixed modality generation module into a deep learning model, and realizes the alignment between visible light images and mixed modality images, and between infrared images and mixed modality images by learning modality embedding and using a generative adversarial network, thereby effectively reducing the modality difference between the infrared modality and the visible light modality during actual retrieval.

[0071] The present invention introduces component detection and exchange modules into the deep learning model to explore subtle differences in images while reducing the difficulty of alignment. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings.

[0073] Figure 1 is a flow chart of the method of the present invention;

[0074] Figure 2 is a schematic diagram of the structure of a deep learning model in an embodiment of the present invention;

[0075] Figure 3 is a schematic diagram of the Swin Transformer block structure in an embodiment of the present invention;

[0076] Figure 4 Schematic diagram of the structure of a hybrid mode generation module in an embodiment of the present invention;

[0077] Figure 5 It is a schematic diagram of the structure of the component detection and exchange module in an embodiment of the present invention. DETAILED DESCRIPTION

[0078] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0079] Embodiment 1

[0080] Reference Figure 1 The present invention relates to a cross-modal pedestrian re-identification method, comprising:

[0081] Step S1: In each training batch, a number of pedestrians are randomly selected from the infrared and visible light dataset, and each pedestrian includes 4 visible light images and 4 infrared images;

[0082] Step S2: preprocessing the visible light image and the infrared image;

[0083] Step S3: inputting the preprocessed visible light image and infrared image into a deep learning model, and extracting visible light features, infrared features, and mixed modal features through the deep learning model;

[0084] Step S4: construct a total loss, and judge whether the deep learning model training is completed by the total loss. If the total loss is lower than a preset threshold, the training is completed; if the total loss is higher than the preset threshold, the training is continued until the total loss is lower than the preset threshold, wherein,

[0085] The total loss includes a local maximum average difference loss constructed based on the visible light feature and the infrared feature, an inter-modal divergence loss constructed based on the visible light feature and the infrared feature, a circle loss constructed based on the visible light feature, the infrared feature and the mixed modal feature, and a cross entropy loss constructed based on the visible light feature, the infrared feature and the mixed modal feature;

[0086] Step S5: Use the trained deep learning model to identify pedestrians from the visible light image or infrared image to be identified. Specifically, the model identifies pedestrians in the infrared image to be identified based on a given visible light image, or identifies pedestrians in the visible light image to be identified based on a given infrared image. It should be noted that since the trained deep learning model in this embodiment has three branches, in actual use, the branch corresponding to the infrared image or visible light image can be selected according to different application scenarios.

[0087] The following is a detailed introduction to this embodiment:

[0088] The deep learning model proposed in the present invention adopts a three-branch network for training, and its basic network is SwinTransformerV2. The three branches of the model are visible light branch, infrared branch and mixed modal branch, and the model parameters of the three branches are shared. The training data of the mixed modal branch is a mixture of visible light images and infrared images, which can effectively increase the number of samples for model training. At the same time, the model can align the visible light image features and infrared image features with the mixed modal features, which can effectively reduce the difference between the modalities and reduce the overfitting of the training samples. In order to explore the subtle differences between images, the model extracts local features by generating masks in the third stage, and gradually replaces the local features of the human body between the three modal branches, thereby replacing the overlapping and redundant local features of the human body in the mixed modality, making the local features of the mixed modality purer. Figure 2 The structural diagram of the entire model is given, and the specific training process of the model is as follows:

[0089] S1. During each batch training, 4 pedestrians are randomly selected from the infrared and visible light dataset, and each pedestrian contains 4 visible light images and 4 infrared images.

[0090] S2. Preprocess all pedestrian images. First, resize all images to 288×144×3, and perform data augmentation. Random cropping, random horizontal flipping, and random erasing are used for visible light images, as well as random enhancement methods such as channel random erasing, random channel swapping enhancement, color jitter enhancement, and spectral jitter enhancement. Only random cropping, random horizontal flipping, and random erasing are used for infrared images. Finally, all images are normalized with a mean of [0.485, 0.456, 0.406] and a standard deviation of [0.229, 0.224, 0.225].

[0091] S3. Input the processed visible light and infrared images into the model to obtain the features of the three modes of visible light, infrared and mixed mode generated and extracted by the model;

[0092] S4. Construct a total loss, and determine whether the deep learning model training is completed by the total loss. If the total loss is lower than a preset threshold, the training is completed; if the total loss is higher than the preset threshold, the training is continued until the total loss is lower than the preset threshold;

[0093] Specifically, the total loss includes local maximum average difference loss, inter-modal divergence loss, circle loss, and cross entropy loss, as follows:

[0094] (1) The extracted visible light feature f V and infrared signature f I The local maximum average difference loss and inter-modal divergence loss are supervised. The local maximum average difference loss function is as follows:

[0095]

[0096] Where H represents the regenerated Hilbert space, φ(·) represents the regenerated Hilbert space transformation mapping, and N id Represents the total number of categories of pedestrian identities, N V represents the number of visible light images in the training batch, f i V represents the features of the i-th visible light image, represents f i V The probability weight of belonging to category k, N I represents the number of infrared images in the training batch, represents the features of the jth infrared image, express The probability weight of belonging to category k, ||ab|| 2 Represents the square of the distance between a and b.

[0097] (2) The inter-modal divergence loss is as follows:

[0098]

[0099] Among them, N V 、N I Represent the number of visible light images and infrared images in the training batch, respectively, and C V , C I Represent visible light image classifier and infrared image classifier respectively, represents the feature of the jth infrared image, f i V Represents the features of the i-th visible light image.

[0100] (3) The visible light feature f V 、Infrared characteristicsf I and the mixed modal features f H Let’s supervise the circle loss together:

[0101]

[0102] in, s p =w y f m / (||w y ||||f m ||),m∈{V,I,H}, s p Respectively represent the intra-class similarity score and the inter-class similarity score, w j , w y Represent the classification weight of non-target and target, respectively, w j , w y Respectively represent w j and w y The transpose of m represents the image feature belonging to modality m, where m is the modality to which the image feature belongs, V, I, and H represent visible light modality, infrared modality, and mixed modality, respectively, and ||·|| represents the L2 norm. Represent the first and second weight coefficients of the circular loss, Δ n , Δ p , γ are the first, second and third hyperparameters respectively, and Δ n , Δ p The values ​​of , are 0.5, 0.5, and 30 respectively.

[0103] (4) The features of the three modalities are passed through the classification layer (i.e., the fully connected layer, Figure 2 The FC layer in the V 、z I and z H , the category scores are supervised by cross entropy loss, and the loss function formula is as follows:

[0104]

[0105] Among them, N V , N I , N H Respectively represent the number of visible light images, infrared images and mixed modality images in the training batch, N id The total number of categories representing pedestrian identities, y i,k Indicates the true category score that the i-th image belongs to the k-th pedestrian, It represents the predicted category score that the i-th image belongs to the k-th pedestrian, m is the modality to which the image feature belongs, V, I, and H represent the visible light modality, infrared modality, and mixed modality, respectively.

[0106] S5. Use the trained deep learning model to identify pedestrians from the visible light images and infrared images to be identified.

[0107] The specific process of step S2 is:

[0108] S2-1. The deep learning model of this embodiment uses Swin TransformerV2 as the backbone network. The deep learning model includes the first stage (Stage 1), the hybrid modal generation module (HMG Module), the second stage (Stage 2), the third stage (Stage 3), and the fourth stage (Stage 4) which are connected in sequence. The third stage (Stage 3) is also provided with a component detection and switching module (CDS Module). Swin TransformerV2 first divides the obtained visible light and infrared images into blocks, specifically through Figure 2The Patch Partition module (patch partitioning module) in the image is used to divide the image into patches. In order to make each patch fully aware of the features near the patch and allow the model to make full use of the image features, the model uses overlapping sampling blocks. The specific operation is to perform a two-dimensional convolution operation with a step size of 3 and a convolution kernel of 4 on the input image. The input image size received by SwinTransformerV2 is [288, 144, 3]. Through the convolution block operation, 96×48 patches are obtained, and the dimension of each patch is 128. Then, after linear embedding (Linear Embedding module in the first stage), a total of 4608 patches are flattened to obtain image features of size [4608, 128], and then sent to the Swin Transformer block, which is divided into four stages. The number of Swin Transformer blocks in the four stages is [2, 2, 18, 2] respectively.

[0109] In this embodiment, the Swin Transformer blocks composed of W-MSA and SW-MSA always appear in pairs, see Figure 3 , each pair of Swin Transformer blocks includes a first sub-block and a second sub-block connected in sequence, where

[0110] The first sub-block includes a first normalization layer LN, a window-based multi-head self-attention layer W-MSA, a second normalization layer LN, and a first multi-layer perceptron MLP connected in sequence, and the input of the first normalization layer LN and the output of the window-based multi-head self-attention layer W-MSA are added by matrix elements as the input of the second normalization layer LN, and the input of the second normalization layer LN and the first multi-layer perceptron MLP are added by matrix elements as the input of the second sub-block;

[0111] The second sub-block includes a third normalization layer LN, a multi-head self-attention layer SW-MSA based on a shift window, a fourth normalization layer LN, and a second multi-layer perceptron MLP connected in sequence, and the input of the third normalization layer LN and the output of the multi-head self-attention layer SW-MSA based on a shift window are added as the input of the fourth normalization layer LN after matrix elements are added, and the input of the fourth normalization layer LN and the second multi-layer perceptron MLP are added as the output of the paired SwinTransformer block;

[0112] In the first stage, a Linear Embedding module is provided before the first Swin Transformer block. The Linear Embedding module is used to convert the image into the patch required for the Swin Transformer block operation (i.e., the patch obtained by block division is mapped to a new dimension (three-dimensional to two-dimensional, paving effect) for subsequent Swin Transformer block processing). In the second, third, and fourth stages, a Patching Merging module is provided before the first Swin Transformer block. The Patching Merging module is used to reduce the number of patches while increasing the dimension of the patches, which is conducive to forming a hierarchical representation of features. Since the Linear Embedding module and the Patching Merging module belong to the prior art, they are not described in detail in this embodiment.

[0113] The multi-layer perceptron MLP consists of two fully connected layers, and the GELU nonlinear activation function is used between the two fully connected layers.

[0114] Furthermore, the Swin Transformer block composed of W-MSA and SW-MSA always appears in pairs and is divided into two sub-blocks, namely the first sub-block and the second sub-block, through which the relationship between patches in different regions can be established. The output image features of the Swin Transformer block with two sub-blocks are calculated as follows:

[0115]

[0116] in, They are the features of the first sub-block output through the multi-head self-attention layer and the features of the second stage output through the multi-head self-attention layer, They are the features of the first sub-block output by the multi-layer perceptron and the features of the second sub-block output by the multi-layer perceptron, WMSA() is a window-based multi-head self-attention layer, MLP() is a multi-layer perceptron, SWMSA() is a shifted window-based multi-head self-attention layer, m is the modality to which the image feature belongs, V, I, H represent visible light, infrared and mixed modality features respectively, in order to produce hierarchical representation, the patch merging layer is used between Swin Transformer blocks at different stages to reduce the number of patches and increase the dimension of the patches. Assuming that the dimension of each patch is C, the patch merging layer concatenates the features of each group of 2×2 adjacent patches, and uses a linear layer to reduce the dimension of the concatenated features with a dimension of 4C to 2C.

[0117] Please combine Figure 2After the Swin Transformer block in the first stage (Stage 1), the output visible light and infrared modal features will pass through the hybrid modal generation module (HMG Module) to obtain three branches, which output visible light features, hybrid modal image features and infrared image features respectively. Then, the features of the three modalities are sent to the Swin Transformer blocks of the next three stages (Stage2, 3, 4) respectively.

[0118] Please combine Figure 2 , the global image features corresponding to the three modalities output by the last Swin Transformer block in the third stage of the model (Stage 3) (i.e. Figure 2 G in V , G H , G I ), and the features output by the first and second stages (Stage 1, 2) are sent to the component detection and exchange module (CDS Module) to extract the corresponding component local features P V , P H , P I , and the global image feature G V , G H , G I Output together, the global image features of the next three modalities (G V , G H , G I ) and component local features (P V , P H , P I ) will be sent to the Swin Transformer block of the last stage (Stage 4), and finally after feature concatenation (i.e. Figure 2 The "C" after the fourth stage in the figure represents connect) concatenates the respective global image features and component local features in the patch dimension, and passes through an average pooling layer (POOL) and a batch normalization layer (BN) to obtain the final feature output f V 、f I and f H .

[0119] S2-2. Please refer to Figure 4 , the hybrid modal generation module (HMG Module) is as follows:

[0120] For each pedestrian in a training batch, there are 4 visible light images and 4 infrared images. After the hybrid modality generation module (HMG Module), 16 hybrid modality images are generated (1 visible light image can generate hybrid modality images with the corresponding 4 infrared images respectively, then 4 visible light images can generate 16 images with the corresponding 4 infrared images in total). The specific generation process is as follows: First, the visible light image features and infrared image features are added with their respective modality compensation embeddings and restored to the original image ratio. For each visible light image feature x output by the first stage (Stage 1), V and infrared image feature x I In order to capture and generate image information of different scales, it is necessary to convert x V Through the first dilated convolution with dilation rates of 1, 2, and 4, the results are added and averaged, and then through a fully connected layer and a ReLU activation function, the generated features of the visible light part are obtained. I A similar operation is performed, and the second dilated convolution with the same dilation rates of 1, 2, and 4 but different parameters from the first dilated convolution is used, and a fully connected layer and a ReLU activation function are used to obtain the generated features of the infrared part. In addition, in order to extract the common scene-level context information of the infrared image and the visible light image, and to generate features for the background features, it is necessary to convert x V and x I The common background features are obtained by performing the same adaptive average pooling and 1x1 convolution and performing the dot product of the two on the pixels. Then, the bilinear interpolation is adjusted to the same size as the image features output by the first stage (Stage 1), and the global background generated features are obtained by ReLU activation. Finally, the generated features of multiple parts are fused, that is, the generated features of the visible light part, the generated features of the infrared part, and the generated features of the global background are added and averaged to obtain the final mixed-modal image features. In order to keep the generated mixed-modal image features consistent, this embodiment introduces the modal consistency loss in the total loss:

[0121]

[0122] Among them, l 2 (·) represents L2 regularization, f t H represents the tth image of the generated mixed modality, f s H represents the sth image of the generated mixed modality, ||·|| 2 represents the Euclidean distance, and n is the number of mixed modes of the same pedestrian.

[0123] For the 16 generated mixed-modal image features, the subsequent Patch Gan discriminator is used to obtain the predicted labels, and the 4 mixed-modal image features closest to the mixed-modal labels are selected as the final output (the 4 mixed-modal image features are selected here to ensure that the number of visible light images / infrared images is the same as the original input deep learning model). It should be noted that the mixed-modal generation module (HMG Module) can not only generate the final mixed-modal image features, but also generate the mixed-modal image features based on the visible light image features x V The forged mixed modal features, based on the infrared image features x I In short, the hybrid modal generation module (HMGModule) can generate features of three branches: visible light, hybrid modal, and infrared, and input the features of the three branches generated into the second stage (stage 2) respectively.

[0124] The model then uses adversarial learning to align the features between visible light-mixed modality and infrared-mixed modality. The core of the alignment part is the generator and the discriminator D. Through the mutual game between the generator and the discriminator D, the modal differences between visible light, infrared image features and mixed modality features are narrowed. The purpose of the generator is to make the mixed modality features generated by other modalities as close as possible to the real mixed modality features, and the purpose of the discriminator D is to distinguish the mixed modality features from the features of the other two modalities. The addition of each modality compensation embedding (i.e. Figure 4 The visible light features and infrared features of the two embedding vectors (the length and width of the original input image and the dimension of the embedding vector) are passed through the generator (Generator), which is divided into a visible light-mixed mode generator G V-H and infrared-hybrid mode generator G I-H Each generator consists of three fully connected layers in cascade. There is a ReLU activation function between the fully connected layers to increase the nonlinearity of the generator. After the features pass through their respective generators, they obtain fake mixed-modal features. Figure 4 D V-H and D I-H ) is a classic Patch Gan discriminator structure, which is based on the visible light image feature x V Or infrared image feature x IThe forged mixed modal features and the mixed modal image features generated by the mixed modal generation module are input into the corresponding discriminator D, and the adversarial loss about the discriminator D is constructed in the total loss function. The adversarial loss is used to judge whether the forged mixed modal features are the same as the mixed modal image features generated by the mixed modal generation module (the adversarial loss is less than the preset value, indicating that they are the same, indicating that the actually generated mixed modal image features are valid; the adversarial loss is greater than the preset value, indicating that they are different, indicating that the actually generated mixed modal image features are not ideal), so as to achieve feature alignment. The adversarial loss includes the discriminator loss, the generator loss and the joint adversarial loss, and the formula is as follows:

[0125]

[0126]

[0127] L GAN =L D +L G ;

[0128] Among them, L D is the discriminator loss, L G is the generator loss, L GAN To jointly combat the loss, N V is the number of visible light images in the training batch, N I is the number of infrared images in the training batch, N H is the number of mixed-modality images in the training batch, D(·) is the discriminator, is the i-th visible light image, is the jth infrared image, is the kth mixed modality image, y V is the label of the visible light image, y I is the label of the infrared image, y H is the mixed-modal label, G V-H is the visible light-mixed mode generator, G I-H It is an infrared-mixed mode generator.

[0129] Finally, the model sends the aligned infrared image, visible light image and mixed modal image to the subsequent model for processing.

[0130] S2-3. Please refer to Figure 5 , the component detection and switching module (CDS Module) is as follows:

[0131] In this embodiment, a component detection and exchange module is further provided after the last Swin Transformer block in the third stage, which converts the global image features G corresponding to the three modes output by the last Swin Transformer block in the third stage into V , G H , G I , and sending the features outputted by the first and second stages to the body part detector ComponentDetector to obtain initial component local features composed of several human body local features. The initial component local features will pass through a component prediction network Predictor Network, which is composed of a fully connected layer and a Sigmoid function, and is used to predict the score of each human body local feature in the initial component local features. The human body local features in the initial component local features are replaced according to the predicted scores. Since the generated initial component local features inevitably have overlapping and redundant human body local features, this embodiment uses Figure 5 The swapping module in the hybrid modality replaces the local features of the human body corresponding to the hybrid modality with the local features of the human body detected by the infrared and visible light modalities in an asymptotic manner: if the score of the local features of the human body in the hybrid modality is smaller than the scores of the local features of the human body in the other two modalities, the local features of the human body with the largest scores in the other two modalities are replaced with the local features of the human body in the hybrid modality, so as to replace the overlapping and redundant local features of the human body in the hybrid modality and obtain the local features of the components P. V , P H , P I , making the local features of the mixed mode purer. It should be noted that the initial component local features corresponding to the infrared branch and the visible light branch are different from the final component local features P I , P V The same, only the local features of the initial components corresponding to the mixed mode branches are changed.

[0132] Specifically, the body part detector Component Detector combines the features output by the first and second stages, as well as the global image features (G V , G H , G I) are respectively passed through a 1×1 convolutional neural network (CNN) and a Sigmoid function to obtain attention masks at different levels, and the attention masks are adjusted to the same size and multiplied pixel by pixel to obtain the final mask (Mask), and then the final mask (Mask) is combined with the visible light, mixed, and infrared modal data (G V , G H , G I ) are multiplied and finally concatenated in the dimension to obtain the local features of the initial components.

[0133] The loss of local features of the human body is constructed in the total loss function, and the formula is:

[0134]

[0135] Where N represents the total number of images in the training batch, N c Represents the number of local features of the human body, N id is the total number of categories of pedestrian identities, y i,k Indicates the true category score of the i-th image belonging to the k-th pedestrian, z i,j,k Indicates the predicted category score that the jth human local feature in the i-th image belongs to the k-th pedestrian.

[0136] The experimental analysis is as follows:

[0137] 1. Implementation details

[0138] This embodiment implements a deep learning model based on the PyTorch framework on an NVIDIA GTX 4090D GPU. The backbone network uses Swin TransformerV2 pre-trained on ImageNet22K, and removes the last classification layer of SwinTransformerV2 to directly output a 1024-dimensional feature. During testing, this embodiment uses cosine similarity to compare the distance between the features of pedestrians in the query set and the gallery set.

[0139] 2. Sampling strategy

[0140] During training, this embodiment randomly samples 4 pedestrians in each batch, and extracts 4 visible light images and 4 infrared images for each pedestrian to form small batch training samples. Before sending them to the model for training, all images are resized to 288×144×3, and data enhancement is performed at the same time. Random cropping, random horizontal flipping and random erasing, as well as random channel erasing, random channel exchange enhancement, color jitter enhancement, and spectral jitter enhancement are used for visible light images. Only random cropping, random horizontal flipping and random erasing are used for infrared images. Finally, all images are normalized with a mean of [0.485, 0.456, 0.406] and a standard deviation of [0.229, 0.224, 0.225] to reduce the risk of overfitting in training.

[0141] During the test, this embodiment uses a batch size of 64 to extract features of the query set and the gallery set respectively. Before being fed into the model, the visible light image and the infrared image are resized to 288×144×3, and then the images are normalized with a mean of [0.485, 0.456, 0.406] and a standard deviation of [0.229, 0.224, 0.225].

[0142] 3. Training parameter settings

[0143] The model was trained for a total of 40 epochs. Before starting training, all parameters of the network model were initialized randomly using Kaiming initialization. During training, the SGD optimizer was used to update the parameters, with a momentum of 0.9, a weight decay coefficient of, and the initial gradient descent learning rate was set to 0.001. Warmup was used to warm up the learning rate, with 5 warm-up iterations and a warm-up factor of 0.01. The warm-up learning rate used a linear growth method. At the same time, the learning rate cosine decay strategy was followed, and the learning rate eventually decayed to 0.002 times the initial learning rate.

[0144] 4. Performance comparison with common models

[0145] The deep learning model constructed in this embodiment is compared with the current advanced model on the SYSU-MM01 dataset. When the model is used for pedestrian re-identification, no re-ranking, fusion ranking or other processing techniques for retrieval ranking are adopted. It is not difficult to find from Table 1 that the deep learning model constructed in this embodiment has a better effect.

[0146] Table 1 Model comparison table

[0147]

[0148] Embodiment 2

[0149] This embodiment provides a cross-modal person re-identification system, including:

[0150] Selection module: used to randomly select several pedestrians from the infrared and visible light dataset in each training batch. Each pedestrian includes 4 visible light images and 4 infrared images.

[0151] Preprocessing module: used to preprocess visible light images and infrared images;

[0152] Feature extraction module: used to input the preprocessed visible light image and infrared image into the deep learning model, and extract visible light features, infrared features and mixed modal features through the deep learning model;

[0153] Construction module: used to construct the total loss, and judge whether the training of the deep learning model is completed by the total loss. If the total loss is lower than the preset threshold, the training is completed; if the total loss is higher than the preset threshold, the training is continued until the total loss is lower than the preset threshold, wherein,

[0154] The total loss includes a local maximum average difference loss constructed based on the visible light feature and the infrared feature, an inter-modal divergence loss constructed based on the visible light feature and the infrared feature, a circle loss constructed based on the visible light feature, the infrared feature and the mixed modal feature, and a cross entropy loss constructed based on the visible light feature, the infrared feature and the mixed modal feature;

[0155] Recognition module: used to use the trained deep learning model to identify pedestrians from the visible light image or infrared image to be identified.

[0156] Embodiment 3

[0157] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the cross-modal pedestrian re-identification method described in Embodiment 1 are implemented.

[0158] Embodiment 4

[0159] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the cross-modal pedestrian re-identification method described in the first embodiment are implemented.

[0160] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0161] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0162] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0163] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0164] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0165] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.

Claims

1. A cross-modal person re-identification method, characterized by: include: Step S1: In each training batch, a number of pedestrians are randomly selected from the infrared and visible light dataset, and each pedestrian includes 4 visible light images and 4 infrared images; Step S2: preprocessing the visible light image and the infrared image; Step S3: inputting the preprocessed visible light image and infrared image into a deep learning model, and extracting visible light features, infrared features, and mixed modal features through the deep learning model; Step S4: construct a total loss, and judge whether the deep learning model training is completed by the total loss. If the total loss is lower than a preset threshold, the training is completed; if the total loss is higher than the preset threshold, the training is continued until the total loss is lower than the preset threshold, wherein, The total loss includes a local maximum average difference loss constructed based on the visible light feature and the infrared feature, an inter-modal divergence loss constructed based on the visible light feature and the infrared feature, a circle loss constructed based on the visible light feature, the infrared feature and the mixed modal feature, and a cross entropy loss constructed based on the visible light feature, the infrared feature and the mixed modal feature; Step S5: Use the trained deep learning model to identify pedestrians from the visible light image or infrared image to be identified.

2. The cross-modal person re-identification method according to claim 1, characterized in that: The formula of the local maximum average difference loss constructed based on the visible light feature and the infrared feature in step S4 is: Where H represents the regenerated Hilbert space, φ(·) represents the regenerated Hilbert space transformation mapping, and N id Represents the total number of categories of pedestrian identities, N V represents the number of visible light images in the training batch, represents the features of the i-th visible light image, express The probability weight of belonging to category k, N I represents the number of infrared images in the training batch, represents the features of the jth infrared image, express The probability weight of belonging to category k, ||ab|| 2 Represents the square of the distance between a and b.

3. The cross-modal person re-identification method according to claim 1, characterized in that: The formula of the inter-modal divergence loss constructed based on the visible light feature and the infrared feature in step S4 is: Among them, N V 、N I Represent the number of visible light images and infrared images in the training batch, respectively, and C V , C I Represent visible light image classifier and infrared image classifier respectively, represents the features of the jth infrared image, Represents the features of the i-th visible light image.

4. The cross-modal person re-identification method according to claim 1, characterized in that: The formula of the circle loss constructed based on the visible light feature, infrared feature and mixed modal feature in step S4 is: in, s p =w y f m / (||w y ||||f m ||),m∈{V,I,H}, s p Respectively represent the intra-class similarity score and the inter-class similarity score, w j , w y Represent the classification weight of non-target and target, respectively, w j , w y Respectively represent w j and w y The transpose of m represents the image feature belonging to modality m, where m is the modality to which the image feature belongs, V, I, and H represent visible light modality, infrared modality, and mixed modality, respectively, and ||·|| represents the L2 norm. Represent the first and second weight coefficients of the circular loss, Δ n , Δ p , γ are the first, second and third hyperparameters respectively.

5. The cross-modal person re-identification method according to claim 1, characterized in that: The cross entropy loss constructed based on the visible light feature, infrared feature and mixed modal feature in step S4 is specifically: The visible light features, infrared features and mixed modal features are passed through the classification layer to obtain their respective category scores, and the cross entropy loss is constructed based on their respective category scores. The formula is: Among them, N V , N I , N H Respectively represent the number of visible light images, infrared images and mixed modality images in the training batch, N id Represents the total number of categories of pedestrian identities, y i,k Indicates the true category score that the i-th image belongs to the k-th pedestrian, It represents the predicted category score that the i-th image belongs to the k-th pedestrian, m is the modality to which the image feature belongs, V, I, and H represent the visible light modality, infrared modality, and mixed modality, respectively.

6. The cross-modal person re-identification method according to claim 1, characterized in that: The deep learning model in step S3 includes a first stage, a mixed modality generation module, a second stage, a third stage, and a fourth stage connected in sequence, each stage includes at least one Swin Transformer block that appears in pairs, and each Swin Transformer block that appears in pairs includes a first sub-block and a second sub-block that are connected in sequence, wherein: The first sub-block includes a first normalization layer LN, a window-based multi-head self-attention layer W-MSA, a second normalization layer LN, and a first multi-layer perceptron MLP connected in sequence, and the input of the first normalization layer LN and the output of the window-based multi-head self-attention layer W-MSA are added by matrix elements as the input of the second normalization layer LN, and the input of the second normalization layer LN and the first multi-layer perceptron MLP are added by matrix elements as the input of the second sub-block; The second sub-block includes a third normalization layer LN, a multi-head self-attention layer SW-MSA based on a shift window, a fourth normalization layer LN, and a second multi-layer perceptron MLP connected in sequence, and the input of the third normalization layer LN and the output of the multi-head self-attention layer SW-MSA based on a shift window are added as the input of the fourth normalization layer LN after matrix elements are added, and the input of the fourth normalization layer LN and the second multi-layer perceptron MLP are added as the output of the paired Swin Transformer block; In the first stage, a Linear Embedding module is set before the first Swin Transformer block. The Linear Embedding module is used to convert the image into patches required for the Swin Transformer block operation. In the second, third, and fourth stages, a Patching Merging module is set before the first Swin Transformer block. The Patching Merging module is used to reduce the number of patches and increase the dimension of the patches to form a hierarchical representation of features.

7. The cross-modal person re-identification method according to claim 6, characterized in that: The visible light image features and infrared image features outputted in the first stage are used to obtain mixed modality image features through the mixed modality generation module, including: Suppose the feature of each visible light image output in the first stage is x V , infrared image feature is x I ; x V Through the first dilated convolution with dilation rates of 1, 2, and 4, the results are added and averaged, and then through a fully connected layer and a ReLU activation function, the generated features of the visible light part are obtained; x I The generated features of the infrared part are obtained through the second dilated convolution with dilation rates of 1, 2, and 4, and through a fully connected layer and a ReLU activation function; In order to extract the common scene-level context information of infrared images and visible light images and generate features for background features, x V and x I The common background features are obtained by performing the dot product of the two pixels through the same adaptive average pooling and 1x1 convolution. Then, the features are adjusted to the same size as the image features output in the first stage through bilinear interpolation, and the global background generation features are obtained through the ReLU activation function. Finally, the generated features of the visible light part, the generated features of the infrared part and the generated features of the global background are added and averaged to obtain the final mixed-modal image features; In order to keep the final mixed modal features consistent, the modal consistency loss is introduced into the total loss, and the formula is: Among them, l2(·) represents L2 regularization, f t H represents the tth image of the generated mixed modality, f s H represents the sth image of the generated mixed mode, ||·||2 represents the Euclidean distance, and n is the number of mixed modes of the same pedestrian.

8. The cross-modal person re-identification method according to claim 7, characterized in that: The hybrid modality generation module further includes: performing adversarial learning to generate the visible light image feature x output in the first stage. V The mixed modality image features generated by the mixed modality generation module and the infrared image features x output by the first stage I Feature alignment with the mixed modality image features generated by the mixed modality generation module, including: Narrowing the visible light image feature x through the generator and discriminator V The mixed modality image features and infrared image features x generated by the mixed modality generation module I The modality difference between the mixed modality image features generated by the mixed modality generation module, where The generator consists of three cascaded fully connected layers. There is a ReLU activation function between the fully connected layers to increase the nonlinearity of the generator. The visible light image feature x output in the first stage V and infrared image feature x I After adding a compensating embedding vector with the same length and width as the original input image and dimension 1, the forged mixed modal features are obtained after passing through their respective generators; The discriminator is a Patch Gan discriminator, which is based on the visible light image feature x V Or infrared image feature x I The forged mixed modal features and the mixed modal image features generated by the mixed modal generation module are input into the discriminator, and the adversarial loss about the discriminator is constructed in the total loss function, and the adversarial loss is used to judge whether the forged mixed modal features are the same as the mixed modal image features generated by the mixed modal generation module, so as to achieve feature alignment. The adversarial loss includes the discriminator loss, the generator loss and the joint adversarial loss, and the formula is: L GAN =L D +L G ; Among them, L D is the discriminator loss, L G is the generator loss, L GAN To jointly combat the loss, N V is the number of visible light images in the training batch, N I is the number of infrared images in the training batch, N H is the number of mixed-modality images in the training batch, D(·) is the discriminator, is the i-th visible light image, is the jth infrared image, is the kth mixed modality image, y V is the label of the visible light image, y I is the label of the infrared image, y H is the mixed-modal label, G V-H is the visible light-mixed mode generator, G I-H It is an infrared-mixed mode generator.

9. The cross-modal person re-identification method according to claim 7, characterized in that: A component detection and exchange module is also provided after the last Swin Transformer block of the third stage, and the component detection and exchange module includes: The global image features G corresponding to the visible light, mixed, and infrared modalities output by the last Swin Transformer block in the third stage V , G H , G I , and sending the features output by the first and second stages to the body part detector Component Detector to obtain an initial component local feature composed of several human body local features. The initial component local feature will pass through a component prediction network Predictor Network, which is composed of a fully connected layer and a Sigmoid function, and is used to predict the score of each human body local feature in the initial component local feature. The human body local feature in the initial component local feature is replaced according to the predicted score: if the human body local feature score of the mixed modality is smaller than the human body local feature scores of the other two modalities, the human body local feature with the largest score in the other two modalities is replaced with the human body local feature of the mixed modality, so as to replace the overlapping and redundant human body local features in the mixed modality and obtain the component local feature P. V , P H , P I ,in, The body part detector Component Detector passes the features output by the first and second stages and the features output by the last Swin Transformer block in the third stage through a 1×1 convolutional neural network and a Sigmoid function to obtain attention masks at different levels, adjusts the attention masks to the same size and multiplies them by pixel to obtain the final mask, and then respectively compares the final mask with the global image features G corresponding to the visible light, mixed, and infrared modes output by the last Swin Transformer block in the third stage. V , G H , G I Multiply them, and finally concatenate them in the dimension to obtain the local features of the initial components; The loss of local features of the human body is constructed in the total loss function, and the formula is: Where N represents the total number of images in the training batch, N c Represents the number of local features of the human body, N id is the total number of categories of pedestrian identities, y i,k Indicates the true category score of the i-th image belonging to the k-th pedestrian, z i,j,k Indicates the predicted category score that the jth human local feature in the i-th image belongs to the k-th pedestrian.

10. A cross-modal person re-identification system, characterized by: include: Selection module: used to randomly select several pedestrians from the infrared and visible light dataset in each training batch. Each pedestrian includes 4 visible light images and 4 infrared images. Preprocessing module: used to preprocess visible light images and infrared images; Feature extraction module: used to input the preprocessed visible light image and infrared image into the deep learning model, and extract visible light features, infrared features and mixed modal features through the deep learning model; Construction module: used to construct the total loss, and judge whether the training of the deep learning model is completed by the total loss. If the total loss is lower than the preset threshold, the training is completed; if the total loss is higher than the preset threshold, the training is continued until the total loss is lower than the preset threshold, wherein, The total loss includes a local maximum average difference loss constructed based on the visible light feature and the infrared feature, an inter-modal divergence loss constructed based on the visible light feature and the infrared feature, a circle loss constructed based on the visible light feature, the infrared feature and the mixed modal feature, and a cross entropy loss constructed based on the visible light feature, the infrared feature and the mixed modal feature; Recognition module: used to use the trained deep learning model to identify pedestrians from the visible light image or infrared image to be identified.

Images