Cross-modal person re-identification method based on cross-modal fusion and modal decorrelation

Abstract

The present invention relates to a cross-modal pedestrian re-identification method based on cross-modal fusion and modal decorrelation, which fully exploits the association between sample images and features of two modalities by using two parts: cross-modal channel fusion and modal decorrelation quintuple loss. As data enhancement, cross-modal channel fusion is equivalent to adding new samples with both RGB and infrared image information to the data set, making the input space of the model more continuous. CCM does not introduce additional parameters or models and can be easily combined with other methods. The method considers the image information of the two modalities at the same time, prompting the model to learn the association between the modalities. The modal decorrelation quintuple loss constrains the distance relationship within the positive sample pairs and the negative sample pairs, thereby reducing the correlation between the features and the modalities to which they belong, and eliminating redundant modal-specific information in the features. The method solves the modal difference problem at the image and feature levels at the same time, and can better improve the performance of cross-modal retrieval.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a pedestrian re-identification method based on cross-modal fusion and modal decorrelation. Background Art

[0002] The task of pedestrian re-identification aims to retrieve specific pedestrians across cameras. In recent years, deep learning technology has achieved remarkable results on a series of computer vision tasks and has been widely used in the field of pedestrian re-identification. In actual scenarios, in order to build an all-weather video surveillance system, infrared cameras are used to collect clear and effective pedestrian images at night. The cross-modal pedestrian re-identification task between RGB and infrared images has emerged and has attracted the attention of more and more researchers.

[0003] Cross-modal person re-identification means that given a pedestrian image of a certain modality, a pedestrian image of another modality with the same identity as the image is retrieved from the gallery. The core difficulty of cross-modal person re-identification lies in the huge modality difference between the two modalities. RGB images are three-channel images, while infrared images contain only one channel, and the amount of information between the two is not equal. In addition, RGB images collect visible light information, while infrared images are produced by collecting infrared light that is invisible to the human eye. The wavelength ranges of the two are very different, and the discriminative color information in single-modal tasks is no longer valid. Due to the interference of modal information that is not related to identity information, the similarity of the same pedestrian in different modalities is greatly reduced, resulting in the intra-class distance of features often being greater than the inter-class distance. Most current methods alleviate modality differences at the image or feature level, but ignore the high correlation between features and their modalities, resulting in a large amount of modality-specific redundant information in the features. Summary of the invention

[0004] In view of the above problems existing in the prior art, the technical problem to be solved by the present invention is: the present invention adopts a public cross-modal pedestrian dataset, based on cross-modal fusion and modal decorrelation, to reduce the huge differences between modalities, thereby improving the accuracy of pedestrian re-identification.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solution: a cross-modal pedestrian re-identification method based on cross-modal fusion and modal decorrelation, comprising the following steps:

[0006] S101: Establish a two-stream network for pedestrian feature extraction and initialize network parameters

[0007] S102: Obtain pedestrian images of two modalities, sample P identity categories, and sample K RGB images for each identity and K infrared images and denote the i-th RGB image and the i-th infrared image, respectively, which have the same identity label y i , each RGB image consists of three channels, namely R, G and B, expressed as Each infrared image contains only one grayscale channel; all and all Constructing a set of original images;

[0008] S103: Select some pedestrian images from the original image set for cross-modal fusion, and obtain a new RGB image for each selected pedestrian image. All new RGB images and pedestrian images in the original image set that are not selected for cross-modal fusion constitute a training sample set.

[0009] S104: Calculate the modality decorrelation quintuple loss L based on the embedded features MDQ ;

[0010] Based on the embedded features, calculate the identity loss L ID ;

[0011] S105: For each training sample, back-propagate and update the network parameters based on the final loss function value. When the loss no longer decreases, the training ends.

[0012] The final loss function is defined as:

[0013] L total =L ID +L MDQ #(1)

[0014] S106: For the image to be predicted, if the image to be predicted is an RGB image, the image to be predicted is input into the RGB branch, the distance between the feature of the image to be predicted and the features of each infrared image in the database is calculated, and the feature image whose distance between the predicted image and the feature of the image to be predicted ranks first M in descending order is output.

[0015] If the image to be predicted is an infrared image, the image to be predicted is input into the infrared branch, the distance between the features of the image to be predicted and the features of each RGB image in the database is calculated, and the feature images whose distance to the features of the image to be predicted ranks first M in descending order are output.

[0016] As an improvement, the S101 pedestrian feature extraction two-stream network uses ResNet-50 as the backbone network, the first two convolutional block parameters are independent, and the remaining convolutional block parameters are shared. The last ReLU activation function and the final classification layer of the ResNet-50 network are replaced by the BNNeck layer, and the step size of the last residual block of the original ResNet-50 is reduced from 2 to 1. The network is initialized using the parameters pre-trained on imagenet.

[0017] As an improvement, the step of obtaining a new RGB image for each pedestrian image selected in S103 is as follows:

[0018] Given an RGB image Randomly select a channel from Then with an infrared image Mixed, where C∈{R,G,B}. and have the same identity tag. The specific expression of this process is as follows:

[0019]

[0020]

[0021] in, is to generate a new RGB image, μ is a value between 0 and 1 randomly sampled from the Beta distribution, is the RGB image in channel C, or or represents the jth infrared image.

[0022] As an improvement, the S104 calculates the modality decorrelation quintuple loss L MDQ The steps are as follows:

[0023] In a batch, samples are collected from two modalities, denoted as M1 and M2. If M1 represents the RGB modality, then M2 represents the infrared modality, and vice versa. Given an image The corresponding hard positive samples and hard negative samples are selected from the same modality and another modality respectively. and Respectively Hard positive and negative samples in the same modality, and express Hard positive and negative samples in different modalities. The entire quintuple is represented as Let f(·) and ||·|| represent the feature mapping function and the Euclidean distance function respectively, then the distance d between positive samples in the same modality is pos,same It is expressed as:

[0024]

[0025] Distance d between positive sample pairs in different modalities pos,other It is expressed as:

[0026]

[0027] At the same time, define Distance d between negative samples in the same modality neg,same for:

[0028]

[0029] but Distance d between negative samples in different modalities neg,other for:

[0030]

[0031] A hard-sample quintuple loss is used to learn discriminative information in RGB and infrared modalities. HQ It is the combination of two triplet losses, namely the cross-modal triplet loss L cross and intramodal loss L intra , the specific expression of the five-tuple loss of difficult samples is as follows:

[0032] L MD =L cross +L intra #(8)

[0033] in,

[0034] L cross =max(m1+d pos,other -d neg,other ,0)#(9)

[0035] L intra =max(m1+d pos,same -d neg,same ,0)#(10)

[0036] m1 is a preset interval threshold.

[0037] Modal decorrelation loss L MD As shown in formula (11):

[0038] L MD =L neg +L pos #(11)Among them,

[0039] L neg =max(m2+d neg,other -d neg,same ,0)#(12)

[0040] L pos =max(m2+d pos,other -d pos,same ,0)#(13)

[0041] m2 is a preset interval threshold, and the complete modal decorrelation quintuple loss is as shown in formula (14):

[0042] K MDQ =L HQ +λL MD #(14)

[0043] Where λ is a hyperparameter that controls the loss weight.

[0044] As an improvement, the S104 calculates the identity loss L ID The steps are as follows:

[0045] The identity loss is used to map the features of RGB and infrared images into a shared identity space. ID It is expressed in the form of cross entropy loss between predicted identity and true label as follows:

[0046]

[0047] Among them, f is the output of the feature after the last fully connected layer, f j represents the category score of the jth identity, j∈[1,P total ], P total is the total number of identities in the training set.

[0048] Compared with the prior art, the present invention has at least the following advantages:

[0049] 1. Solving the modal difference problem at both the image and feature levels can better improve the performance of cross-modal retrieval.

[0050] 2. The present invention proposes cross-modal fusion, which does not introduce additional models or parameters while ensuring the quality of the generated image, and does not cause additional computational burden.

[0051] 3. The present invention proposes a modality decorrelation quintuple loss, which constrains the distance relationship within the positive sample pairs and the negative sample pairs, thereby reducing the correlation between the features and the modalities to which they belong, and eliminating redundant modality-specific information in the features. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is the main structure of the CMMD method of the present invention.

[0053] Figure 2 is the influence of parameters m2 and λ on the results. DETAILED DESCRIPTION

[0054] The present invention is described in further detail below.

[0055] The present invention proposes a cross-modal person re-identification method based on cross-modal fusion and modal decorrelation [CMMD for short], which alleviates the modal difference problem from both the image and feature levels. The method uses two parts to fully explore the association between the sample images and features of the two modalities: cross-modality channel mixup (CCM) and modality decorrelation quintuplet loss (MDQL). First, in order to reduce the difference between RGB and infrared images, the present invention designs cross-modal channel fusion as data enhancement. The method generates a new image by randomly mixing a channel of an RGB image and the corresponding infrared image as a new RGB image input. Therefore, cross-modal channel fusion is equivalent to supplementing the data set with new samples that have both RGB and infrared image information, making the input space of the model more continuous. CCM does not introduce additional parameters or models and can be easily combined with other methods. The method simultaneously considers the image information of the two modalities and prompts the model to learn the association between the modalities. The present invention designs a modality decorrelation quintuplet loss to reduce the modal difference at the feature level. The starting point of this loss is that if the feature is independent of the modality to which it belongs, then given an arbitrary sample, its hardest negative sample and hardest positive sample should appear in both modalities with equal probability. Specifically, this loss mines the hardest positive and hardest negative samples within and across modalities to construct a quintuple, and designs two constraints: 1) MDQL ensures that the distance between all positive sample pairs is smaller than the distance between all negative sample pairs. 2) MDQL forces the distance of the hardest negative sample within the modality to be greater than the distance of the hardest negative sample across modalities, and at the same time makes the distance of the hardest positive sample within the modality greater than the distance of the hardest positive sample across modalities, thereby ensuring that the probability of hard samples appearing in both modalities is equal. Through the constraints of MDQL, the huge intra-class differences are alleviated, and the redundant modality-specific information in image features is greatly reduced.

[0056] A cross-modal person re-identification method based on cross-modal fusion and modal decorrelation comprises the following steps:

[0057] S101: Establish a two-stream network for pedestrian feature extraction and initialize network parameters using the parameters pre-trained on ImageNet.

[0058] S102: Obtain pedestrian images of two modalities, sample P identity categories, and sample K RGB images for each identity and K infrared images and denote the i-th RGB image and the i-th infrared image, respectively, which have the same identity label y i, each RGB image consists of three channels, namely R, G and B, expressed as Each infrared image contains only one grayscale channel; all and all The goal of cross-modal person re-identification is to learn a mapping network F(·|θ) from images to features, so that the similarity of the same pedestrian features in the two modalities is as high as possible, where θ represents the parameters in the network.

[0059] S103: Select some pedestrian images from the original image set for cross-modal fusion, and each selected pedestrian image obtains a new RGB image. All new RGB images and pedestrian images in the original image set that are not selected for cross-modal fusion constitute a training sample set. In specific implementation, cross-modal fusion is performed on each pedestrian image with a probability of 0.5.

[0060] The shallow network in the two-stream network is used to extract pedestrian features. Each image in the training sample set is mapped to a shared feature space to obtain preliminary features. Then, the parameter-sharing convolutional block is used to further extract the deep semantic information of each image in the training sample set to obtain embedded features. For the specific process, please refer to formulas (4)-(7).

[0061] S104: Calculate the modality decorrelation quintuple loss L based on the embedded features MDQ ; Modality decorrelation quintuple loss (MDQL) is proposed to ensure two points: 1) As with most metric learning methods, the distance between positive pairs should be smaller than the distance between negative pairs to ensure discrimination between categories. 2) For an arbitrary sample, its hardest positive sample and hardest negative sample appear in both modalities with equal probability. Compared with previous methods, this loss not only constrains the relationship between positive and negative pairs, but also constrains the relationship within positive and negative pairs. Using this loss can make the model learn less modality-specific knowledge, which is beneficial to subsequent retrieval tasks.

[0062] Based on the embedded features, calculate the identity loss L ID ;

[0063] S105: For each training sample, back-propagate and update the network parameters based on the final loss function value. When the loss no longer decreases, the training ends.

[0064] The final loss function is defined as:

[0065] L total =L ID +L MDQ #(1)

[0066] S106: For the image to be predicted, if the image to be predicted is an RGB image, the image to be predicted is input into the RGB branch, the distance between the feature of the image to be predicted and the features of each infrared image in the database is calculated, and the feature image whose distance between the predicted image and the feature of the image to be predicted ranks first M in descending order is output.

[0067] If the image to be predicted is an infrared image, the image to be predicted is input into the infrared branch, the distance between the features of the image to be predicted and the features of each RGB image in the database is calculated, and the feature images whose distance to the features of the image to be predicted ranks first M in descending order are output.

[0068] Specifically, the S101 pedestrian feature extraction dual-stream network uses ResNet-50 as the backbone network, the first two convolutional block parameters are independent, and the remaining convolutional block parameters are shared. The last ReLU activation function and the final classification layer of the ResNet-50 network are replaced by the BNNeck layer, and the step size of the last residual block of the original ResNet-50 is reduced from 2 to 1. The network is initialized using the parameters pre-trained on imagenet.

[0069] Specifically, the step of obtaining a new RGB image for each pedestrian image selected in S103 is as follows:

[0070] Given an RGB image Randomly select a channel from Then with an infrared image Mixed, where C∈{R,G,B}. and have the same identity tag. The specific expression of this process is as follows:

[0071]

[0072]

[0073] in, is to generate a new RGB image, μ is a value between 0 and 1 randomly sampled from the Beta distribution, is the RGB image in channel C, or or represents the jth infrared image.

[0074] Cross-modal channel Mixup has the following advantages: 1) This method effectively expands the training set, making the model more robust to different inputs. 2) This method explicitly reduces the modality difference at the image level, so that the distribution of features of the same identity in the latent space is more continuous. 3) This method does not introduce additional models or use adversarial learning strategies, and has very low computational complexity.

[0075] In addition, in order to ensure that each RGB image can find a matching infrared image in each training batch, the present invention adopts a special sampling strategy. First, pedestrians with P identities are randomly selected for each batch, and for each identity, K RGB images and K infrared images are randomly sampled. Then, the sampled P×K=N RGB images and N infrared images are input into the RGB and infrared branches respectively. In this way, each identity in the batch has the same number of RGB and infrared images.

[0076] Specifically, S104 calculates the modality decorrelation quintuple loss L MDQ The steps are as follows:

[0077] In a batch, samples are collected from two modalities, denoted as M1 and M2. If M1 represents the RGB modality, then M2 represents the infrared modality, and vice versa. Given an image The corresponding hard positive samples and hard negative samples are selected from the same modality and another modality respectively. and Respectively Hard positive and negative samples in the same modality, and express Hard positive and negative samples in different modalities. The entire quintuple is represented as Let f(·) and ||·|| represent the feature mapping function and the Euclidean distance function respectively, then the distance d between positive samples in the same modality is pos,same It is expressed as:

[0078]

[0079] Distance d between positive sample pairs in different modalities pos,other It is expressed as:

[0080]

[0081] At the same time, define Distance d between negative samples in the same modality neg,same for:

[0082]

[0083] but Distance d between negative samples in different modalities neg,other for:

[0084]

[0085] MDQL can be expressed by the above four items. This loss consists of two parts: the five-tuple loss of difficult samples L HQ and the modal decorrelation loss L MD .

[0086] A hard-sample quintuple loss is used to learn discriminative information in RGB and infrared modalities. HQ It is the combination of two triplet losses, namely the cross-modal triplet loss L cross and intra-modal loss L intra , the specific expression of the five-tuple loss of difficult samples is as follows:

[0087] L MD =L cross +L intra #(8)

[0088] in,

[0089] L cross =max(m1+d pos,other -d neg,other ,0)#(9)

[0090] L intra =max(m1+d pos,same -d neg,same ,0)#(10)

[0091] m1 is a preset interval threshold. cross and L intra , which enhances the robustness of the model to intra-modal and cross-modal features. Compared with the original triplet loss, the hard sample quintuple loss considers both RGB and infrared modalities and is more suitable for use in cross-modal tasks.

[0092] However, it is still difficult to eliminate the strong correlation between features and their modalities using only the hard sample quintuple loss. Features extracted from the same modality are likely to have more similarities than features from different modalities, which affects the retrieval performance. Therefore, a modality decorrelation loss is designed to eliminate the modality-specific information in the features. MD As shown in formula (11):

[0093] L MD =L neg +L pos #(11)

[0094] in,

[0095] L neg =max(m2+d neg,other -d neg,same ,0)#(12)

[0096] L pos =max(m2+d pos,other -d pos,same ,0)#(13)

[0097] m2 is the preset interval threshold. neg The purpose is to constrain the hard negative samples to be more likely to appear in another modality, L pos The purpose is to constrain the hard positive samples to appear in the same mode as the anchor point. In this way, as the training progresses, the probability of hard samples appearing in the two modes gradually converges to the same, that is, decorrelation is achieved between features and modes. The complete modality decorrelation quintuple loss is as shown in formula (14):

[0098] L MDQ =L HQ +λL MD #(14)

[0099] Where λ is a hyperparameter that controls the loss weight.

[0100] Specifically, S104 calculates the identity loss L ID The steps are as follows:

[0101] The identity loss is used to map the features of RGB and infrared images into a shared identity space. ID It is expressed in the form of cross entropy loss between predicted identity and true label as follows:

[0102]

[0103] Among them, f is the output of the feature after the last fully connected layer, f j represents the category score of the jth identity, j∈[1,P total ], P total is the total number of identities in the training set. By optimizing the distance between each image feature and the category center, the identity loss can effectively reduce the intra-class differences.

[0104] Experimental analysis

[0105] The data sets selected by the present invention are RegDB and SYSU-MM01. RegDB uses a visible light camera and a thermal camera to capture two images at the same time. The data set contains 412 pedestrians of different identities. For each identity, 10 RGB images and 10 infrared images are collected, totaling 4120 RGB images and 4120 infrared images. During training, pedestrian images with 206 identity labels are randomly sampled, and pedestrian images with the remaining 206 identity labels are used during testing. During testing, the performance of using RGB images to retrieve infrared images and using infrared images to retrieve RGB images is evaluated separately. This random division is performed 10 times in total, and the average value is taken as the final performance. The shooting process of SYSU-MM01 involves 4 RGB cameras and 2 infrared cameras. The RGB camera is responsible for collecting images in bright scenes, and the infrared camera collects images in dim scenes. The data set collects 30,071 RGB images and 15,792 infrared images of 491 pedestrians of different identities. The training set contains 19,659 RGB images and 12,792 infrared images of 395 pedestrians, of which 296 identities are used for training and 99 identities are used for verification. The test set contains 96 images of different identities, from which 301 RGB images and 3,803 infrared images are randomly sampled as the images to be retrieved.

[0106] The evaluation indicators are the cumulative matching curve CMC, the average precision mAP and the average inverse negative sample penalty mINP.

[0107] The cumulative matching curve is based on the formula Calculate, {q1,…,q N} is the query set image, {g1,…,g M} are M pedestrian images to be matched in the gallery set.

[0108] The average Kyoto mean is based on the formula Calculate, where Indicates that there are n in the first m results p Pedestrians of the same identity.

[0109] The average inverse negative sample penalty is based on the formula in represents the ranking of the most difficult positive sample in the retrieval list, |G i | represents the query image q i The number of all correct matches.

[0110] In order to achieve the best performance of the present invention, firstly, two parameters m2 and λ are optimized and selected. m2 determines the strictness of the loss function constraint, and the weight parameter λ is mainly used to balance the weights between the proposed modal decorrelation quintuple loss items. By experimenting with all parameter combinations, the optimal parameter values ​​are found to be m2=0.2 and λ=0.1.

[0111] In order to further verify the effectiveness of the present invention, it is compared with the popular mainstream recommendation algorithms, which are as follows: Zero-Pad, HCML, cmGAN, HSME, AliGAN, XIV, DG-VAE, HAT, MACE, MSR, DDAG, Hi-CMD, cm-SSFT, AGW, G2DA, NFS, MCLNet and SCML.

[0112] Table 1 compares with some state-of-the-art methods on the SYSU-MM01 dataset.

[0113]

[0114] Table 2 compares with some state-of-the-art methods on the RegDB dataset.

[0115]

[0116] For the full search mode, this method achieved 71.3% Rank-1 accuracy, 66.2% mAP and 50.64% mINP, far exceeding the benchmark AGW method, and the accuracy increased by 5.9%, 4.2% and 3.3% respectively compared with MCLNet. For the indoor search mode, this method also achieved 75.2% Rank-1 accuracy, 78.3% mAP and 73.9% mINP, which were 2.6%, 2.0% and 1.8% higher than MCLNet. As a method to reduce modal differences, this method outperforms methods using adversarial generative networks or auxiliary modalities, including cmGAN, AliGAN, XIV, Hi-CMD, etc. In addition, CMMD also has higher performance than feature-level methods, including Zero-Pad, HCML, HSME, MACE, MSR, DDAG, cm-SSFT, AGW and SMCL. The experimental results in both modes prove the effectiveness of this method.

[0117] In the RGB retrieval infrared image mode, CMMD achieved 87.8% Rank-1 accuracy, 78.4% mAP, and 63.7% mINP, respectively, which were 5.3%, 6.3%, and 6.3% higher than MCLNet, respectively. In the infrared retrieval RGB image mode, this method achieved 85.7% Rank-1 accuracy, 76.3% mAP, and 58.0% mINP, respectively, which were 9.8%, 6.8%, and 5.4% higher than MCLNet, respectively. The accuracy of the method in the two modes is not much different, indicating that CMMD enables the model to extract modal shared information.

[0118] In summary, our method achieves competitive performance on all metrics and also proves to be robust to different settings of cross-modal retrieval (full search, indoor search, infrared to RGB retrieval, RGB to infrared retrieval).

[0119] In order to verify the effectiveness of each module, an ablation experiment was performed to eliminate a part of CMMD, and then the accuracy was compared. First, cross-modal channel fusion was used as data enhancement in the baseline method, and the accuracy increased to 64.0% Rank-1, 61.4% mAP, and 47.4% mINP. This shows that it is suitable for cross-modal pedestrian re-identification tasks. By generating new data at the image level, the robustness of the model is significantly increased.

[0120] Next, the effectiveness of each part of the modal decorrelation quintuple loss was verified through experiments. The third row of Table 3 shows the results of adding the difficult sample quintuple loss to the baseline method, with the accuracy improved to 64.4% Rank-1, 61.3% mAP, and 47.0% mINP, which are 5.6%, 3.6%, and 3.7% higher than the baseline method, respectively. On this basis, the modal decorrelation loss was added again, and the accuracy was significantly improved to 67.2% Rank-1, 64.4% mAP, and 49.8% mINP. When L neg and L pos When used separately, the results were improved to varying degrees. pos The performance is slightly better than using L alone neg . This series of results shows that the proposed modality decorrelation quintuple loss can effectively extract modality shared information and promote the learning of discriminative features. Finally, when using CMMD, the method achieved 71.3% Rank-1, 66.2% mAP and 50.6% mINP, significantly outperforming the baseline method, with accuracy improved by 11.6%, 6.6% and 7.3% respectively. In summary, the results fully demonstrate the effectiveness of the CMMD method.

[0121] In short, the method of the present invention alleviates the common modality difference problem in cross-modal person re-identification and reduces the correlation between features and modalities. The performance of the method of the present invention is better than the current popular mainstream cross-modal pedestrian re-identification algorithm.

[0122] Table 3 Ablation experiments to verify the effectiveness of different parts of the proposed CMMD.

[0123]

[0124]

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A cross-modal person re-identification method based on cross-modal fusion and modal decorrelation, characterized by: The steps include: S101: Establish a two-stream network for pedestrian feature extraction and initialize network parameters S102: Obtain pedestrian images of two modalities, sample P identity categories, and sample K RGB images for each identity and K infrared images and denote the i-th RGB image and the i-th infrared image, respectively, which have the same identity label y i , each RGB image consists of three channels, namely R, G and B, expressed as Each infrared image contains only one grayscale channel; all and all Constructing a set of original images; S103: Selecting some pedestrian images from the original image set for cross-modal fusion, each selected pedestrian image obtains a new RGB image, and all new RGB images and pedestrian images in the original image set that are not selected for cross-modal fusion constitute a training sample set; S104: Calculate the modality decorrelation quintuple loss L based on the embedded features MDQ ; Based on the embedded features, calculate the identity loss L ID ; S105: For each training sample, back propagate and update the network parameters based on the final loss function value. When the loss no longer decreases, the training ends. The final loss function is defined as: L total =L ID +L MDQ #(1) S106: For the image to be predicted, if the image to be predicted is an RGB image, the image to be predicted is input into the RGB branch, the distance between the feature of the image to be predicted and the features of each infrared image in the database is calculated, and the feature image whose distance between the predicted image and the feature of the image to be predicted ranks first M in descending order is output; If the image to be predicted is an infrared image, the image to be predicted is input into the infrared branch, the distance between the features of the image to be predicted and the features of each RGB image in the database is calculated, and the feature images whose distance to the features of the image to be predicted ranks first M in descending order are output.

2. The cross-modal person re-identification method based on cross-modal fusion and modal decorrelation according to claim 1, characterized in that: The S101 pedestrian feature extraction dual-stream network uses ResNet-50 as the backbone network. The parameters of the first two convolution blocks are independent, and the parameters of the remaining convolution blocks are shared. The last ReLU activation function of the ResNet-50 network and the final classification layer are replaced by the BNNeck layer, and the step size of the last residual block of the original ResNet-50 is reduced from 2 to 1.

3. The cross-modal person re-identification method based on cross-modal fusion and modal decorrelation according to claim 2, characterized in that: The steps of obtaining a new RGB image for each pedestrian image selected in S103 are as follows: Given an RGB image Randomly select a channel from Then with an infrared image Mixed, where C∈{R,G,B}, and With the same identity tag, the process is specifically expressed as formulas (2) and (3): in, is to generate a new RGB image, μ is a value between 0 and 1 randomly sampled from the Beta distribution, is the RGB image in channel C, or or represents the jth infrared image.

4. The cross-modal person re-identification method based on cross-modal fusion and modal decorrelation as claimed in claim 3, characterized in that: The S104 calculates the modality decorrelation quintuple loss L MDQ The steps are as follows: In a batch, samples are collected from two modalities, denoted as M1 and M2. If M1 represents the RGB modality, then M2 represents the infrared modality, and vice versa. Given an image The corresponding hard positive samples and hard negative samples are selected from the same modality and another modality respectively, using and Respectively Hard positive and negative samples in the same modality, and express The hard positive and negative samples in different modalities, the entire quintuple is represented as Let f(·) and ||·|| represent the feature mapping function and the Euclidean distance function respectively, then the distance d between positive samples in the same modality is pos,same It is expressed as: Distance d between positive sample pairs in different modalities pos,other It is expressed as: At the same time, define Distance d between negative samples in the same modality neg,same for: but Distance d between negative samples in different modalities neg,other for: Using hard sample quintuple loss to learn discriminative information in RGB and infrared modalities, L HQ It is the combination of two triplet losses, namely the cross-modal triplet loss L cross and intra-modal loss L intra , the specific expression of the five-tuple loss of difficult samples is as follows: L MD =L cross +L intra #(8) in, L cross =max(m1+d pos，other -d neg，other ,0)#(9) L intra =max(m1+d pos，same -d neg，same ,0)#(10) m1 is the preset interval threshold; Modal decorrelation loss L MD As shown in formula (11): L MD =L neg +L pos #(11) in, L neg =max(m2+d neg，other -d neg，same ,0)#(12) L pos =max(m2+d pos，other -d pos，same ,0)#(13) m2 is a preset interval threshold, and the complete modal decorrelation quintuple loss is as shown in formula (14): THE MDQ =L HQ +λL MD #(14) Where λ is a hyperparameter that controls the loss weight.

5. The cross-modal person re-identification method based on cross-modal fusion and modal decorrelation according to claim 4, characterized in that: The S104 calculates the identity loss L ID The steps are as follows: The identity loss is used to map the features of RGB and infrared images into a shared identity space. ID It is expressed in the form of cross entropy loss between predicted identity and true label as follows: Among them, f is the output of the feature after the last fully connected layer, f j represents the category score of the jth identity, j∈[1,P total ], P total is the total number of identities in the training set.

Images