A Multimodal Person Re-identification Method Based on Complementary Data Augmentation

Abstract

This invention relates to a multimodal person re-identification method based on complementary data augmentation, comprising: performing dual-color space data augmentation on initial modal image data to obtain an initial modal augmented image; extracting initial modal image features from the initial modal augmented image using a trained feature extraction module; when the initial modal image data includes one or two of RGB, TI, and NI images, generating predictive features for missing modalities based on the initial modal image features using different modal reconstructors in a trained modal reconstruction module to obtain multimodal image features; when the initial modal image data includes three of RGB, TI, and NI images, constructing multimodal image features from the initial modal image features; concatenating the category labels of the same identity and the same sample in the multimodal image features to obtain positive samples, and obtaining the person re-identification result. This method improves the response speed and retrieval accuracy of person re-identification.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a multimodal pedestrian re-identification method based on complementary data augmentation. Background Art

[0002] Person Re-identification (Person ReID) is a technology used to retrieve and identify the same person under different cameras, and it has wide applications in security, surveillance and intelligent transportation.

[0003] Unlike traditional pedestrian re-identification tasks based on single visible light images, multimodal pedestrian re-identification aims to achieve robust re-identification by introducing multiple complementary modalities of imagery for each person. This approach helps handle more complex lighting scenarios, greatly assisting traditional pedestrian re-identification tasks and overcoming their application limitations. Furthermore, the widespread availability of various cameras (such as various infrared and RGB cameras) has made multimodal pedestrian re-identification possible and has garnered increasing attention in recent years. Therefore, due to its strong complementary advantages, multimodal pedestrian re-identification has enormous potential application value in the field of intelligent surveillance systems.

[0004] In multimodal pedestrian re-identification tasks, visible light, thermal infrared (TI), and near infrared (NI) images are widely used due to their modal complementarity and ability to meet the needs of all-weather, all-scene pedestrian recognition. Visible light modal images are implemented using RGB images. Under good lighting conditions, RGB images provide rich color and texture information; NI images are unaffected by lighting conditions and provide clear edge information; while TI images can distinguish people from their surroundings using temperature and are unaffected by complex environments. Therefore, fully utilizing the complementary information of different modalities is key to multimodal pedestrian re-identification. Multimodal pedestrian re-identification differs significantly from traditional cross-modal pedestrian re-identification. Traditional cross-modal pedestrian re-identification focuses on reducing modal differences and learning modal-shared features. In contrast, multimodal pedestrian re-identification emphasizes effective modal fusion to absorb complementary information from multimodal data, thereby improving the discriminability of pedestrians.

[0005] Current multimodal person re-identification methods employ various interactive modules to process multimodal data, aiming to fully leverage complementary information between modalities. However, these methods are not only limited by the underlying architecture but also introduce additional computational overhead during the inference phase due to modal interactions. Furthermore, existing contrastive learning-based multimodal alignment methods can achieve global feature alignment between modalities, but their mandatory alignment strategy requires strict consistency in the embedding space for single-modal representations with the same identity (ID). This strict constraint can excessively suppress modality-specific features (such as texture details) during processing. In fine-grained classification tasks like person re-identification, where subtle differences need to be distinguished, this phenomenon significantly reduces the model's discriminative ability. Moreover, labeling multimodal data is more challenging than labeling single-modal data, as the number of samples is typically limited. Due to equipment failure, environmental limitations, or sensor malfunctions, sample quality may be relatively low, and some modalities may even be missing. These issues pose challenges to the practical application of person re-identification technology. Summary of the Invention

[0006] To address the aforementioned problems in the existing technology, this invention provides a multimodal person re-identification method based on complementary data augmentation. The technical problem to be solved by this invention is achieved through the following technical solution:

[0007] This invention provides a multimodal person re-identification method based on complementary data augmentation, comprising the following steps:

[0008] Dual color space data enhancement is performed on the initial modal image data to obtain an initial modal enhanced image, wherein the initial modal image data includes at least one of RGB image, TI image, and NI image;

[0009] The trained feature extraction module is used to extract global semantic information and local detail information from the initial modality enhancement image to obtain the initial modality image features;

[0010] When the initial modal image data includes one or two of RGB images, TI images, and NI images, based on the features of the initial modal images, the predicted features of the missing modalities are generated using different modal reconstructors in the trained modal reconstruction module, thus obtaining multimodal image features.

[0011] When the initial modal image data includes three types of images: RGB, TI, and NI, the initial modal image features constitute multimodal image features.

[0012] By concatenating the category labels of the same identity and the same sample in the multimodal image features, positive samples are obtained, and pedestrian re-identification results are obtained.

[0013] In one embodiment of the present invention, the training method for the feature extraction module and the modality reconstruction module includes the following steps:

[0014] Multimodal training data is augmented using dual color space data to obtain multimodal augmented training data, wherein the multimodal training data includes RGB images, TI images, and NI images;

[0015] The feature extraction module is used to extract global semantic information and local detail information from the multimodal augmented training data to obtain multimodal image training features;

[0016] The modality reconstruction module is trained using the multimodal image training features, and constrained by the total loss of salient feature reconstruction. The total loss of salient feature reconstruction includes the total loss of generating NI and TI modalities from RGB modality, the total loss of generating RGB and TI modalities from NI modality, and the total loss of generating RGB and NI modalities from TI modality.

[0017] From the multimodal image training features, class labels of the same identity and the same sample are selected and concatenated to obtain training positive samples, and class labels of samples of different identities or samples of the same identity but different samples are randomly selected and concatenated to obtain training negative samples.

[0018] The training positive samples and the training negative samples are input into a binary classifier for discrimination.

[0019] Combining the predicted labels output by the binary classifier, the modality-aware soft alignment total loss is used to constrain the binary classifier and the feature extraction module. The modality-aware soft alignment total loss includes the first cross-entropy loss and the cross-modal hard triplet loss.

[0020] The training positive samples are input into a linear classification layer to classify different identities;

[0021] The linear classification layer and the feature extraction module are constrained using the second cross-entropy loss and the triplet loss;

[0022] Repeat the iteration until the total loss converges to obtain the trained feature extraction module and the trained modality reconstruction module, wherein the total loss includes the total loss of the salient feature reconstruction, the first cross-entropy loss, the cross-modal hard triplet loss, the second cross-entropy loss and the triplet loss.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0024] This invention provides a multimodal person re-identification method based on complementary data augmentation. First, it performs dual-color space data augmentation on the initial modal image and extracts features from it, achieving data expansion and improved data quality. Then, in the case of missing modalities, it utilizes different modal reconstructors in a pre-trained modal reconstruction module to generate predicted features for the missing modalities, effectively solving the problem of incomplete modalities. Finally, it uses multimodal image features to construct positive samples to obtain the person re-identification result. This method promotes collaborative perception of multimodal features by multiple networks, enhances shared features, and solves the problems of additional computational overhead introduced by modal interaction during the inference stage, as well as low sample quality and missing specific modalities in practical applications. Therefore, this method requires no interaction module and, in the case of missing modalities, does not propagate noise in the reconstructed features to other modalities, thereby improving response speed and retrieval accuracy in practical applications and making it applicable to more general person re-identification scenarios. Attached Figure Description

[0025] Figure 1 A flowchart illustrating a multimodal pedestrian re-identification method based on complementary data augmentation provided in an embodiment of the present invention;

[0026] Figure 2 A schematic diagram illustrating dual color space data enhancement provided in an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram illustrating the extraction of different modal features using the ViT-B / 16 network under different modalities, provided in an embodiment of the present invention.

[0028] Figure 4 This is a schematic diagram of the structure of a modal reconstructor provided in an embodiment of the present invention;

[0029] Figure 5 A schematic diagram of a training method for a feature extraction module and a modality reconstruction module provided in an embodiment of the present invention;

[0030] Figure 6 This is a schematic diagram illustrating the construction of positive and negative samples according to an embodiment of the present invention. Detailed Implementation

[0031] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0032] Example 1

[0033] Please see Figure 1 , Figure 1 This is a flowchart illustrating a multimodal pedestrian re-identification method based on complementary data augmentation, provided in an embodiment of the present invention.

[0034] This embodiment of the multimodal person re-identification method based on complementary data augmentation includes the following steps:

[0035] S11. Perform dual-color space data enhancement on the initial modal image data to obtain the initial modal enhanced image.

[0036] Please see Figure 2 , Figure 2 This is a schematic diagram of dual-color space data enhancement provided in an embodiment of the present invention. Specifically, the dual color space includes RGB space and HSV space, and the dual-color space data enhancement includes two parts: initial modal image block rotation and RGB image enhancement. The initial modal image data includes at least one of RGB image, TI image, and NI image.

[0037] Step S11 specifically includes:

[0038] S111. Rotate the initial modal image data into image blocks to obtain a rotated image of the initial modal image. Specific steps include:

[0039] S1111. For images of the same identity and the same sample in the initial modality image data, initialize the probabilities. In this context, "same identity" refers to having the same ID, such as the same pedestrian; "same sample" refers to the same time and the same space. Taking the initial modal image data, which includes RGB images, TI images, and NI images, as an example, images with the same identity and the same sample refer to images with the same ID and the same time and space among RGB images, TI images, and NI images.

[0040] S1112, when probability When the probability is less than or equal to a preset value, images of the same identity and the same sample are retained. Specifically, the preset value can be 0.5, meaning that when the probability is less than or equal to a preset value, images of the same identity and the same sample are retained. At the same time, images of the same identity and the same sample are retained.

[0041] When probability When the value exceeds a preset value, the position and size of the image blocks are randomly initialized. Based on the position and size of the image blocks, image blocks are cut from the same position in each modal image. Then, the image blocks of different modalities are iteratively replaced to obtain a rotated image of each initial modal image. Specifically, the following steps are included:

[0042] 1) When probability At that time, the position of the image patch is randomly initialized. and size .

[0043] Specifically, when the probability At that time, the row coordinates of the starting point of the image patch are randomly generated. Column coordinates and determine the size of the image patch. The image patch is a rectangular image patch, located at... The row coordinates of the top-left vertex of the rectangular image patch Column coordinates ,size This refers to a rectangular image block whose length and width are both [missing information]. Row coordinates Column coordinates Must meet and , Image height, Image width. Image tile size. Values Random values ​​within, For fixed values, Determine the maximum size of the image patch. This refers to the image size.

[0044] 2) Cut out image blocks from the same position in each modal image according to the position and size of the image blocks.

[0045] Specifically, taking the initial modal image data, which includes RGB, TI, and NI images, as an example, based on the position of the image blocks... and size For images of the same identity and the same sample in RGB, TI, and NI images, cut out images of size [missing information] at the same location. The image blocks are used to obtain RGB image blocks, TI image blocks, and NI image blocks.

[0046] 3) The image blocks of different modalities are replaced cyclically to obtain the rotated image of each initial modal image.

[0047] Specifically, when the initial modal image data includes RGB images, TI images, and NI images, the RGB image blocks, TI image blocks, and NI image blocks are cyclically replaced; for example, RGB image blocks are replaced with TI image blocks, TI image blocks are replaced with NI image blocks, and NI image blocks are replaced with RGB image blocks; or, RGB image blocks are replaced with NI image blocks, NI image blocks are replaced with TI image blocks, and TI image blocks are replaced with RGB image blocks, resulting in rotated images of RGB images, rotated images of TI images, and rotated images of NI images.

[0048] When the initial modal image data includes two of the following: RGB, TI, and NI images, the image blocks of the two modalities are replaced to obtain rotated images of the two modalities. For example, replacing RGB image blocks with TI image blocks and vice versa yields rotated images of both the RGB and TI images.

[0049] When the initial modal image data includes any of the RGB, TI, and NI images, no replacement is performed; the rotated image of the initial modal image is directly output. For example, if the initial modal image data includes an RGB image, the RGB image is directly output as a rotated image of the RGB image.

[0050] S112. When the initial modal image data includes an RGB image, the saturation and brightness of the rotated RGB image are enhanced to obtain an enhanced RGB image. When the initial modal image data includes TI images, the rotated image of the TI image is used as the enhanced TI image. When the initial modal image data includes NI images, the rotated image of the NI image is used as the enhanced NI image. .

[0051] It is understandable that when the initial modal image data includes an RGB image, the rotated image of the RGB image is enhanced; when the initial modal image data does not include an RGB image, the rotated image is directly output as the enhanced image.

[0052] Specifically, the process of enhancing the saturation and brightness of a rotated RGB image to obtain an enhanced RGB image includes the following steps:

[0053] 1) Convert the rotated RGB image to HSV space to obtain the converted image.

[0054] 2) Calculate the saturation and brightness of the converted image. The formulas for calculating saturation and brightness are as follows:

[0055] ;

[0056] ;

[0057] in, For saturation, For brightness, The maximum pixel value of the R, G, and B channels. This represents the minimum pixel value for the R, G, and B channels.

[0058] 3) Perform saturation and brightness enhancement on the converted image to obtain the SV-enhanced image. The formulas for saturation and brightness enhancement are as follows:

[0059] ;

[0060] in, To enhance the saturation, This is the enhanced brightness.

[0061] 4) Convert the SV enhanced image to RGB space to obtain the enhanced RGB image. .

[0062] For example, the initial modal image data input in step S1 is a size-normalized image of 256×128×3, and the size of the output initial modal enhancement image remains unchanged.

[0063] S12. Use the trained feature extraction module to extract global semantic information and local detail information from the initial modality enhancement image to obtain the initial modality image features.

[0064] Specifically, under different modalities, the trained ViT-B / 16 network is used as the backbone network to extract global semantic information and local detail information from the initial modal enhancement image to obtain the initial modal image features. The initial modal image features include category labels representing the global semantic information of the image.

[0065] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating the extraction of different modal features using the ViT-B / 16 network under different modalities, as provided in an embodiment of the present invention. Taking initial modal image data including RGB images, TI images, and NI images as an example, step S12 includes:

[0066] Using the pre-trained first ViT-B / 16 network, enhance RGB images Global semantic information and local detail information are extracted to obtain RGB modal image features:

[0067] ;

[0068] in, For RGB modal image features, To enhance RGB images, This is the first ViT-B / 16 network in RGB mode.

[0069] Using the trained second ViT-B / 16 network, from enhanced TI images Global semantic information and local detail information are extracted to obtain TI modal image features:

[0070] ;

[0071] in, For TI modal image features, To enhance TI images, This is the second ViT-B / 16 network in TI mode.

[0072] Using a pre-trained third ViT-B / 16 network, enhance NI images Global semantic information and local detail information are extracted to obtain NI modal image features:

[0073] ;

[0074] in, For NI modal image features, To enhance NI images, This is the third ViT-B / 16 network in the NI mode.

[0075] For example, RGB modal image features TI modal image features NI modal image features The dimensions of each feature are 768×129, and each modal image feature includes a category label representing the global semantic information of the image. The dimension of the category label is 768×1.

[0076] S13. When the initial modal image data includes one or two of RGB images, TI images, and NI images, based on the initial modal image features, the predicted features of the missing modal are generated using different modal reconstructors in the trained modal reconstruction module to obtain multimodal image features; when the initial modal image data includes three of RGB images, TI images, and NI images, the multimodal image features are constructed from the initial modal image features.

[0077] Specifically, the modality reconstruction module includes a first modality reconstructor that generates TI modality image features from RGB modality image features. A second modal reconstructor that generates NI modal image features from RGB modal image features. A third-mode reconstructor that generates RGB modal image features from TI modal image features. A fourth modal reconstructor that generates NI modal image features from TI modal image features. A fifth modal reconstructor that generates RGB modal image features from NI modal image features. And a sixth modal reconstructor that generates TI modal image features from NI modal image features. .

[0078] Please see Figure 4 , Figure 4This is a schematic diagram of a modal reconstructor provided in an embodiment of the present invention. The first modal reconstructor, second modal reconstructor, third modal reconstructor, fourth modal reconstructor, fifth modal reconstructor, and sixth modal reconstructor have the same structure, all including:

[0079] The first normalization layer is used to perform layer normalization processing on the initial modal image features of the input to standardize the input features and obtain the first normalized features.

[0080] The multi-head self-attention module is used to process the first-layer normalized features using a multi-head self-attention mechanism to learn how to reconstruct image features of another modality from image features of one modality, and obtain multi-head self-attention output features.

[0081] The first residual fusion module is used to add and fuse the input initial modality image features with the multi-head self-attention output features to achieve residual fusion and obtain fused features;

[0082] The second normalization layer is used to perform layer normalization on the fused features to ensure the stability of the subsequent multilayer perceptron input, resulting in the second normalized features.

[0083] A multilayer perceptron is used to generate preliminary reconstructed features based on the normalized features of the second layer.

[0084] The second residual fusion module is used to add and fuse the fused features with the preliminary reconstructed features to achieve residual fusion and obtain the predicted features of the missing modalities.

[0085] For example, the predicted features output by each modality reconstructor have the same dimension as the features of the input modality image, which is 768×129.

[0086] Specifically, when the initial modal image data includes any one of RGB, TI, or NI images, that is, when the initial modal image data is missing two modalities, the image features of that modality are input into the corresponding modal reconstructor to generate the predicted features of the other two missing modalities, thereby obtaining the image features of the three modalities of RGB, TI, and NI.

[0087] From RGB modal image features Generated TI modal image features and NI modal image features for:

[0088] ;

[0089] .

[0090] Features of TI modal images Generated RGB modal image features and NI modal image features for:

[0091] ;

[0092] .

[0093] Features from NI modal images Generated RGB modal image features and TI modal image features for:

[0094] ;

[0095] .

[0096] When the initial modal image data includes two of the following: RGB, TI, and NI images (meaning one modality is missing), the features of the two modal images are input into their respective modal reconstructors to generate features for the missing modality. The average of these features is then used as the predicted feature for the missing modality. For example, when the initial modal image data includes RGB and TI images, but the NI image is missing, the RGB modal image features are input into a second reconstructor to generate NI modal image features, and the TI modal image features are input into a fourth reconstructor to generate NI modal image features. The average of the two generated NI modal image features is then used to obtain the predicted feature for the NI modality.

[0097] In this embodiment, when one modality is missing, the features of the missing modality are constructed by using two modalities. The features of the two available modalities can provide more comprehensive information, make up for the limitations of a single modality, and realize the utilization of complementary information. The combined use of the features of the two modalities can reduce noise interference, improve reconstruction quality, and has good noise resistance. Through multimodal collaborative reconstruction, the model can maintain high accuracy when some modalities are missing, and has good robustness.

[0098] When the initial modal image data includes three types of images: RGB, TI, and NI, the multimodal image features are formed from the features of the initial modal images. That is, the multimodal image features are directly formed from the features of the RGB, TI, and NI modal images.

[0099] S14. Connect the category labels of the same identity and the same sample in the multimodal image features to obtain positive samples, and obtain the pedestrian re-identification results.

[0100] Specifically, positive samples are obtained by concatenating the category labels of the same identity and the same sample from the image features of the RGB, TI, and NI modalities, thus obtaining the pedestrian re-identification result. Here, the same sample refers to samples that have the same time and space.

[0101] For example, the category label dimension of the three modal image features is 768, and the dimension of the final positive sample representation is 768×3=2304.

[0102] The multimodal person re-identification method in this embodiment first performs dual-color space data augmentation on the initial modal image and extracts features from it, achieving data augmentation and improved data quality. Then, in the case of missing modalities, it uses different modal reconstructors in the trained modal reconstruction module to generate predicted features for the missing modalities, effectively solving the problem of incomplete modalities. Finally, it uses multimodal image features to construct positive samples to obtain the person re-identification results, promoting the collaborative perception of multimodal features by multiple networks, enhancing shared features, and solving the problems of additional computational overhead introduced by modal interaction in the inference stage and low sample quality and missing specific modalities in practical applications. Therefore, this method does not require any interaction module, and in the case of missing modalities, it does not propagate noise in the reconstructed features to other modalities, thereby improving the response speed and retrieval accuracy in practical applications and making it applicable to more general person re-identification scenarios.

[0103] The method in this embodiment significantly reduces inference time by independently extracting single-modal features and directly concatenating them, while maintaining high accuracy, achieving a better balance between efficiency and performance.

[0104] Please see Figure 5 , Figure 5 This is a schematic flowchart of a training method for a feature extraction module and a modality reconstruction module provided in an embodiment of the present invention. The training method includes the following steps:

[0105] S501. Perform dual-color space data augmentation on the multimodal training data to obtain multimodal augmented training data, which includes RGB images, TI images, and NI images.

[0106] S502. Use the feature extraction module to extract global semantic information and local detail information from the multimodal augmented training data to obtain multimodal image training features.

[0107] For the specific execution steps of steps S501 and S502, please refer to steps S11 and S12 above.

[0108] S503. The modality reconstruction module is trained using multimodal image training features, and the modality reconstruction module is constrained by combining the total loss of salient feature reconstruction. The total loss of salient feature reconstruction includes the total loss of generating NI and TI modal from RGB modality, the total loss of generating RGB and TI modal from NI modality, and the total loss of generating RGB and NI modal from TI modality.

[0109] Specifically, for each embedded image patch in the feature map generated by the modality reconstructor, pixel-level constraints are applied using the mean squared error loss function (MSE):

[0110] ;

[0111] ;

[0112] ;

[0113] ;

[0114] ;

[0115] ;

[0116] in, To generate NI mode from RGB mode The mean square error loss, The number of image patches embedded in the generated modal features. NI modal image features The An embedded image patch, NI modal feature map generated from RGB modalities The One embedded image block; To generate TI modes from RGB modes The mean square error loss, TI modal image features The An embedded image patch, TI modal feature map generated from RGB modalities The One embedded image block; To generate RGB modes from NI modes The mean square error loss, RGB modal image features The An embedded image patch, RGB modal feature maps generated from NI modalities The One embedded image block; To generate TI modes from NI modes The mean square error loss, TI modal feature map generated from NI modality The One embedded image block; To generate RGB modes from TI modes The mean squared error loss, RGB modal feature map generated by TI modality The Image blocks; To generate NI modes from TI modes The mean square error loss, NI modal feature map generated from TI modalities The One embedded image block; , , , To enhance RGB images, This is the first ViT-B / 16 network in RGB mode. To enhance TI images, This is the second ViT-B / 16 network in TI mode. To enhance NI images, This is the third ViT-B / 16 network in the NI mode; , , , , , , For the first mode reconfigurator, For the second modal reconfigurator, For the third modal reconfigurator, For the fourth mode reconfigurator, For the fifth mode reconfigurator, It is the sixth mode reconfigurator.

[0117] Furthermore, class labels are extracted from the feature maps generated by the modality reconstructor and the training features of the input multimodal image, and the focal frequency loss function (FFL) is constructed using the class labels to constrain the modality reconstructor.

[0118] ;

[0119] ;

[0120] ;

[0121] ;

[0122] ;

[0123] ;

[0124] in, For focus frequency loss, for , , , , or , Dimensions for labeling categories, The first category label A spectral weight matrix with multiple dimensions. , for The first category label in the middle Fast Fourier Transform in 1000 Dimensions Represents RGB, NI, or TI. for The first category label in the middle Fast Fourier Transform in 1000 Dimensions.

[0125] In summary, the total loss for salient feature reconstruction is:

[0126] ;

[0127] ;

[0128] ;

[0129] ;

[0130] in, The total loss is used to reconstruct salient features. The total loss for generating NI and TI modes from RGB modes. The total loss for generating RGB and TI modes from NI modes. This represents the total loss generated from the TI mode to the RGB and NI modes.

[0131] It is understandable that during the training phase, the modality reconstruction module learns to reconstruct features of other modalities from features of one modality using complete multimodal data; when a modality is missing during the testing phase, it can generate features of the missing modality based on the existing modalities.

[0132] S504. Select class labels of the same identity and the same sample from the multimodal image training features and connect them to obtain training positive samples. Randomly select class labels of samples with different identities or samples with the same identity but different identities and connect them to obtain training negative samples.

[0133] Specifically, the class labels of the same identity and the same sample in the training features of the three modalities are concatenated to obtain training positive samples, and a first pseudo-label 1 is assigned to the training positive samples:

[0134] ;

[0135] in, Indicates the first The labels of the training positive samples, The first feature in training RGB modality images Category labeling of each sample, The first feature in the training of NI modality images Category labeling of each sample, The first feature in the training of TI modality images The category label of each sample.

[0136] Training negative samples are obtained by randomly selecting an equal number of category labels belonging to different identities or different samples of the same identity from the training features of the three modalities, and then concatenating them. A second pseudo-label 0 is assigned to the training negative samples.

[0137] ;

[0138] in, Indicates the first The labels of each training negative sample. The first feature in training RGB modality images Category labeling of each sample, The first feature in the training of NI modality images Category labeling of each sample, The first feature in the training of TI modality images Category labeling of each sample, , There are at least two different values.

[0139] Here, "same sample" refers to samples with the same time and space; "different identity sample" refers to samples with different identities; "same identity different sample" refers to samples with the same identity but different times or different spaces.

[0140] Please see Figure 6 , Figure 6This diagram illustrates the construction of positive and negative samples according to an embodiment of the present invention. From left to right, the images represent RGB modal image training features, NI modal image training features, and TI modal image training features. In the diagram, positive training samples are obtained by concatenating the category labels of the same identity (ID#1) and the same sample (sample#1) from all three modal image training features. Negative training samples are obtained by concatenating the category labels of the same identity (ID#1) from all three modal image training features, sample#2 from the RGB modal image training features, and sample#1 from both the TI and NI modal image training features; or, negative training samples are obtained by concatenating the category labels of sample#1 from all three modal image training features, ID#2 from the RGB modal image training features, and ID#1 from both the TI and NI modal image training features.

[0141] For example, the category label dimension of the three modal image features is 768, and the final dimension of the training positive sample and training negative sample representation is 768×3=2304.

[0142] S505. Input the training positive samples and training negative samples into the binary classifier for discrimination.

[0143] Specifically, the binary classifier uses a binary linear layer, where training positive and training negative samples are distinguished and the predicted label of the sample is output.

[0144] S506. Combining the predicted labels output by the binary classifier, the modality-aware soft alignment total loss is used to constrain the binary classifier and the feature extraction module. The modality-aware soft alignment total loss includes the first cross-entropy loss and the cross-modal hard triplet loss.

[0145] Specifically, the first cross-entropy loss is used as the classification loss to guide sample classification:

[0146] ;

[0147] in, For the first cross-entropy loss, The number of samples in the batch. It is the first The true label of each sample It is the first output of the binary classifier Predicted labels for each sample.

[0148] Please see again Figure 6 , Figure 6The method of constructing positive and negative samples in a batch can lead to a problem: non-corresponding samples with the same identity in a batch may be labeled as negative samples. This can cause the distance between non-corresponding cross-modal samples (positive samples pos) of the same identity to be greater than or equal to the distance between cross-modal samples (negative samples neg) of different identities, guided by the classification loss. In other words, the distance between different samples with the same ID is greater than the distance between different IDs. Therefore, a cross-modal hard triplet loss is used to impose constraints to prevent the above problem in the batch, ensuring that the feature distance between samples with the same identity is as small as possible, while the feature distance between samples with different identities is as large as possible. The cross-modal hard triplet loss function includes:

[0149] ;

[0150] ;

[0151] ;

[0152] in, For RGB cross-modal hard triplet loss, For NI cross-modal hard triplet loss, For TI cross-modal hard triplet loss, For boundary parameters, Indicates negative samples Perform hard sample mining. Represents the distance metric function. For the first Features of each anchor point sample The most difficult positive sample Features For the most difficult negative sample Its characteristics.

[0153] In summary, the total loss of the modality-aware soft alignment module is... for:

[0154] ;

[0155] in, These are the weight parameters.

[0156] S507. Input the training positive samples into the linear classification layer to classify different identities.

[0157] S508. Constrain the linear classification layer and feature extraction module using the second cross-entropy loss and triplet loss.

[0158] Specifically, the second cross-entropy loss and triplet loss are used to constrain the training positive samples, which solves the feature distribution differences between different samples, making the feature distances of different samples with the same identity closer and the feature distances of different identities farther.

[0159] Specifically, the second cross-entropy loss is:

[0160] ;

[0161] in, For the second cross-entropy loss, The number of samples in the batch. For the number of identity IDs, In the constructed multimodal training positive samples, the first One sample, For the first Does each sample belong to a certain identity? The true label, It is the first Each sample belongs to a certain identity. Predicted labels.

[0162] The loss of the triplet is:

[0163] ;

[0164] in, For triple loss, It is the first positive sample in the constructed multimodal training. An identity, It is the first Hard sample mining is performed on negative samples from each identity. It is the first positive sample in the constructed multimodal training. Anchor point samples of each identity, It is the first positive sample in the constructed multimodal training. Positive samples of each identity. It is the first positive sample in the constructed multimodal training. Negative samples of each identity, This represents the anchor point sample of the triple. This represents a positive sample of a triple. This represents a negative sample in a triplet. The square of the Euclidean distance. This is the interval parameter.

[0165] S509. Repeat the iteration until the total loss converges to obtain the trained feature extraction module and the trained modality reconstruction module. The total loss includes the total loss of salient feature reconstruction, the first cross-entropy loss, the cross-modal hard triplet loss, the second cross-entropy loss, and the triplet loss.

[0166] Specifically, the total loss is:

[0167] .

[0168] Furthermore, during training, the parameters for the maximum size of image patches are gradually determined using a validation set. Boundary parameters in cross-modal hard triplet loss Weight parameters in the total loss of the modality-aware soft alignment module .

[0169] parameter The method for determining the value is as follows: The value is fixed at 0.01. Using mAP, Rank-1, Rank-5, and Rank-10 of the RGBNT201 dataset as indicators, different values ​​are set. Execute step S501 to observe the different values. Given the values ​​of mAP, Rank-1, Rank-5, and Rank-10, select the values ​​when mAP, Rank-1, Rank-5, and Rank-10 are at their highest levels. The value is taken as the final result. This embodiment selects the value based on experimental results. .

[0170] Boundary parameters The method for determining the value is: fixed The value is 0.5, using mAP, Rank-1, Rank-5, and Rank-10 of the RGBNT201 dataset as metrics, with different settings... Execute steps S501-S506 to observe the different values. Given the values ​​of mAP, Rank-1, Rank-5, and Rank-10, select the values ​​when mAP, Rank-1, Rank-5, and Rank-10 are at their highest levels. The value is taken as the final result. This embodiment selects the value based on experimental results. .

[0171] Weight parameters The method for determining the value is: fixed The value is 0.5. The value is 0.3, using mAP, Rank-1, Rank-5, and Rank-10 of the RGBNT201 dataset as metrics, with different settings... Execute steps S501-S506 to observe the different values. Given the values ​​of mAP, Rank-1, Rank-5, and Rank-10, select the values ​​when mAP, Rank-1, Rank-5, and Rank-10 are at their highest levels. The value is taken as the final result. This embodiment selects the value based on experimental results. .

[0172] In the training method of this embodiment, the mean squared error loss function and the focal frequency loss function are used to guide the reconstruction of salient features, effectively solving the problem of incomplete modality; based on the total loss of the modality-aware soft alignment module, the collaborative perception of multimodal features by multiple networks is promoted, and the shared features are enhanced.

[0173] This embodiment further verifies the multimodal pedestrian re-identification method based on complementary data augmentation through the following simulation.

[0174] Simulation Databases: This embodiment evaluates and validates the model on two multimodal person re-identification datasets: RGBNT201 and MARKET1501_RGBNT. RGBNT201 is the first multimodal person re-identification dataset, with 141 identities in the training set, 30 identities in the validation set, and 30 identities in the test set. MARKET1501_RGBNT is an extension of the Market1501 dataset, incorporating multimodal information. This dataset has 750 identities for training and 751 identities for testing.

[0175] Evaluation Criteria: Existing re-identification methods are compared with this implementation method. The cumulative matching characteristic (CMC) of first-place hit rate (RankR, R=1, 5, 10, expressed as a percentage) and mean average accuracy (mAP, expressed as a percentage) are used as evaluation metrics. Existing re-identification methods include: Multispectral Vehicle Re-identification (HAMNet), Robust Multimodal Pedestrian Re-identification (PFNet), Dynamic Augmentation Network: A Method for Partial Multimodal Pedestrian Re-identification (DENet), Low-Rank Fusion Network-Based Multimodal Pedestrian Re-identification (LRFNet), and Modality-Specific Characterization Enhancement Method for Multimodal Pedestrian Re-identification (IEEE). The evaluation results are as follows: Multimodal Consistency Collaborative Assisted Training Pedestrian Re-identification Method (MMCF); Key Marker Selection: Diverse Feature Selection Mechanism in Multimodal Target Re-identification (EDITOR); Application of Representation Selective Coupling Method Based on Marker Sparsity in Multispectral Target Re-identification (RSCNet); Heterogeneous Test-Time Training: Dynamic Adaptation Method for Multimodal Pedestrian Re-identification (HTT); Marker Permutation-Based Multispectral Target Re-identification Method (TOP-REID); Visual Mamba Model-Based Multimodal Target Re-identification Method (MambaReID); and Low-Rank Multi-Scale Multimodal Fusion Pedestrian Re-identification Method Based on RGB-NI-TI (LRMM). Please refer to Tables 1 and 2 for the evaluation results.

[0176] Table 1. Accuracy of multimodal person re-identification (RGBNT201 dataset)

[0177]

[0178] Table 2. Multimodal person re-identification accuracy (MARKET1501_RGBNT dataset)

[0179]

[0180] As shown in Tables 1 and 2, the method in this embodiment achieved excellent performance on both the RGBNT201 and MARKET1501_RGBNT datasets. On the RGBNT201 dataset, the mAP reached 74.6%, an improvement of at least 2.3% over the existing state-of-the-art TOP-REID method; the Rank-1 accuracy was 77.6%, an improvement of 1.0% over the existing state-of-the-art TOP-REID method. On the MARKET1501_RGBNT dataset, the mAP reached 82.5%, an improvement of 2.1% over the existing state-of-the-art TOP-REID method; the Rank-1 accuracy was 92.7%, an improvement of 0.5% over the existing state-of-the-art TOP-REID method.

[0181] The method presented in this embodiment significantly improves the robustness and inference efficiency of multimodal ReID, addressing issues such as modal interaction redundancy, rigid feature alignment, and low-quality modal noise propagation found in existing methods. By employing HSV spatial enhancement and RGB-NI-TI cross-modal image patch rotation strategies, complementary information from multiple data sources is fully exploited. Combined with global feature reconstruction constrained by frequency domain focus loss and cross-modal hard sample triplet alignment, efficient feature fusion and noise suppression are achieved, significantly improving cross-modal feature alignment efficiency and noise immunity. This method is suitable for all-weather intelligent monitoring systems in low-light, complex occlusion, and multimodal missing scenarios, providing a high-precision, low-latency solution for practical deployment.

[0182] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A multimodal person re-identification method based on complementary data augmentation, characterized in that, Including the following steps: Rotating image patches of the initial modality image data to obtain a rotated image of the initial modality image includes: initializing probabilities for images of the same identity and the same sample in the initial modality image data. When probability When the value is less than or equal to a preset value, images of the same identity and the same sample are retained; when the probability is less than or equal to a preset value, images of the same identity and the same sample are retained. When the value is greater than a preset value, the position and size of the image blocks are randomly initialized, and image blocks are cut out from the same position of each modal image according to the position and size of the image blocks. Then, the image blocks of different modalities are cyclically replaced to obtain a rotated image of each initial modal image. The initial modal image data includes at least one of RGB image, TI image, and NI image. The rotated image of the initial modality image is augmented using dual color space data to obtain the initial modality-enhanced image; The trained feature extraction module is used to extract global semantic information and local detail information from the initial modality enhancement image to obtain the initial modality image features; When the initial modal image data includes one or two of RGB images, TI images, and NI images, based on the initial modal image features, different modal reconstructors in the trained modal reconstruction module are used to generate predicted features of the missing modalities, thus obtaining multimodal image features. The modal reconstruction module includes a first modal reconstructor that generates TI modal image features from RGB modal image features, a second modal reconstructor that generates NI modal image features from RGB modal image features, a third modal reconstructor that generates RGB modal image features from TI modal image features, a fourth modal reconstructor that generates NI modal image features from TI modal image features, a fifth modal reconstructor that generates RGB modal image features from NI modal image features, and a sixth modal reconstructor that generates TI modal image features from NI modal image features. The first modality reconstructor, the second modality reconstructor, the third modality reconstructor, the fourth modality reconstructor, the fifth modality reconstructor, and the sixth modality reconstructor have the same structure, each including: a first normalization layer, used to perform layer normalization processing on the input initial modality image features to obtain first-layer normalized features; a multi-head self-attention module, used to process the first-layer normalized features using a multi-head self-attention mechanism to obtain multi-head self-attention output features; a first residual fusion module, used to add and fuse the input initial modality image features with the multi-head self-attention output features to obtain fused features; a second normalization layer, used to perform layer normalization processing on the fused features to obtain second-layer normalized features; a multilayer perceptron, used to generate preliminary reconstructed features based on the second-layer normalized features; and a second residual fusion module, used to add and fuse the fused features with the preliminary reconstructed features to obtain the predicted features of the missing modality. When the initial modal image data includes three types of images: RGB, TI, and NI, the initial modal image features constitute multimodal image features. By concatenating the category labels of the same identity and the same sample in the multimodal image features, positive samples are obtained, and pedestrian re-identification results are obtained.

2. The multimodal pedestrian re-identification method based on complementary data augmentation according to claim 1, characterized in that, The rotated image of the initial modality image is augmented using dual color space data to obtain the initial modality-enhanced image, including: When the initial modal image data includes an RGB image, the saturation and brightness of the rotated image of the RGB image are enhanced to obtain an enhanced RGB image; when the initial modal image data includes a TI image, the rotated image of the TI image is used as an enhanced TI image; when the initial modal image data includes an NI image, the rotated image of the NI image is used as an enhanced NI image.

3. The multimodal pedestrian re-identification method based on complementary data augmentation according to claim 2, characterized in that, The saturation and brightness of the rotated RGB image are enhanced to obtain an enhanced RGB image, including: The rotated RGB image is converted to HSV space to obtain the converted image; Calculate the saturation and brightness of the converted image, wherein the formulas for calculating the saturation and brightness are: ； ； in, For saturation, For brightness, The maximum pixel value of the R, G, and B channels. The minimum pixel value for the R, G, and B channels; The converted image is then subjected to saturation and brightness enhancement to obtain an SV-enhanced image, wherein the enhancement formulas for saturation and brightness are: ； in, To enhance the saturation, For enhanced brightness; The SV enhanced image is converted to RGB space to obtain the enhanced RGB image.

4. The multimodal person re-identification method based on complementary data augmentation according to claim 1, characterized in that, The trained feature extraction module is used to extract global semantic information and local detail information from the initial modality enhancement image to obtain the initial modality image features, including: Under different modalities, the trained ViT-B / 16 network is used to extract global semantic information and local detail information from the initial modality enhancement image to obtain the initial modality image features, wherein the initial modality image features include category labels representing the global semantic information of the image.

5. The multimodal person re-identification method based on complementary data augmentation according to claim 1, characterized in that, The training methods for the feature extraction module and the modality reconstruction module include the following steps: Multimodal training data is augmented using dual color space data to obtain multimodal augmented training data, wherein the multimodal training data includes RGB images, TI images, and NI images; The feature extraction module is used to extract global semantic information and local detail information from the multimodal augmented training data to obtain multimodal image training features; The modality reconstruction module is trained using the multimodal image training features, and constrained by the total loss of salient feature reconstruction. The total loss of salient feature reconstruction includes the total loss of generating NI and TI modalities from RGB modality, the total loss of generating RGB and TI modalities from NI modality, and the total loss of generating RGB and NI modalities from TI modality. From the multimodal image training features, class labels of the same identity and the same sample are selected and concatenated to obtain training positive samples, and class labels of samples of different identities or samples of the same identity but different samples are randomly selected and concatenated to obtain training negative samples. The training positive samples and the training negative samples are input into a binary classifier for discrimination. Combining the predicted labels output by the binary classifier, the modality-aware soft alignment total loss is used to constrain the binary classifier and the feature extraction module. The modality-aware soft alignment total loss includes the first cross-entropy loss and the cross-modal hard triplet loss. The training positive samples are input into a linear classification layer to classify different identities; The linear classification layer and the feature extraction module are constrained using the second cross-entropy loss and the triplet loss; Repeat the iteration until the total loss converges to obtain the trained feature extraction module and the trained modality reconstruction module, wherein the total loss includes the total loss of the salient feature reconstruction, the first cross-entropy loss, the cross-modal hard triplet loss, the second cross-entropy loss and the triplet loss.

6. The multimodal person re-identification method based on complementary data augmentation according to claim 5, characterized in that, The total loss for the reconstruction of the salient features is: ； ； ； ； in, The total loss is used to reconstruct salient features. The total loss for generating NI and TI modes from RGB modes. The total loss for generating RGB and TI modes from NI modes. This represents the total loss for generating RGB and NI modes from TI modes; ； ； ； ； ； ； in, To generate NI mode from RGB mode The mean square error loss, The number of image patches embedded in the generated modal features. NI modal image features The An embedded image patch, NI modal feature map generated from RGB modalities The One embedded image block; To generate TI modes from RGB modes The mean square error loss, TI modal image features The An embedded image patch, TI modal feature map generated from RGB modalities The One embedded image block; To generate RGB modes from NI modes The mean square error loss, RGB modal image features The An embedded image patch, RGB modal feature maps generated from NI modalities The One embedded image block; To generate TI modes from NI modes The mean square error loss, TI modal feature map generated from NI modality The One embedded image block; To generate RGB modes from TI modes The mean square error loss, RGB modal feature map generated by TI modality The Image blocks; To generate NI modes from TI modes The mean square error loss, NI modal feature map generated from TI modalities The One embedded image block; , , , To enhance RGB images, This is the first ViT-B / 16 network in RGB mode. To enhance TI images, This is the second ViT-B / 16 network in TI mode. To enhance NI images, This is the third ViT-B / 16 network in the NI mode; , , , , , , For the first mode reconfigurator, For the second modal reconfigurator, For the third modal reconfigurator, For the fourth mode reconfigurator, For the fifth mode reconfigurator, For the sixth mode reconfigurator; ； ； ； ； ； ； in, For focus frequency loss, for , , , , or , Dimensions for labeling categories, The first category label A spectral weight matrix with multiple dimensions. , for The first category label in the middle Fast Fourier Transform in 1000 Dimensions Represents RGB, NI, or TI. for The first category label in the middle Fast Fourier Transform in 1000 Dimensions; The total loss of the modality-aware soft alignment is: ； ； ； ； ； in, The total loss for modality-aware soft alignment is... For weight parameters, For the first cross-entropy loss, The number of samples in the batch. It is the first The true label of each sample It is the first output of the binary classifier Predicted label for each sample, For RGB cross-modal hard triplet loss, For NI cross-modal hard triplet loss, For TI cross-modal hard triplet loss, For boundary parameters, Indicates negative samples Perform hard sample mining. Represents the distance metric function. For the first Features of each anchor point sample The most difficult positive sample Features For the most difficult negative sample Features; The second cross-entropy loss is: ； in, For the second cross-entropy loss, The number of samples in the batch. For the number of identities, In the constructed multimodal training positive samples, the first One sample, For the first Does each sample belong to a certain identity? The true label, It is the first Each sample belongs to a certain identity. Predicted labels; The triplet loss is: ； in, For triple loss, The first of the multimodal positive samples constructed during training An identity, For the first Hard sample mining is performed on negative samples from each identity. In the constructed multimodal training positive samples, the first Anchor point samples of each identity, In the constructed multimodal training positive samples, the first Positive samples of each identity. In the constructed multimodal training positive samples, the first Negative samples of each identity, Anchor point samples for triples, For positive samples of triples, For negative samples of triples, The square of the Euclidean distance. For interval parameters; The total loss is: 。 7. The multimodal person re-identification method based on complementary data augmentation according to claim 5, characterized in that, From the multimodal image training features, class labels of the same identity and the same sample are selected and concatenated to obtain positive training samples, and class labels of samples with different identities or samples with the same identity but different identities are randomly selected and concatenated to obtain negative training samples, including: The class labels of the same identity and the same sample in the multimodal image training features are concatenated to obtain positive training samples, and a first pseudo-label is assigned to the positive training samples: ； in, Indicates the first The labels of the training positive samples, The first feature in training RGB modality images Category labeling of each sample, The first feature in the training of NI modality images Category labeling of each sample, The first feature in the training of TI modality images The category label of each sample; From the multimodal image training features, class labels of samples with different identities or samples with the same identity but different identities are randomly selected and concatenated to obtain training negative samples, and a second pseudo-label is assigned to the training negative samples: ； in, For the first The labels of each training negative sample. The first feature in training RGB modality images Category labeling of each sample, The first feature in the training of NI modality images Category labeling of each sample, The first feature in the training of TI modality images Category labeling of each sample, , There are at least two different values.

Images