A cross-modal person re-identification method based on feature enhancement

Abstract

This invention discloses a cross-modal person re-identification method based on feature enhancement, belonging to the field of computer vision. The method comprises the following steps: Step 1: Acquire and preprocess a dataset, and generate new modalities using a channel enhancement strategy; Step 2: Construct a cross-modal person re-identification network model based on feature enhancement; Step 3: Construct a loss function and train and optimize the model based on the training dataset and the loss function; Step 4: Perform cross-modal person re-identification based on the trained model. This invention improves the accuracy of the cross-modal person re-identification model through feature enhancement.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and specifically relates to a cross-modal pedestrian re-identification method based on feature enhancement. Background Art

[0002] In modern society, intelligent surveillance systems play a key role in maintaining public safety and improving urban management efficiency. As the core of the surveillance system, pedestrian re-identification technology has the main task of identifying and tracking the same pedestrian in images captured by different cameras. Although visible light images provide rich visual information for pedestrian recognition, the recognition effect will drop significantly at night or under poor lighting conditions. To overcome this limitation, cross-modal pedestrian re-identification technology has emerged, which combines visible light and infrared images to achieve stable recognition under various lighting conditions. However, the current cross-modal pedestrian re-identification dataset is limited in size, and due to insufficient training data, there is a high possibility of overfitting. A feasible solution to this problem is to implement feature enhancement technology during the training process.

[0003] The continuous advancement of deep learning technology, especially the introduction of the Vision Transformer architecture, has provided a new perspective for processing this type of cross-modal data. This paper explores the use of feature enhancement algorithms to optimize the cross-modal person re-identification model, aiming to improve its recognition accuracy and robustness under different imaging conditions, thereby achieving more reliable pedestrian recognition in all-weather surveillance scenarios. Summary of the Invention

[0004] To address these issues, this paper proposes a cross-modal person re-identification method based on feature enhancement. This method uses a parameter-sharing Vision Transformer as the backbone network. During the patch embedding phase, the crossover and mutation processes of biological evolution are simulated, and an attention fusion feature module and a Gaussian mutation feature module are proposed. This not only optimizes the effect of feature fusion but also enables the model to better adapt to differences between cross-modal data, improving its adaptability to environmental changes.

[0005] The technical solutions of the present invention are as follows:

[0006] A cross-modal person re-identification method based on feature enhancement includes the following steps:

[0007] Step 1: Obtain the data set and preprocess it, and use the channel enhancement strategy to generate a new modality;

[0008] Step 2: Construct a cross-modal person re-identification network model based on feature enhancement;

[0009] Step 3: Construct a loss function and train and optimize the model based on the training data set and loss function;

[0010] Step 4: Perform cross-modal person re-identification based on the trained model.

[0011] Furthermore, the specific process of step 1 is as follows:

[0012] Step 1.1: Obtain the public datasets SYSU-MM01 and RegDB as training datasets. Collect all pedestrian images from various surveillance cameras as the test dataset. The test dataset consists of two parts: a query set and a gallery set. The query set is the set of pedestrian images to be queried, and the gallery set is the set of candidate pedestrian images that match the query set. The data in the dataset includes both visible light images and infrared images.

[0013] Step 1.2: Generate new enhanced images using the channel enhancement method for the visible light images in the training dataset. The training dataset now contains images of three modalities: visible light images, infrared images, and enhanced images, with one image corresponding to each modality.

[0014] Step 1.3: Perform image preprocessing on the three modal images in the training dataset. Perform random horizontal flipping and regularization on the visible light images, Gaussian blur and brightness adjustment on the infrared images, and random erasure on the enhanced images.

[0015] Step 1.4: Resize the images of the three modalities to 256*128 pixels.

[0016] Furthermore, in step 2, the cross-modal person re-identification network model includes three parts: a patch embedding part, a feature enhancement algorithm part, and an encoding part; the patch embedding part includes image segmentation and embedding operations; the feature enhancement algorithm part includes an attention fusion feature module and a Gaussian variation feature module; the encoding part includes three weight-sharing encoders;

[0017] The working process of the cross-modal person re-identification network model is as follows:

[0018] Step 2.1: Perform image segmentation and embedding in the patch embedding part to obtain a series of patch sets p of the image set t t ;

[0019] Step 2.2, obtain the fused patches based on the attention fusion feature module and update the patch set;

[0020] Step 2.3: The Gaussian mutation feature module uses the Gaussian mutation strategy to mutate the patch and update the patch set;

[0021] Step 2.4: transform the original input image set t into a series of patch sets p tInput feature enhancement algorithm part, the attention fusion feature module and Gaussian variation feature module in the feature enhancement algorithm part are executed in parallel according to the process of step 2.2 and step 2.3, and finally a new series of patch sets p are optimized. t′ , and p t′ The input is fed into three weight-sharing encoders, where one encoder processes the image of one modality to obtain predicted value features for each modality.

[0022] Furthermore, the specific process of step 2.1 is: Infrared Image CollectionX r = Enhanced Image Set Input to the model, where is the i-th visible light image, N v is the number of visible light images; It is Infrared images, N r is the number of infrared images; It is enhanced images, N c is the number of enhanced images; for the visible light, infrared and enhanced image sets, the corresponding identification label sets are defined, namely the visible light image identification label set Y v = Infrared image identification label set and enhanced image identification label set Define image set X t Represents the visible light image set X v , infrared image set X r Or enhanced image set X c One of them, X t ∈{X v ,X r ,X c};

[0023] Before inputting into the model, the input image needs to be patch-embedded: each image in the image set is divided into a series of patches, and the patches are mapped to a high-dimensional feature space to obtain the corresponding patch set. The formula is:

[0024] p t =PatchEmbedding(X t );

[0025] Among them, p t For image set X t is a series of patch sets; PatchEmbedding(·) is the patch embedding operation; the number of images in the image set corresponds to the number of patch sets.

[0026] Furthermore, the specific process of step 2.2 is as follows:

[0027] Step 2.2.1: Pre-set the crossover rate and randomly select two different patch sets with the same identity based on the crossover rate. Perform attention fusion feature operation, p a The i-th and I-th patches in ; p b The j-th and J-th patches in , multiply these two sets of patches and normalize them to construct the attention matrix, the formula is:

[0028] W=softmax(p a *p b );

[0029] Where W is the attention matrix; softmax(·) is the softmax function;

[0030] Step 2.2.2, from patch set p a Randomly select a patch from , using the attention matrix, find the patch set p b Zhongyu The most similar patch Determine the current The index j and the adjacent indexes around j are valid indexes; according to the valid index from p b Extract the corresponding eigenvalues ​​and extract the attention scores corresponding to the indices from the attention matrix W. Then, renormalize the extracted attention scores and construct a new attention matrix, specifically:

[0031] W′=softmax(w j ),j∈vaild indices ;

[0032] Among them, W′ is the new attention matrix; w j is the attention score at index j in the attention matrix W; vaild indices is a valid index;

[0033] Step 2.2.3. According to the new attention matrix and the corresponding eigenvalue, the fused patch is obtained. The formula is:

[0034]

[0035] Among them, p r is the fused patch; w j ′ is the attention score at index j in the new attention matrix W′, and the attention matrix corresponds to the index number of the new attention matrix;

[0036] Step 2.2.4, finally, p a in Replaced with the fused patch p r , get the updated patch set

[0037] Furthermore, the specific process of step 2.3 is as follows: first, define a series of input patches with the same identity as the original patch set p s , calculate p s Statistics, including mean and variance; according to the calculated mean μ and variance σ 2 , defines a Gaussian distribution N(μ,σ 2 ); from p s A series of patches of one of the identities are selected to form the original mutation patch set s is a specific identity, m is a sample in a specific identity, For p m Middle Patches, For p m Middle patches; sample from the Gaussian distribution and traverse the original mutation patch set p m For each patch in , the mutation rate is pre-set, and the individuals are replaced by points sampled from the Gaussian distribution according to the mutation rate; the calculation formulas for the mean and variance are:

[0038]

[0039]

[0040] in, For the original patch set p s The hth patch in the set; H is the original patch set p s The number of patches in

[0041] New feature sample p extracted from the Gaussian model new It conforms to the Gaussian distribution, specifically:

[0042] p new ~N(μ,σ 2 );

[0043] Finally, the samples in the original mutation patch set Replace with the new feature sample p obtained by sampling new , get the new mutation patch set

[0044] Furthermore, the formula of step 2.4 is:

[0045]

[0046] Among them, F t For image set X t The predicted value characteristics of is a weight-sharing encoder.

[0047] Furthermore, the specific process of step 3 is as follows:

[0048] Step 3.1: Input the predicted value features of each image set into the classification layer to obtain the predicted probability of the image for each identity, and calculate the identity loss L based on the predicted probability. id :

[0049] S t =Classification(F t );

[0050]

[0051] Among them, S t For image set X t The predicted probability of Classification(·) is the classification layer; Y t Represents the image set X t The true label of represents the total number of identity samples; Represents the image set X t For the first The predicted probability of each identity;

[0052] Step 3.2: Calculate the triplet loss L of the image set tri :

[0053]

[0054] Among them, F is the predicted value feature; Represents a randomly selected sample image; o represents Positive sample images with the same sample identity; o′ represents Negative sample images with different sample identities; margin is the parameter boundary;

[0055] Step 3.3: Calculate the discriminative center loss L of the image set dcl :

[0056]

[0057] in, for The center vector of For the identities; K represents the number of predictive value features of the same modality in the same identity; For visible light images The z-th predicted value feature; In infrared images The k-th predicted value feature; To remove The predicted value characteristics of all identities except The average distance of the center vector of z， Indicates the The predicted value features of the z′th sample with different identities; F k′ For the The predicted value features of the k′th sample with the same identity;

[0058] Step 3.4: Construct the least squares error loss function L msel , randomly select a prediction value feature of an infrared image or a visible light image, denoted as Prediction value features of infrared images or Prediction value features of visible light images First calculate The average distance to other samples of the same identity in intramodality and cross-modality is calculated as:

[0059]

[0060] Among them, S intra is the average distance under the internal mode; S cross is the average distance under cross-modality; S(·) is the Euclidean distance; For the The predicted value features of infrared images; For the The predicted value features of visible light images;

[0061] Calculate in the same way as above the average distance to other samples of the same identity in both intramodality and crossmodality;

[0062] Calculate the difference L between the mean distance under intramodality and the mean distance under cross-modality msel :

[0063]

[0064] in, For the same identity The average distance under the internal mode corresponding to the predicted value features; For the same identity The average cross-modal distance corresponding to the predicted value features;

[0065] Step 3.5. Finally, the overall loss function L of the training process is:

[0066] L=λL msel +λL dcl +L id +L tri ;

[0067] Among them, λ is a hyperparameter.

[0068] Furthermore, the specific process of step 4 is as follows:

[0069] Step 4.1: Use the query set and gallery set of the test dataset as input to the cross-modal person re-identification model trained in Step 3. Concatenate the image features of the three modalities output by the model in the channel dimension to obtain the final pedestrian prediction value features.

[0070] Step 4.2: Calculate the similarity between the pedestrian images in the query set and the pedestrian images in the gallery set. The similarity calculation formula is:

[0071]

[0072] in, Pedestrian images in the query set Pedestrian images with gallery set similarity; Pedestrian images representing the query set The predicted value characteristics of Pedestrian images representing the gallery set The predicted value features; |·| represents the modulus;

[0073] Step 4.3: Sort all similarity values ​​in descending order, and output the images corresponding to the pedestrian prediction value features with the highest similarity values ​​as the re-identification results.

[0074] The beneficial technical effects of the present invention are as follows: The method of the present invention proposes cross-modal person re-identification based on feature enhancement to address the significant differences between visible light and infrared images. By simulating the crossover and mutation operations in biological evolution, it aims to optimize the feature fusion process. This approach not only improves the model's adaptability to changes between different modalities, but also enhances its robustness to interference factors such as noise and occlusion. Through the feature enhancement strategy, the model can automatically explore and utilize complementary information in multimodal data, thereby achieving higher accuracy in complex cross-modal recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 This is a flowchart of the cross-modal pedestrian re-identification method based on feature enhancement of the present invention.

[0076] Figure 2 This is a structural diagram of the cross-modal pedestrian re-identification model based on feature enhancement in the present invention.

[0077] Figure 3 for Figure 2 Schematic diagram of the attention fusion feature module.

[0078] Figure 4 for Figure 2 Schematic diagram of the Gaussian variation feature module. DETAILED DESCRIPTION

[0079] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0080] First, the following terms are explained:

[0081] PatchEmbedding: PatchEmbedding is a technology commonly used in Vision Transformer. Its core function is to split the input image into multiple small blocks and convert each small block into a vector of fixed dimension so that the subsequent Transformer model can process it.

[0082] Softmax: Softmax is a commonly used activation function, primarily used in multi-classification problems, to convert real-valued inputs into probability distributions. It is widely used in machine learning and deep learning, especially in the output layer of neural networks, to convert model outputs into class probabilities.

[0083] CA: Channel Attention (CA) is a technique used in deep learning, particularly in image processing, to enhance a model's attention to image channels (features). The core idea behind this technique is to enable the model to adaptively emphasize important channel features while suppressing less important ones. CA is commonly used in convolutional neural networks (CNNs) to improve the model's ability to represent image features.

[0084] Vision transformer: The Vision Transformer (ViT) is a deep learning model based on the Transformer architecture. It was originally designed for natural language processing (NLP) tasks, but has since been successfully applied to computer vision, particularly image classification. The core idea of ​​ViT is to segment an image into multiple small patches (patches), then treat these patches as words or tokens in a sequence, and use the Transformer model to process these sequences to achieve image understanding and classification.

[0085] SYSU-MM01 Dataset: The SYSU-MM01 dataset was created by researchers at South China University of Technology to address the problem of identifying the same person across different imaging modalities. SYSU-MM01, proposed in 2017, uses six cameras, including two infrared cameras and four RGB cameras. It includes RGB and IR images of 491 identities, resulting in a total of 287,628 RGB images and 15,792 IR images. This dataset has two modes: full search mode and indoor search mode.

[0086] RegDB Dataset: The RegDB dataset, proposed in 2017, is a small-scale dataset collected using a dual-camera system consisting of a visible light camera and a thermal camera. The dataset contains 412 person identities, each with 10 visible light images and 10 infrared images. The database contains 4,120 visible light images and 4,120 corresponding infrared images. The training and test sets each contain 206 pedestrians.

[0087] like Figure 1 As shown, the method of the present invention comprises the following steps:

[0088] Step 1: Obtain the data set and preprocess it, and use the channel enhancement strategy to generate a new modality. The specific process is as follows:

[0089] Step 1.1: Obtain the public datasets SYSU-MM01 and RegDB as training datasets. Collect all pedestrian images from various surveillance cameras as the test dataset. The test dataset consists of two parts: a query set and a gallery set. The query set is the set of pedestrian images to be queried, and the gallery set is the set of candidate pedestrian images that match the query set. The data in the dataset are visible light images and infrared images.

[0090] Step 1.2: Generate new enhanced images using the channel enhancement method for the visible light images in the training dataset. The training dataset now contains images of three modalities: visible light images, infrared images, and enhanced images, with one image corresponding to each modality.

[0091] Step 1.3: Perform image preprocessing on the three modal images in the training dataset. This includes performing random horizontal flipping and regularization on visible light images, Gaussian blurring and brightness adjustment on infrared images, and random erasing on enhanced images.

[0092] Step 1.4: Resize the images of the three modalities to 256*128 pixels.

[0093] Step 2: Construct a cross-modal person re-identification network model based on feature enhancement, which incorporates the attention fusion feature module and the Gaussian variation feature module;

[0094] The cross-modal person re-identification network model based on feature enhancement consists of three parts: patch embedding part, feature enhancement algorithm part and encoding part; the patch embedding part includes image segmentation and embedding operations; the feature enhancement algorithm part includes attention fusion feature module and Gaussian variation feature module; the encoding part is a Vision transformer structure, which includes regularization, attention mechanism module, etc.

[0095] like Figure 2 、 Figure 3 、 Figure 4 As shown in Figure 2, the working process of the cross-modal person re-identification network model based on feature enhancement is as follows:

[0096] Step 2.1: Perform image segmentation and embedding in the patch embedding part; the specific process is: transform the visible light image set (RGB image) Infrared image collection Enhanced Image Set Input to the model, where is the i-th visible light image, N v is the number of visible light images; It is Infrared images, N r is the number of infrared images; It is enhanced images, N c is the number of enhanced images. For the visible light, infrared and enhanced image sets, the corresponding identification label sets are defined, namely the visible light image identification label set Infrared image identification label set and enhanced image identification label set Its label candidate set is shared. For simplicity, define the image set X t Represents the visible light image set X v , infrared image set X r Or enhanced image set X c One of them, X t ∈{X v,X r ,X c}.

[0097] Before inputting into the model, the input image needs to be patch-embedded: each image in the image set is divided into a series of patches, and the patches are mapped to a high-dimensional feature space to obtain the corresponding patch set. The formula is:

[0098] p t =PatchEmbedding(X t );

[0099] Among them, p t For image set X t A series of patch sets; PatchEmbedding(·) is the patch embedding operation; the number of images in the image set corresponds to the number of patch sets;

[0100] Step 2.2: Obtain the fused patches based on the attention fusion feature module and update the patch set. The specific process is as follows:

[0101] Step 2.2.1: Preset the crossover rate (set to 10% in this invention) and randomly select two sets of patches with the same identity based on the crossover rate. Perform attention fusion feature operation, p a The i-th and I-th patches in ; p b The j-th and J-th patches in , multiply these two sets of patches and normalize them to construct the attention matrix, the formula is:

[0102] W=sogmtax(p a *p b );

[0103] Where W is the attention matrix; softmax(·) is the softmax function; p a 、p b For two different sets of patches;

[0104] Step 2.2.2, from patch set p a Randomly select a patch from Using the attention matrix, find the patch set p b Zhongyu The most similar patch Determine the current The index j and the adjacent indexes around j (such as j+1, j-1) are valid indexes. bExtract the corresponding eigenvalues ​​from the attention matrix W, extract the corresponding attention scores, and then renormalize the extracted attention scores and construct a new attention matrix, specifically:

[0105] W′=softmax(w j ),j∈vaild indices ;

[0106] Among them, W′ is the new attention matrix; w j is the attention score at index j in the attention matrix W; vaild indices is a valid index;

[0107] Step 2.2.3. According to the new attention matrix and the corresponding eigenvalue, the fused patch is obtained. The formula is:

[0108]

[0109] Among them, p r is the fused patch; w j ′ is the attention score at index j in the new attention matrix W′, and the attention matrix corresponds to the index number of the new attention matrix;

[0110] Step 2.2.4, finally, p a in Replaced with the fused patch p r , get the updated patch set

[0111] Step 2.3, Gaussian mutation feature module uses Gaussian mutation strategy to mutate the patch and update the patch set; the core of the mutation strategy is to select elements from the input tensor, perform splicing and statistical analysis, and random mutation based on Gaussian distribution. First, define a series of patches with the same identity as the input as the original patch set p s , calculate p s Statistics, including mean and variance. These statistics are used to define a Gaussian distribution, which will be used to generate random mutation points. Based on the calculated mean μ and variance σ 2 , defines a Gaussian distribution N(μ,σ 2 ). From p s A series of patches of one of the identities are selected to form the original mutation patch set s is a specific identity, m is a sample in a specific identity, For p m Middle Patches, For p m Middle patches; sample from the Gaussian distribution and traverse the original mutation patch set p m For each patch in , a mutation rate is pre-set (set to 20% in this invention), and individuals are replaced with points sampled from a Gaussian distribution according to the mutation rate. The calculation formulas for the mean and variance are:

[0112]

[0113]

[0114] in, For the original patch set p s The hth patch in the set; H is the original patch set p s The number of patches in the .

[0115] New feature sample p extracted from the Gaussian model new It conforms to the Gaussian distribution, specifically:

[0116] p new ~N(μ,σ 2 );

[0117] Finally, the samples in the original mutation patch set Replace with the new feature sample p obtained by sampling new , get the new mutation patch set

[0118] Step 2.4: transform the original input image set t into a series of patch sets p t Input feature enhancement algorithm part, the attention fusion feature module and Gaussian variation feature module in the feature enhancement algorithm part are executed in parallel according to the process of step 2.3 and step 2.4, and finally a new series of patch sets p are optimized. t′ , and p t′ The input is fed into three weight-sharing encoders, where one encoder processes the image of one modality to obtain the predicted value features of each modality, specifically:

[0119]

[0120] Among them, F t For image set X t The predicted value characteristics of is a weight-sharing encoder.

[0121] Step 3: Construct a loss function and train and optimize the model based on the training dataset and loss function. The specific process is as follows:

[0122] Step 3.1: Input the predicted value features of each image set into the classification layer to obtain the predicted probability of the image for each identity, and calculate the identity loss L based on the predicted probability. id, the calculation formula is as follows:

[0123] S t =Classification(F t );

[0124]

[0125] Among them, S t For image set X t The predicted probability of Classification(·) is the classification layer; Y t Represents the image set X t The true label of represents the total number of identity samples; Represents the image set X t For the first The predicted probability of an identity.

[0126] Step 3.2: Calculate the triplet loss L of the image set tri , the calculation formula is as follows:

[0127]

[0128] Among them, F is the predicted value feature; Represents a randomly selected sample image; o represents Positive sample images with the same sample identity; o′ represents Negative sample images with different sample identities; margin is the parameter boundary, set to 0.3.

[0129] Step 3.3, calculate the identification center loss of the image set, the identification center loss L dcl The calculation formula is as follows:

[0130]

[0131] in, for The center vector of For the identities; K represents the number of predictive value features of the same modality in the same identity; For visible light images The z-th predicted value feature; In infrared images The k-th predicted value feature; To remove The predicted value characteristics of all identities except The average distance of the center vector of z′ Indicates the The predicted value features of the z′th sample with different identities; Fk′ For the The predicted value features of the k′th sample with the same identity;

[0132] Step 3.4: Construct the least squares error loss function L msel , randomly select a prediction value feature of an infrared image or a visible light image, denoted as Prediction value features of infrared images or Prediction value features of visible light images First calculate The average distance to other samples of the same identity in intramodality and cross-modality is calculated as:

[0133]

[0134] Among them, D intra is the average distance under the internal mode; D cross is the average distance under cross-modality; D(·) is the Euclidean distance; For the The predicted value features of infrared images; For the The predicted value features of visible light images;

[0135] Calculate in the same way as above the average distance to other samples of the same identity in both intramodality and crossmodality;

[0136] Calculate the difference L between the mean distance under intramodality and the mean distance under cross-modality msel :

[0137]

[0138] in, For the same identity The average distance under the internal mode corresponding to the predicted value features; For the same identity The average cross-modal distance corresponding to the predicted value features;

[0139] Step 3.5. Finally, the overall loss function L of the training process is defined as:

[0140] L=λL msel +λL dcl +L id +L tri ;

[0141] Among them, λ is a hyperparameter, which is set to 0.5 to balance the importance of loss.

[0142] The cross-modal person re-identification model is constrained by an overall loss function to train and optimize to obtain a more effective and robust cross-modal person re-identification model.

[0143] Step 4: Perform cross-modal person re-identification based on the trained model. The specific steps are as follows:

[0144] Step 4.1: Use the query set and gallery set of the test dataset as input to the cross-modal person re-identification model trained in Step 3. Concatenate the image features of the three modalities output by the model in the channel dimension to obtain the final pedestrian prediction value features.

[0145] Step 4.2: Calculate the similarity between the pedestrian images in the query set and the pedestrian images in the gallery set;

[0146] The similarity calculation formula is:

[0147]

[0148] in, Pedestrian images in the query set Pedestrian images with gallery set similarity; Pedestrian images representing the query set The predicted value characteristics of Pedestrian images representing the gallery set The predicted value features; |·| represents the modulus;

[0149] Step 4.3: Sort all similarity values ​​in descending order, and output the images corresponding to the pedestrian prediction value features with the highest similarity values ​​as the re-identification results.

[0150] In the embodiment of the present invention, the dimension of the feature vector finally identified is 768. The present invention is implemented under the PyTorch framework, and the Adam algorithm is used to optimize the model, with the learning rate set to 3.5e-4 and the maximum number of iterations being 100.

[0151] To verify the feasibility and superiority of the present invention, the following comparative experiments were conducted on two cross-modal pedestrian datasets: SYSU-MM01 and RegDB.

[0152] Four methods, SPOT, FMCNet, PMT, and DART, were selected for cross-modal pedestrian re-identification, and the recognition results were compared with those of the present invention. The comparison results are shown in Table 1. The SPOT method proposed a structure-aware position transformer network, which uses human body structure information to enhance the representation ability of cross-modal features, thereby improving the accuracy of pedestrian recognition under different lighting and environmental conditions. The FMCNet method proposed a feature-level modality compensation network to improve the performance of visible light-infrared pedestrian re-identification by compensating for the missing modality-specific information. The PMT method proposed a progressive learning strategy by using the grayscale image as an auxiliary modality and using the Vision Transformer model to learn modality-independent features. The DART method proposed a new framework for the dual-noise label problem in visible light-infrared pedestrian re-identification, correcting the noise correspondence by estimating the confidence of the clean annotation, and dividing the data into four groups to achieve robust learning. The present invention uses two evaluation indicators, Rank-1 and mean average precision (mAP), to evaluate the trained model. The higher the values ​​of Rank-1 and mAP, the higher the model accuracy.

[0153] Table 1 Comparison results of the method of the present invention with other methods on the SYSU-MM01 and RegDB datasets

[0154]

[0155]

[0156] As shown in Table 1, the proposed method achieves Rank-1 scores of 73.02% and 90.49% and mAP values ​​of 68.88% and 87.45% on the SYSU-MM01 and RegDB datasets, respectively, for cross-modal person re-identification. This method achieves optimal results on both the SYSU-MM01 and RegDB datasets, effectively improving the accuracy of cross-modal person re-identification.

[0157] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the scope of protection of the present invention.

Claims

1. A cross-modal person re-identification method based on feature enhancement, characterized in that: The steps include: Step 1: Obtain the data set and preprocess it, and use the channel enhancement strategy to generate a new modality; Step 2: Construct a cross-modal person re-identification network model based on feature enhancement; The cross-modal person re-identification network model consists of three parts: patch embedding, feature enhancement algorithm, and encoding. The patch embedding part includes image segmentation and embedding operations; the feature enhancement algorithm includes an attention fusion feature module and a Gaussian variation feature module; and the encoding part includes three weight-sharing encoders. The working process of the cross-modal person re-identification network model is as follows: Step 2.1: Perform image segmentation and embedding in the patch embedding part to obtain a series of patch sets p of the image set t t ; The specific process is: Infrared image collection Enhanced Image Set Input to the model, where It is visible light images, N v is the number of visible light images; It is Infrared images, N r is the number of infrared images; It is enhanced images, N c is the number of enhanced images; for the visible light, infrared and enhanced image sets, the corresponding identification label sets are defined, namely the visible light image identification label set Infrared image identification label set and enhanced image identification label set Define image set X t Represents the visible light image set X v , infrared image set X r Or enhanced image set X c One of them, X t ∈{X v ,X r ,X c }; Before inputting into the model, the input image needs to be patch-embedded: each image in the image set is divided into a series of patches, and the patches are mapped to a high-dimensional feature space to obtain the corresponding patch set. The formula is: p t =PatchEmbedding(X t ); Among them, p t For image set X t A series of patch sets; PatchEmbedding(·) is the patch embedding operation; the number of images in the image set corresponds to the number of patch sets; Step 2.2: Obtain the fused patches based on the attention fusion feature module and update the patch set. The specific process is as follows: Step 2.2.1: Pre-set the crossover rate and randomly select two different patch sets with the same identity based on the crossover rate. Perform attention fusion feature operation, p a The i-th and I-th patches in ; p b The j-th and J-th patches in , multiply these two sets of patches and normalize them to construct the attention matrix, the formula is: W=softmax(p a *p b ); Where W is the attention matrix; softmax(·) is the softmax function; Step 2.2.2, from patch set p a Randomly select a patch from Using the attention matrix, find the patch set p b Zhongyu The most similar patch Determine the current The index j and the adjacent indexes around j are valid indexes; according to the valid index from p b Extract the corresponding eigenvalues ​​and extract the attention scores corresponding to the indices from the attention matrix W. Then, renormalize the extracted attention scores and construct a new attention matrix, specifically: W′=softmax(w j ),j∈vaild indices ; Among them, W′ is the new attention matrix; w j is the attention score at index j in the attention matrix W; vaild indices is a valid index; Step 2.2.

3. According to the new attention matrix and the corresponding eigenvalue, the fused patch is obtained. The formula is: Among them, p r is the fused patch; w j ′ is the attention score at index j in the new attention matrix W′, and the attention matrix corresponds to the index number of the new attention matrix; Step 2.2.4, finally, p a in Replaced with the fused patch p r , get the updated patch set Step 2.3: The Gaussian mutation feature module uses the Gaussian mutation strategy to mutate the patch and update the patch set. The specific process is as follows: First, define a series of patches with the same identity as the input as the original patch set p s , calculate p s Statistics, including mean and variance; according to the calculated mean μ and variance σ 2 , defines a Gaussian distribution N(μ,σ 2 ); from p s A series of patches of one of the identities are selected to form the original mutation patch set s is a specific identity, m is a sample in a specific identity, For p m Middle Patches, For p m Middle patches; sample from the Gaussian distribution and traverse the original mutation patch set p m For each patch in , the mutation rate is pre-set, and the individuals are replaced by points sampled from the Gaussian distribution according to the mutation rate; the calculation formulas for the mean and variance are: in, For the original patch set p s The hth patch in the set; H is the original patch set p s The number of patches in New feature sample p extracted from the Gaussian model new It conforms to the Gaussian distribution, specifically: p new ~N(μ,σ 2 ); Finally, the samples in the original mutation patch set Replace with the new feature sample p obtained by sampling new , get the new mutation patch set Step 2.4: transform the original input image set t into a series of patch sets p t Input feature enhancement algorithm part, the attention fusion feature module and Gaussian variation feature module in the feature enhancement algorithm part are executed in parallel according to the process of step 2.2 and step 2.3, and finally a new series of patch sets p are optimized. t′ , and p t′ Input into three weight-sharing encoders, one encoder processes the image of one modality to obtain the predicted value features of each modality; the formula is: Among them, F t For image set X t The predicted value characteristics of is a weight-sharing encoder; Step 3: Construct a loss function and train and optimize the model based on the training data set and loss function; Step 4: Perform cross-modal person re-identification based on the trained model.

2. The cross-modal person re-identification method based on feature enhancement according to claim 1, characterized in that: The specific process of step 1 is: Step 1.1: Obtain the public datasets SYSU-MM01 and RegDB as training datasets. Collect all pedestrian images from various surveillance cameras as the test dataset. The test dataset consists of two parts: a query set and a gallery set. The query set is the set of pedestrian images to be queried, and the gallery set is the set of candidate pedestrian images that match the query set. The data in the dataset includes both visible light images and infrared images. Step 1.2: Generate new enhanced images using the channel enhancement method for the visible light images in the training dataset. The training dataset now contains images of three modalities: visible light images, infrared images, and enhanced images, with one image corresponding to each modality. Step 1.3: Perform image preprocessing on the three modal images in the training dataset. Perform random horizontal flipping and regularization on the visible light images, Gaussian blur and brightness adjustment on the infrared images, and random erasure on the enhanced images. Step 1.4: Resize the images of the three modalities to 256*128 pixels.

3. The cross-modal person re-identification method based on feature enhancement according to claim 2, characterized in that: The specific process of step 3 is as follows: Step 3.1: Input the predicted value features of each image set into the classification layer to obtain the predicted probability of the image for each identity, and calculate the identity loss L based on the predicted probability. id : S t =Classification(F t ); Among them, S t For image set X t The predicted probability of Classification(·) is the classification layer; Y t Represents the image set X t The true label of represents the total number of identity samples; Represents the image set X t For the first The predicted probability of each identity; Step 3.2: Calculate the triplet loss L of the image set tri : Among them, F is the predicted value feature; Represents a randomly selected sample image; o represents Positive sample images with the same sample identity; o′ represents Negative sample images with different sample identities; margin is the parameter boundary; Step 3.3: Calculate the discriminative center loss L of the image set dcl : in, for The center vector of For the identities; K represents the number of predictive value features of the same modality in the same identity; For visible light images The z-th predicted value feature; In infrared images The k-th predicted value feature; To remove The predicted value characteristics of all identities except The average distance of the center vector of z′ Indicates the The predicted value features of the z′th sample with different identities; F k′ For the The predicted value features of the k′th sample with the same identity; Step 3.4: Construct the least squares error loss function L msel , randomly select a prediction value feature of an infrared image or a visible light image, denoted as Prediction value features of infrared images or Prediction value features of visible light images First calculate The average distance to other samples of the same identity in intramodality and cross-modality is calculated as: Among them, D intra is the average distance under the internal mode; D cross is the average distance under cross-modality; D(·) is the Euclidean distance; For the The predicted value features of infrared images; For the The predicted value features of visible light images; Calculate in the same way as above the average distance to other samples of the same identity in both intramodality and crossmodality; Calculate the difference L between the mean distance under intramodality and the mean distance under cross-modality msel : in, For the same identity The average distance under the internal mode corresponding to the predicted value features; For the same identity The average cross-modal distance corresponding to the predicted value features; Step 3.

5. Finally, the overall loss function L of the training process is: L=λL msel +λL dcl +L id +L tri ; Among them, λ is a hyperparameter.

4. The cross-modal person re-identification method based on feature enhancement according to claim 3 is characterized in that: The specific process of step 4 is as follows: Step 4.1: Use the query set and gallery set of the test dataset as input to the cross-modal person re-identification model trained in Step 3. Concatenate the image features of the three modalities output by the model in the channel dimension to obtain the final pedestrian prediction value features. Step 4.2: Calculate the similarity between the pedestrian images in the query set and the pedestrian images in the gallery set. The similarity calculation formula is: in, Pedestrian images in the query set Pedestrian images with gallery set similarity; Pedestrian images representing the query set The predicted value characteristics of Pedestrian images representing the gallery set The predicted value characteristics of |·| represents the module length; Step 4.3: Sort all similarity values ​​in descending order, and output the images corresponding to the pedestrian prediction value features with the highest similarity values ​​as the re-identification results.

Images