Cross-modal pedestrian re-identification method based on data enhancement and graph matching

Abstract

The invention provides a cross-modal pedestrian re-identification method based on data enhancement and graph matching. A visible light camera and an infrared camera are used for collecting pedestrian images as visible light domain training images and infrared domain training images respectively; inputting the pedestrian images of the visible light domain and the infrared domain into a cyclic generative adversarial network with a U-Net structure for data enhancement; the backbone networks ResNet50 and AGW are used to extract features of the visible light image and the infrared image, and the features are clustered by using DBSCAN; and finding out a visible light-infrared corresponding relation through graph matching to obtain a matching result. According to the method, data enhancement is carried out on the images, feature extraction and classification are respectively carried out, and finally, image matching is carried out to obtain different-modal pedestrian image matching results. According to the method, more reliable cross mode correspondence can be provided while the difference between modes is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of machine vision, and specifically is a cross-modal pedestrian re-identification method based on data enhancement and graph matching. Background Art

[0002] The goal of visible-infrared pedestrian re-identification is to identify the same pedestrian in a set of images of another modality given an image of one modality. Due to its importance in nighttime intelligent surveillance and public safety, it has received widespread attention in recent years. Existing cross-modal pedestrian re-identification technologies mainly rely on high-quality labeled training sets, which are difficult to obtain. At the same time, there are large modal differences between visible light images and infrared images, and the amount of infrared image data is far less than that of visible light images, which is not conducive to model training. Therefore, their application in real scenes is limited. To address the above problems, existing methods that do not rely on high-quality annotated data mainly use unsupervised solutions, but in the visible-infrared heterogeneous space, since the cross-modal differences are much larger than the inter-class differences within each modality, that is, there is a large modal gap, it is impossible to maintain the consistency of features and semantics. The main existing method for modal differences is based on shared modal feature learning, that is, using a shared network to extract modality-independent features of visible light images and infrared images for cross-modal pedestrian matching. However, specific modal information is of great value for pedestrian re-identification. Using this method will limit the representation ability of pedestrian features and affect the performance of visible light-infrared pedestrian re-identification. Summary of the invention

[0003] Purpose of the invention: In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a cross-modal pedestrian re-identification method based on data enhancement and graph matching to solve the problem of low accuracy of cross-modal pedestrian re-identification.

[0004] Technical solution: The present invention discloses a cross-modal pedestrian re-identification method based on data enhancement and graph matching, comprising the following steps:

[0005] Step 1: Use a visible light camera to collect pedestrian images as visible light domain training images, and use an infrared camera to collect pedestrian images as infrared domain training images;

[0006] Step 2: Input the pedestrian images in the visible light domain and infrared domain into the recurrent generative adversarial network with U-Net structure for data enhancement, and generate pseudo visible light images and pseudo infrared images; combine the visible light images and pseudo visible light images into a new visible light image dataset, and combine the infrared images and pseudo infrared images into a new infrared image dataset;

[0007] Step 3: Use the backbone network ResNet50 and AGW to extract the features of visible light images and infrared images respectively, and use DBSCAN to cluster the extracted features respectively;

[0008] Step 4: Find the correspondence between visible light and infrared through graph matching to obtain the cross-modal pedestrian re-identification matching result.

[0009] Furthermore, the specific method of step 1 is:

[0010] Step 1.1: Use a visible light camera to obtain a visible light image dataset; perform data cleaning on the image, including: removing duplicates, Gaussian filtering, and image scaling; use the Labelme annotation tool to annotate pedestrians;

[0011] Step 1.2: Use an infrared camera to obtain an infrared image dataset; perform data cleaning on the image, including: removing duplicates, Gaussian filtering, and image scaling; use the Labelme annotation tool to annotate pedestrians.

[0012] Furthermore, the specific method of step 2 is:

[0013] Step 2.1: Input the image into a recurrent generative adversarial network with a U-Net structure. The recurrent generative adversarial network with a U-Net structure consists of a generator network and a discriminator network. The generator network uses a U-Net structure as a generator. U-Net consists of an encoder and a decoder. The encoder is responsible for extracting features layer by layer, and the decoder is responsible for restoring the image layer by layer. A skip connection is added between the encoder and the decoder to directly pass the features of each layer of the encoder to the corresponding layer of the decoder, retaining more original image details. The discriminator uses a multi-scale discriminator structure.

[0014] Step 2.2: Measure the difference between the generated image and the real image to ensure that the generated image is similar to the target modality image. The adversarial loss function is:

[0015]

[0016]

[0017]

[0018] Among them, X G represents a visible light image sample, X I represents infrared image samples; G G2I represents the generator that converts visible light images into infrared images, G I2G represents the generator that converts infrared images into visible light images; D I Denotes the discriminator used to distinguish the generated infrared image from the real infrared image, D G represents the discriminator used to distinguish the generated visible light image from the real visible light image; p data (XG ) represents the real data distribution of visible light images, p data (X I ) represents the real data distribution of infrared images; Indicates expected value; represents the adversarial loss function between the generated infrared image and the infrared image, represents the adversarial loss function between the generated visible light image and the visible light image, express and sum;

[0019] Step 2.3: Design a cycle consistency loss to ensure that the details of the image are preserved after conversion between the two modalities; the identity loss is used to ensure that the converted image still retains the identity information of the original image; the reconstruction loss ensures that the generated image is consistent with the input image after modality conversion; the specific loss function is as follows:

[0020]

[0021]

[0022]

[0023]

[0024]

[0025]

[0026]

[0027] Among them, ||.||1 represents the L1 norm, ||.||2 represents the L2 norm; G G2I (X G ) represents the conversion from visible light image to infrared image, G I2G (X I ) represents the conversion from infrared image to visible light image; G I2G (G G2I (X G )) means converting the generated infrared image back to a visible light image, G G2I (G I2G (X I )) means converting the generated visible light image back into an infrared image; and denote the cycle consistency loss for generating visible light images and infrared images, respectively. express and sum; and Represent the prediction scores of visible light images and infrared images respectively, y represents the true identity label, and denote the identity loss functions of visible light images and infrared images respectively, express and The sum of identity losses; and Represent the reconstruction losses for generating visible light images and infrared images, respectively. express and sum.

[0028] Step 2.4: Combine the adversarial loss, cycle consistency loss, identity loss, and reconstruction loss to form a comprehensive loss function as follows:

[0029]

[0030] Among them, λ1 and λ2 are weighting coefficients.

[0031] Furthermore, the specific method of step 3 is:

[0032] Step 3.1: Combine the pseudo visible light image generated in step 2 with the visible light image with the identity label removed to form a new visible light dataset, and combine the pseudo infrared image generated in step 2 with the infrared image with the identity label removed to form a new visible light dataset; input the visible light image dataset into the ResNet50 backbone network for feature extraction, and extract the visible light feature set; input the infrared image dataset into the AGW backbone network for feature extraction, and extract the infrared feature set;

[0033] Step 3.2: Use DBSCAN to cluster the visible light image features and infrared image features respectively. After clustering, the visible light image features are labeled with identity tags; after clustering, the infrared image features are labeled with identity tags;

[0034] Step 3.3: Use the following contrastive loss function to train the visible light clustering and infrared clustering respectively:

[0035]

[0036] in, represents the single membrane state loss function; F i represents the eigenvector corresponding to sample i, F j represents the feature vector corresponding to sample j; D(F i ,F j) and represents the Euclidean distance between two features; a is a binary label, indicating whether the sample pair is from the same category, a value of 1 indicates the same category, and a value of 0 indicates different categories; m is a threshold or boundary, when two samples are not similar, their distance must be at least greater than m; after completing the training, the single-modal pedestrian re-identification task is achieved.

[0037] Step 3.4: In the clustering results, find the feature of the sample point closest to the cluster center in each cluster and use this feature to represent the cluster; further, the visible light feature set can be obtained as And the infrared feature set is Prepare for the subsequent cross-membrane pedestrian re-identification task.

[0038] Furthermore, the specific method of step 4 is:

[0039] Step 4.1: Construct two graphs G vis =(V vis ,E vis ) and G ir =(V ir ,E ir ), where the node set V vis and V ir Represent the visible light and infrared image features extracted in step 3.4 respectively, and the edge set E vis and E ir Represents the similarity between features;

[0040] Step 4.2: Calculate the similarity matrix S between the visible light feature and the infrared feature, where the element in the i-th row and j-th column of the matrix is ​​represented as Use cosine similarity; based on the similarity matrix S, traverse each row and column, and select feature pairs with a similarity greater than 0.75 as initial matching pairs;

[0041] Step 4.3: Based on the initial matching results, the remaining visible light and infrared image features of the node set are combined into new graphs and The graph is constructed in the same way as step 4.1; the similarity between the visible light feature and the infrared feature is recalculated to obtain a new similarity matrix S′. The element in the i-th row and j-th column of the new similarity matrix can be expressed as:

[0042]

[0043] Among them, α and β are weight parameters, struct_sim represents structural similarity, and the structural similarity index SSIM is used specifically; based on the updated similarity matrix S′, compare the number of rows and columns. Assuming that the number of rows is small, traverse each row and select the features with the highest similarity in each row as matching pairs until a matching pair is found for each row, and vice versa;

[0044] Step 4.4: After executing step 4.3 once, assume that G vis There are remaining nodes in G that have not found corresponding nodes. ir All nodes in the find corresponding nodes; use G vis The remaining nodes and the edges between them dynamically rebuild a new graph G v ' is ; The new G v ' is and G ir Reorganize into a bipartite graph;

[0045] Step 4.5: Repeat steps 4.2 and 4.3 on the reorganized bipartite graph. When executing step 4.3, there is no need to compare the number of rows and columns. Assuming that G vis There are remaining nodes in which no corresponding nodes have been found. We only need to find G v ' is The task ends when all the matching pairs of the remaining points are met; finally, G vis and G ir Each node can find a corresponding node; the matching pair generated in each step is the matching result of pedestrian images of different modalities.

[0046] Beneficial effects:

[0047] (1) When the present invention enhances data, the U-Net structure is used as a generator in the generative adversarial network part. The unique skip connection structure of U-Net allows the features of each layer of the encoder to be directly passed to the corresponding layer of the decoder, thereby retaining more original image details during the data enhancement process. At the same time, a multi-scale discriminator is used to enhance the ability to distinguish features of different scales and generate more refined images.

[0048] (2) The present invention introduces the expected value, L1 norm and L2 norm in the design of the data enhancement loss function. The use of the expected value can smooth the loss value, reduce the fluctuation caused by the data scale difference, and make the training process more stable. The L1 norm is more robust to outliers and can better maintain the sparsity and details of the image in the image generation task. The L2 norm is more effective in processing small deviations and can better reduce the overall error and ensure the global consistency of the generated image with the original image. In the cycle consistency loss, the L1 norm and the L2 norm are combined to take into account the advantages of outlier processing and overall error control, so that more details can be retained during the image enhancement process.

[0049] (3) The present invention uses a dual-branch feature extraction network in the feature extraction and unsupervised clustering of different modalities. Each branch uses a feature extraction network that is more suitable for a specific modality, which reduces the interference caused by modality differences and retains the unique feature information of each modality. After unsupervised clustering of the two modalities, a single-modality contrast loss function is designed to maximize the similarity of the same identity within a single modality and minimize the distance between different identities within a single modality, thereby achieving better single-modality clustering results and providing a better basis for the correspondence of pedestrian identities across the membrane.

[0050] (4) In the transmembrane image alignment part, the present invention adopts the modal correspondence method of graph matching to find the corresponding relationship of each node through multiple dynamic matching. When the number of features of the visible light image and the infrared image is inconsistent, in order to effectively handle this imbalance and make full use of global information, cyclic graph matching ensures that all nodes can find suitable corresponding nodes through dynamic reconstruction and multiple matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is the overall flow chart of the cross-modal person re-identification method based on data enhancement and graph matching;

[0052] Figure 2 Generate adversarial network graphs for loops combined with U-NET;

[0053] Figure 3 Schematic diagram of feature extraction and clustering for visible light images and infrared images;

[0054] Figure 4 Schematic diagram of the graph matching module. DETAILED DESCRIPTION

[0055] The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0056] The present invention discloses a cross-modal person re-identification method based on data enhancement and graph matching, which specifically comprises the following steps:

[0057] Step 1: Use a visible light camera to collect pedestrian images as visible light domain training images; use an infrared camera to collect pedestrian images as infrared domain training images.

[0058] Step 1.1: Use a visible light camera to obtain a visible light image dataset; perform data cleaning on the image, including: removing duplicates, Gaussian filtering, and image scaling; use the Labelme annotation tool to annotate pedestrian identities.

[0059] Step 1.2: Use an infrared camera to obtain an infrared image dataset; perform data cleaning on the image, including: removing duplicates, Gaussian filtering, and image scaling; use the Labelme annotation tool to annotate pedestrians.

[0060] Step 2: Input the pedestrian images in the visible light domain and infrared domain into the recurrent generative adversarial network with U-Net structure for data enhancement, and generate pseudo visible light images and pseudo infrared images. The visible light images and pseudo visible light images are combined into a new visible light image dataset, and the infrared images and pseudo infrared images are combined into a new infrared image dataset.

[0061] Step 2.1: Input the image into a recurrent generative adversarial network with a U-Net structure, which consists of a generator network and a discriminator network. The generator network uses the U-Net structure as the generator. The U-Net consists of an encoder and a decoder. The encoder is responsible for extracting features layer by layer, and the decoder is responsible for restoring the image layer by layer. A skip connection is added between the encoder and the decoder to pass the features of each layer of the encoder directly to the corresponding layer of the decoder, retaining more details of the original image. The discriminator uses a multi-scale discriminator structure.

[0062] Step 2.2: Further measure the difference between the generated image and the real image to ensure that the generated image is similar to the target modality image. The adversarial loss function is:

[0063]

[0064]

[0065]

[0066] Where X G represents a visible light image sample, X I represents infrared image samples; G G2I represents the generator that converts visible light images into infrared images, G I2G represents the generator that converts infrared images into visible light images; D I Denotes the discriminator used to distinguish the generated infrared image from the real infrared image, D G represents the discriminator used to distinguish the generated visible light image from the real visible light image; p data (X G ) represents the real data distribution of visible light images, p data (X I ) represents the real data distribution of infrared images; Indicates expected value; represents the adversarial loss function between the generated infrared image and the infrared image, represents the adversarial loss function between the generated visible light image and the visible light image, express and sum.

[0067] Step 2.3: Design a cycle consistency loss to ensure that the details of the image are preserved after conversion between the two modalities; the identity loss is used to ensure that the converted image still retains the identity information of the original image; the reconstruction loss ensures that the generated image is consistent with the input image after modality conversion; the specific loss function is as follows:

[0068]

[0069]

[0070]

[0071]

[0072]

[0073]

[0074]

[0075] Where ||.||1 represents the L1 norm, ||.||2 represents the L2 norm; G G2I (X G ) represents the conversion from visible light image to infrared image, G I2G (X I ) represents the conversion from infrared image to visible light image; G I2G (G G2I (X G )) means converting the generated infrared image back to a visible light image, G G2I (G I2G (X I )) means converting the generated visible light image back into an infrared image; and denote the cycle consistency loss for generating visible light images and infrared images, respectively. express and sum; and Represent the prediction scores of visible light images and infrared images respectively, y represents the true identity label, and denote the identity loss functions of visible light images and infrared images respectively, express and The sum of identity losses; and Represent the reconstruction losses for generating visible light images and infrared images, respectively. express and sum.

[0076] Step 2.4: Combine the adversarial loss, cycle consistency loss, identity loss, and reconstruction loss to form a comprehensive loss function as follows:

[0077]

[0078] Among them, λ1 and λ2 are weighting coefficients.

[0079] Step 3: Use the backbone network ResNet50 and AGW to extract the features of visible light images and infrared images respectively, and use DBSCAN to cluster the extracted features respectively.

[0080] Step 3.1: Combine the pseudo visible light image generated in step 2 and the visible light image with the identity label removed to form a new visible light dataset, and combine the pseudo infrared image generated in step 2 and the infrared image with the identity label removed to form a new visible light dataset; input the visible light image dataset into the ResNet50 backbone network for feature extraction to obtain a visible light feature set; input the infrared image dataset into the AGW backbone network for feature extraction to obtain a infrared feature set.

[0081] Step 3.2: Use DBSCAN to cluster the visible light image features and infrared image features respectively. After clustering, the visible light image features are labeled with identity tags; after clustering, the infrared image features are labeled with identity tags.

[0082] Step 3.3: Use the following contrast loss function to train the visible light clustering and infrared clustering respectively:

[0083]

[0084] in, represents the single membrane state loss function; F i represents the eigenvector corresponding to sample i, F j represents the feature vector corresponding to sample j; D(F i ,F j ) and represents the Euclidean distance between two features; a is a binary label, indicating whether the sample pair is from the same category, a value of 1 indicates the same category, and a value of 0 indicates different categories; m is a threshold or boundary, when two samples are not similar, their distance must be at least greater than m; after completing the training, the single-modal pedestrian re-identification task is achieved.

[0085] Step 3.4: In the clustering results, find the feature of the sample point closest to the cluster center in each cluster and use this feature to represent the cluster; further, the visible light feature set can be obtained as And the infrared feature set is Prepare for the subsequent cross-membrane pedestrian re-identification task.

[0086] Step 4: Find the correspondence between visible light and infrared through graph matching to obtain the cross-modal pedestrian re-identification matching result.

[0087] Step 4.1: Construct two graphs G vis =(V vis ,E vis ) and G ir =(V ir ,E ir ), where the node set V vis and V ir Represent the visible light and infrared image features extracted in step 3.4 respectively, and the edge set E vis and E ir Represents the similarity between features;

[0088] Step 4.2: Calculate the similarity matrix S between the visible light feature and the infrared feature, where the element in the i-th row and j-th column of the matrix is ​​represented as Cosine similarity is used; based on the similarity matrix S, each row and column is traversed, and feature pairs with a similarity greater than 0.75 are selected as initial matching pairs.

[0089] Step 4.3: Based on the initial matching results, the remaining visible light and infrared image features of the node set are combined into new graphs and The graph is constructed in the same way as step 4.1; the similarity between the visible light feature and the infrared feature is recalculated to obtain a new similarity matrix S′. The element in the i-th row and j-th column of the new similarity matrix can be expressed as:

[0090]

[0091] Among them, α and β are weight parameters, struct_sim represents structural similarity, and the structural similarity index SSIM is used specifically; based on the updated similarity matrix S′, compare the number of rows and columns. Assuming that the number of rows is small, traverse each row and select the feature with the highest similarity in each row as the matching pair until a matching pair is found for each row, and vice versa.

[0092] Step 4.4: After executing step 4.3 once, assume that G vis There are remaining nodes in G that have not found corresponding nodes. ir All nodes in the find corresponding nodes; use Gvis The remaining nodes and the edges between them dynamically rebuild a new graph G v ' is ; The new G v ' is and G ir Reorganize into a bipartite graph;

[0093] Step 4.5: Repeat steps 4.2 and 4.3 on the reorganized bipartite graph. When executing step 4.3, there is no need to compare the number of rows and columns. Assuming that G vis There are remaining nodes in which no corresponding nodes have been found. We only need to find G v ' is The task ends when all the matching pairs of the remaining points are met; finally, G vis and G ir Each node can find a corresponding node; the matching pair generated in each step is the matching result of pedestrian images of different modalities.

[0094] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent transformation or modification made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A cross-modal person re-identification method based on data enhancement and graph matching, characterized in that: The steps include: Step 1: Use a visible light camera to collect pedestrian images as visible light domain training images, and use an infrared camera to collect pedestrian images as infrared domain training images; Step 2: Input the pedestrian images in the visible light domain and infrared domain into the recurrent generative adversarial network with U-Net structure for data enhancement, and generate pseudo visible light images and pseudo infrared images; combine the visible light images and pseudo visible light images into a new visible light image dataset, and combine the infrared images and pseudo infrared images into a new infrared image dataset; Step 3: Use the backbone network ResNet50 and AGW to extract the features of visible light images and infrared images respectively, and use DBSCAN to cluster the extracted features respectively; Step 4: Find the correspondence between visible light and infrared through graph matching to obtain the cross-modal pedestrian re-identification matching result.

2. According to claim 1, a cross-modal person re-identification method based on data enhancement and graph matching is characterized in that: The specific method of step 1 is: Step 1.1: Use a visible light camera to obtain a visible light image dataset; perform data cleaning on the image, including: removing duplicates, Gaussian filtering, and image scaling; use the Labelme annotation tool to annotate pedestrians; Step 1.2: Use an infrared camera to obtain an infrared image dataset; perform data cleaning on the image, including: removing duplicates, Gaussian filtering, and image scaling; use the Labelme annotation tool to annotate pedestrians.

3. The cross-modal person re-identification method based on data enhancement and graph matching according to claim 1, characterized in that: The specific method of step 2 is: Step 2.1: Input the image into a recurrent generative adversarial network with a U-Net structure. The recurrent generative adversarial network with a U-Net structure consists of a generator network and a discriminator network. The generator network uses a U-Net structure as a generator. U-Net consists of an encoder and a decoder. The encoder is responsible for extracting features layer by layer, and the decoder is responsible for restoring the image layer by layer. A skip connection is added between the encoder and the decoder to directly pass the features of each layer of the encoder to the corresponding layer of the decoder, retaining more original image details. The discriminator uses a multi-scale discriminator structure. Step 2.2: Measure the difference between the generated image and the real image to ensure that the generated image is similar to the target modality image. The adversarial loss function is: Among them, X G represents a visible light image sample, X I represents infrared image samples; G G2I represents the generator that converts visible light images into infrared images, G I2G represents the generator that converts infrared images into visible light images; D I Denotes the discriminator used to distinguish the generated infrared image from the real infrared image, D G represents the discriminator used to distinguish the generated visible light image from the real visible light image; p data (X G ) represents the real data distribution of visible light images, p data (X I ) represents the real data distribution of infrared images; E represents the expected value; represents the adversarial loss function between the generated infrared image and the infrared image, represents the adversarial loss function between the generated visible light image and the visible light image, express and sum; Step 2.3: Design a cycle consistency loss to ensure that the details of the image are preserved after conversion between the two modalities; the identity loss is used to ensure that the converted image still retains the identity information of the original image; the reconstruction loss ensures that the generated image is consistent with the input image after modality conversion; the specific loss function is as follows: Among them, ||.||1 represents the L1 norm, ||.||2 represents the L2 norm; G G2I (X G ) represents the conversion from visible light image to infrared image, G I2G (X I ) represents the conversion from infrared image to visible light image; G I2G (G G2I (X G )) means converting the generated infrared image back to a visible light image, G G2I (G I2G (X I )) means converting the generated visible light image back into an infrared image; and denote the cycle consistency loss for generating visible light images and infrared images, respectively. express and sum; and Represent the prediction scores of visible light images and infrared images respectively, y represents the true identity label, and denote the identity loss functions of visible light images and infrared images respectively, express and The sum of identity losses; and Represent the reconstruction losses for generating visible light images and infrared images, respectively. express and sum. Step 2.4: Combine the adversarial loss, cycle consistency loss, identity loss, and reconstruction loss to form a comprehensive loss function as follows: Among them, λ1 and λ2 are weighting coefficients.

4. The cross-modal person re-identification method based on data enhancement and graph matching according to claim 1, characterized in that: The specific method of step 3 is: Step 3.1: Combine the pseudo visible light images generated in step 2 and the visible light images with identity labels removed to form a new visible light dataset T G The generated pseudo infrared images and infrared images with identity labels removed are combined to form a new visible light dataset T I ; The visible light image dataset is input into the ResNet50 backbone network for feature extraction, and the extracted visible light feature set is obtained; the infrared image dataset is input into the AGW backbone network for feature extraction, and the extracted infrared feature set is obtained; Step 3.2: Use DBSCAN to cluster the visible light image features and infrared image features respectively. After clustering, the visible light image features are labeled with identity tags; after clustering, the infrared image features are labeled with identity tags; Step 3.3: Use the following contrast loss function to train the visible light clustering and infrared clustering respectively: in, represents the single membrane state loss function; F i represents the eigenvector corresponding to sample i, F j represents the feature vector corresponding to sample j; D(F i ,F j ) represents the Euclidean distance between two features; a is a binary label, indicating whether the sample pair is from the same category, a value of 1 indicates the same category, and a value of 0 indicates different categories; m is a threshold or boundary, when two samples are not similar, their distance must be at least greater than m; after completing the training, the single-modal pedestrian re-identification task is achieved; Step 3.4: In the clustering results, find the feature of the sample point closest to the cluster center in each cluster and use this feature to represent the cluster; further obtain the visible light feature set as And the infrared feature set is Prepare for the subsequent cross-membrane pedestrian re-identification task.

5. The cross-modal person re-identification method based on data enhancement and graph matching according to claim 4, characterized in that: The specific method of step 4 is: Step 4.1: Construct two graphs G vis =(V vis ,E vis ) and G ir =(V ir ,E ir ), where the node set V vis and V ir Represent the visible light and infrared image features extracted in step 3.4 respectively, and the edge set E vis and E ir Represents the similarity between features; Step 4.2: Calculate the similarity matrix S between the visible light feature and the infrared feature, where the element in the i-th row and j-th column of the matrix is ​​represented as Use cosine similarity; based on the similarity matrix S, traverse each row and column, and select feature pairs with a similarity greater than 0.75 as initial matching pairs; Step 4.3: Based on the initial matching results, the remaining visible light and infrared image features of the node set are combined into new graphs and The graph is constructed in the same way as step 4.1; the similarity between the visible light feature and the infrared feature is recalculated to obtain a new similarity matrix S′. The element in the i-th row and j-th column of the new similarity matrix can be expressed as: Among them, α and β are weight parameters, struct_sim represents structural similarity, and the structural similarity index SSIM is used specifically; based on the similarity matrix S′, compare the number of rows and columns. Assuming that the number of rows is small, traverse each row and select the features with the highest similarity in each row as matching pairs until a matching pair is found for each row, and vice versa; Step 4.4: After executing step 4.3 once, assume that G vis There are remaining nodes in G that have not found corresponding nodes. ir All nodes in the find corresponding nodes; use G vis The remaining nodes and the edges between them dynamically rebuild a new graph G v ' is ; The new G v ' is and G ir Reorganize into a bipartite graph; Step 4.5: Repeat steps 4.2 and 4.3 on the reorganized bipartite graph. When executing step 4.3, there is no need to compare the number of rows and columns. Assuming that G vis There are remaining nodes in which no corresponding nodes have been found. We only need to find G v ' is The task ends when all the matching pairs of the remaining points are met; finally, G vis and G ir Each node can find a corresponding node; the matching pair generated in each step is the matching result of pedestrian images of different modalities.

Images