Unsupervised person re-identification method and system based on modal invariance modeling

Abstract

The present invention discloses an unsupervised pedestrian re-identification method and system based on modal invariance modeling, which belongs to the technical field of pedestrian re-identification. The image style is processed by using an inter-modal style transfer model; the robustness of the online encoder to modal differences is improved by using modal contrast loss; the feature similarity between the original image and the most similar k images is calculated, and the feature similarity between the corresponding generated image and the corresponding k generated images is calculated; the fuzziness is calculated based on the two sets of feature similarities and a reliable image is selected; the online encoder is optimized and the average encoder is updated by using a cluster contrast loss function, a difficult sample contrast loss function and a modal contrast loss function; finally, feature extraction is performed on the test data to determine the pedestrian image with a specified identity in a cross-modal scenario. The present invention solves the technical problem that the prior art is usually affected by difficult samples and noise labels, and optimizes the comprehensive performance in the cross-modal pedestrian re-identification scenario.

Description

Technical Field

[0001] The present invention relates to the technical field of pedestrian re-identification, and in particular to an unsupervised pedestrian re-identification method and system based on modal invariance modeling. Background Art

[0002] The goal of visible-infrared cross-modal person re-identification is to match the same person between visible and infrared modalities. Currently, cross-modal person re-identification methods have achieved significant performance improvements by relying on large-scale manually annotated training data. However, labeling data for cross-modal image retrieval tasks is a very time-consuming task, which limits the promotion of cross-modal person re-identification methods in reality. To address the above problems, Liang et al. proposed the first unsupervised cross-modal person re-identification method. This method completes the matching between modalities through homogeneous-heterogeneous learning and designs a cross-modal re-ranking algorithm to improve test performance. Recently, Yang et al. proposed the first fully unsupervised cross-modal person re-identification method, which also follows the idea of ​​homogeneous-heterogeneous learning and no longer relies on the labeled source domain.

[0003] Although the above research results have made innovative progress, their performance still has a lot of room for improvement due to the neglect of the impact of difficult samples and noisy labels. Studies have shown that the optimization of difficult samples (i.e., positive samples that are far away from the anchor point or negative samples that are close to the anchor point) can provide significant contributions to performance improvement. However, existing methods can only select positive samples but cannot determine difficult samples. Although single-modal unsupervised pedestrian re-identification methods can generate pseudo-labels for unlabeled data through inter-modal clustering, and then determine difficult samples based on pseudo-labels and feature distances. However, modality differences in cross-modal scenarios prevent samples from different modalities from being clustered into the same cluster, thereby reducing the number of positive sample pairs between modalities and generating a large number of noisy labels. Therefore, how to alleviate modality differences is the key to solving this problem. In addition, existing methods only design a selection algorithm to assign pseudo-labels to samples, while ignoring the noisy labels inherent in the pseudo-labels. Noisy labels will mislead the direction of optimization and ultimately hinder the performance improvement of the model. Summary of the invention

[0004] The purpose of the present invention is to provide an unsupervised pedestrian re-identification method and system based on modal invariance modeling, which solves the technical problem that existing unsupervised cross-modal pedestrian re-identification methods are usually affected by difficult samples and noisy labels, and optimizes the comprehensive performance in cross-modal pedestrian re-identification scenarios.

[0005] To achieve the above object, the present invention provides an unsupervised person re-identification method based on modal invariance modeling, comprising the following steps:

[0006] S1, collect pedestrian image samples in visible light and infrared modalities and build a dataset;

[0007] S2. Use the inter-modal style transfer model to process the image style to obtain a generated image set, and then use the triplet perceptual loss and structural consistency constraint to make the generated image have the target style and maintain the original identity;

[0008] S3, using an online encoder to extract features from the generated image and the original image, and using modality contrast loss to improve the robustness of the online encoder to modality differences;

[0009] S4, using an average encoder to perform feature extraction, respectively obtaining average features of the original image and the generated image, searching for the most similar k images in the original image set based on the average features of the original image, calculating the feature similarity between the original image and the k images, and calculating the feature similarity between the corresponding generated image and the corresponding k generated images;

[0010] S5, calculating the cosine similarity of the above two sets of feature similarities, calculating the fuzziness of the samples based on the cosine similarity, and selecting reliable samples from the samples based on the sample fuzziness;

[0011] S6, optimizing the online encoder using a cluster contrast loss function, a difficult sample contrast loss function, and a modality contrast loss function, and then updating the average encoder using the optimized online encoder;

[0012] S7. Use the updated average encoder to extract features from the test dataset and determine pedestrian images with specified identities in cross-modal scenarios based on the similarity between the features.

[0013] Preferably, step S2 specifically includes:

[0014] S21, inputting the visible light image set or the infrared image set into the generator to obtain a generated image set, wherein the generated images in the generated image set have the style of the infrared modality and the identity of the visible light modality or the style of the visible light modality and the identity of the infrared modality;

[0015] S22, inputting the generated image set into a generator to generate a reconstructed image set, wherein the reconstructed image set has the style and identity of the visible light modality or the style and identity of the infrared modality;

[0016] S23. Based on the triplet perceptual loss, the generated image has the target style, and based on the structural consistency constraint, the generated image maintains the original identity.

[0017] Preferably, step S3 specifically includes:

[0018] S31, extracting features of the generated image and the original image through an online encoder to obtain online features of the generated image and online features of the original image respectively;

[0019] S32, using a multi-layer perceptron to map the online features of the generated image to obtain the mapped online features of the generated image;

[0020] S33, calculating the negative cosine similarity between the online features of the mapped generated image and the online features of the original image.

[0021] Preferably, in step S5, the cosine similarity of the two sets of feature similarities is:

[0022]

[0023] in, is the feature similarity between the original visible light image and the selected k images, is the feature similarity between the corresponding generated image and the corresponding k generated images;

[0024] Calculate fuzziness based on cosine similarity:

[0025]

[0026] Preferably, in step S6:

[0027] 1) Cluster contrast loss function: Calculate the cluster centroid based on the refined results and identity pseudo-labels. For any image x in the pth cluster, p , set the centroid of the pth cluster to the positive cluster centroid, and the centroids of other clusters to the negative cluster centroids; increase x through the cluster contrast loss function p Similarity to the positive cluster centroid, reducing similarity to the negative cluster centroid:

[0028]

[0029] Among them, f p The image x extracted by the online encoder p The online features of v p For x p The centroid of the positive cluster, v i is the centroid of any cluster, n r is the number of clusters in the current stage, τ r is the cluster contrast loss function L rep The temperature hyperparameter of

[0030] 2) Difficult sample contrast loss function: During training, for each image x in a batch d , the positive sample farthest from it is a difficult positive sample, and the first w negative samples closest to it are difficult negative samples. By introducing the difficult sample comparison loss function, x is improved. d Similarity with difficult positive samples, reduce x d Similarity with hard negative samples:

[0031]

[0032] Among them, f d The image x extracted by the online encoder d Online features, The image x extracted by the average encoder d The characteristics of the difficult positive samples, The image x extracted by the average encoder d The characteristics of the difficult negative samples, τ h L is the difficult sample contrast loss function hsc The temperature hyperparameter of

[0033] 3) Modality contrast loss function: Calculate the modality centroid based on the refined results, identity pseudo-labels and modality labels. For any image x with modality label q in the p-th cluster, pq , set the combination of modal centroids whose modal labels are not q in the p-th cluster to the positive cross-modal centroid v pl , and the other clusters with x p The combination of the first u modal centroids closest to each other is called the negative modal centroid set U n , the modality contrast loss function is introduced to optimize the online encoder:

[0034]

[0035] Among them, f pq The image x extracted by the online encoder pq The online features of v ij For U n ∪v pl The modal centroid in r is the modality contrast loss function L mod The temperature hyperparameter of

[0036] 4) The overall loss function is defined as:

[0037] L CMCL =L rep +λ hsc L hsc +λ mod L mod

[0038] Among them, λ hsc and λ mod L hsc and L mod The weight factor of .

[0039] The present invention also provides an unsupervised person re-identification system based on modal invariance modeling, comprising:

[0040] Acquisition module: used to collect pedestrian image samples and build training data sets;

[0041] Image generation module: used to process image style and obtain generated images;

[0042] Invariance learning module: used to improve the robustness of online encoders to modality differences;

[0043] Similarity calculation module: used to calculate the feature similarity between the original image and the selected k images, and the feature similarity between the corresponding generated image and the corresponding k generated images;

[0044] Fuzziness calculation module: used to calculate the fuzziness of samples and select reliable samples for subsequent training;

[0045] Optimization module: used to optimize the online encoder and update the average encoder;

[0046] Recognition module: used to identify pedestrian images with specified identities in cross-modal scenarios.

[0047] Therefore, the present invention adopts the above-mentioned unsupervised pedestrian re-identification method and system based on modal invariance modeling to achieve the following beneficial effects:

[0048] 1. The robustness of the online encoder to modality differences is enhanced by using a modality contrast loss of the negative cosine similarity between the generated image and the original image.

[0049] 2. Through sample fuzziness, reliable images can be selected to reduce the impact of noisy labels on the average encoder performance, which facilitates subsequent training.

[0050] 3. The online encoder can be optimized through the cluster contrast loss function, the difficult sample contrast loss function and the modality contrast loss function to further improve the performance of the average encoder.

[0051] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flow chart of the method of the present invention;

[0053] Figure 2 It is a schematic diagram of the overall structure of the present invention;

[0054] Figure 3 is a diagram of the inter-modal style transfer model of the present invention;

[0055] Figure 4 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0056] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0057] Unless otherwise defined, the technical terms or scientific terms used in the present invention shall have the usual meanings understood by persons with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. The words "include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, but do not exclude other elements or objects. The terms "set", "install" and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be the internal connection of two elements. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0058] Example 1

[0059] 1. If Figure 1-4 As shown, the present invention provides an unsupervised pedestrian re-identification method based on modal invariance modeling, comprising the following steps:

[0060] S1. Collect pedestrian image samples in visible light and infrared modalities and build a dataset.

[0061] S2. Use the inter-modal style transfer model to process the image style and obtain a generated image set. Then use the triplet perceptual loss and structural consistency constraint to make the generated image have the target style and maintain the original identity. Specifically include:

[0062] S21, the visible light image set or infrared image set Input to the generator G s (G ir ) to obtain the generated image set The generated images in the generated image set have the style of the infrared modality and the identity of the visible light modality or the style of the visible light modality and the identity of the infrared modality;

[0063] S22, then generate the image set Input to the generator G s (G ir ) to generate a reconstructed image set These reconstructed image sets have either the style and identity of the visible light modality or the style and identity of the infrared modality;

[0064] S23, 1) Make the generated image have the target style based on triplet perceptual loss:

[0065] L trip =L tripA +L tripB

[0066]

[0067]

[0068] in, are the original visible light image, the reconstructed visible light image, the generated infrared image, the reconstructed infrared image, the original infrared image and the generated visible light image, φ is the VGG-19 network, φ j (·) is the jth activation layer of the network φ, and ω is a hyperparameter.

[0069] 2) Encourage the generated image to maintain the original identity based on structural consistency constraints:

[0070] L scc =L sccA +L sccB

[0071]

[0072]

[0073] in, is an estimator of the relative squared loss mutual information, They are A random variable with a pixel in .

[0074] S3. Use an online encoder to extract features from the above generated image and the original image, and use modality contrast loss to improve the robustness of the online encoder to modality differences. Specifically including:

[0075] S31, extracting features of the generated image and the original image through an online encoder to obtain online features of the generated image and online features of the original image respectively;

[0076] S32, using a multi-layer perceptron to map the online features of the generated image to obtain the mapped online features of the generated image;

[0077] S33, calculating the negative cosine similarity between the online features of the mapped generated image and the online features of the original image:

[0078]

[0079] Among them, q(·) is the processing operation of the online encoder, and p(·) is the processing operation of the multi-layer perceptron. After that, the modality contrast loss is used to improve the robustness of the online encoder to modality differences:

[0080]

[0081] Among them, stopgrad means stopping the gradient operation.

[0082] S4. Use the average encoder to extract features from the original image and the generated image respectively, obtain the average features of the original image and the average features of the generated image, find the most similar k images in the original image set based on the average features of the original image, and calculate the feature similarity between the original image and the k images:

[0083]

[0084] Where M(·) is the operation of the average encoder, and Both The image among the k most similar original images, τ s is the temperature hyperparameter corresponding to the feature similarity; find the corresponding k generated images for the selected k original images, and calculate the feature similarity between the corresponding generated image and the corresponding k generated images:

[0085]

[0086] It should be noted that and Both The image among the k most similar generated images, the above is a visible light image And the generated infrared image Take the feature similarity calculation process as an example, infrared image and the resulting visible light image The same calculation steps are also followed.

[0087] S5. Calculate the cosine similarity of the above two sets of feature similarities:

[0088]

[0089] in, is the feature similarity between the original visible light image and the selected k images, is the feature similarity between the corresponding generated image and the corresponding k generated images;

[0090] Calculate the fuzziness of samples based on cosine similarity:

[0091]

[0092] Afterwards, based on fuzziness Refine the training data and select reliable images for subsequent training.

[0093] S6. Use the cluster contrast loss function, the difficult sample contrast loss function, and the modality contrast loss function to optimize the online encoder, and then use the optimized online encoder to update the average encoder.

[0094] 1) Cluster contrast loss function: Calculate the cluster centroid based on the refined results and identity pseudo-labels. For example, the centroid v of the a-th cluster a is defined as:

[0095]

[0096] in, is the feature of the image in the ath cluster extracted by the average encoder, n a The number of images in this cluster.

[0097] For any image x in the pth cluster p , set the centroid of the pth cluster to the positive cluster centroid, and the centroids of other clusters to the negative cluster centroids; increase x through the cluster contrast loss function p Similarity to the positive cluster centroid, reducing similarity to the negative cluster centroid:

[0098]

[0099] Among them, f p The image x extracted by the online encoder p The online features of v p For x p The centroid of the positive cluster, v i is the centroid of any cluster, n r is the number of clusters in the current stage, τ r is the cluster contrast loss function L rep The temperature hyperparameter of

[0100] 2) Difficult sample contrast loss function: During training, for each image x in a batch d , the positive sample with the farthest distance is the difficult positive sample, and the first w negative samples with the closest distance are the difficult negative samples. Then, by introducing the difficult sample contrast loss function, x is improved. d Similarity with difficult positive samples, reduce x d Similarity with hard negative samples:

[0101]

[0102] Among them, f dThe image x extracted by the online encoder d Online features, The image x extracted by the average encoder d The characteristics of the difficult positive samples, The image x extracted by the average encoder d The characteristics of the difficult negative samples, τ h L is the difficult sample contrast loss function hsc The temperature hyperparameter of

[0103] 3) Modality contrast loss function: The modality centroid is calculated based on the refined results, identity pseudo-labels and modality labels. For example, the modality centroid of the modality label b in the a-th cluster is defined as:

[0104]

[0105] in, is the feature with modality label b in cluster a extracted by the average encoder, n ab is the number of images with modality label b in the cluster.

[0106] For any image x with modality label q in the pth cluster pq , set the combination of modal centroids whose modal labels are not q in the p-th cluster to the positive cross-modal centroid v pl , and the other clusters with x p The combination of the first u modal centroids closest to each other is called the negative modal centroid set U n , the modality contrast loss function is introduced to optimize the online encoder:

[0107]

[0108] Among them, f pq The image x extracted by the online encoder pq The online features of v ij For U n ∪v pl The modal centroid in r is the modality contrast loss function L mod The temperature hyperparameter of

[0109] 4) The overall loss function is defined as:

[0110] L CMCL =L rep +λ hsc L hsc +λ mod L mod

[0111] Among them, λ hsc and λ mod Lhsc and L mod The weight factor of .

[0112] S7. Use the updated average encoder to extract features from the test dataset and determine pedestrian images with specified identities in cross-modal scenarios based on the similarity between the features.

[0113] 2. If Figure 4 As shown, the present invention provides an unsupervised pedestrian re-identification system based on modal invariance modeling, comprising:

[0114] Acquisition module: used to collect pedestrian image samples in visible light and infrared modes and build training data sets.

[0115] Image generation module: It is used to process the image style using the inter-modal style transfer model to obtain a set of generated images. The triplet perceptual loss and structural consistency constraint are used to ensure that the generated images have the target style and maintain the original identity.

[0116] Invariance learning module: used to use the online encoder to extract features from the generated image and the original image, and use the modality contrast loss to improve the robustness of the online encoder to modality differences.

[0117] Similarity calculation module: used to find the most similar k images in the original image set based on the average features of the original image, calculate the feature similarity between the original image and the selected k images, and calculate the feature similarity between the corresponding generated image and the corresponding k generated images.

[0118] Fuzziness calculation module: used to calculate the cosine similarity of two sets of feature similarities, calculate the fuzziness of samples based on cosine similarity, and select reliable samples for subsequent training based on sample fuzziness.

[0119] Optimization module: used to optimize the online encoder using cluster contrast loss function, difficult sample contrast loss function and modality contrast loss function, and use the optimized online encoder to update the average encoder.

[0120] Recognition module: used to extract features from the test dataset using the updated average encoder to determine pedestrian images with specified identities in cross-modal scenarios.

[0121] Example 2

[0122] This embodiment uses mean average precision (mAP) and cumulative matching characteristic (CMC) Rank-1, Rank-5 and Rank-10 to evaluate the performance of the MIMR of the present invention and the existing fully unsupervised methods. Among them, the selected existing fully unsupervised methods are SpCL, MMT, CAP, ICE, H2H, ADCA and DFC.

[0123] The experimental dataset used is SYSU-MM01. This dataset contains 287,628 visible light images and 15,792 infrared images of 491 identities; the training set contains 395 identities and the test set contains 96 identities; in the test phase, infrared images are used for the probe set and visible light images are used for the gallery set; this dataset includes full search and indoor search test modes.

[0124] Table 1 below shows the comparison results of the present invention with the existing unsupervised methods on the SYSU-MM01 dataset:

[0125]

[0126] It can be seen that compared with the above optimal method ADCA, the present invention has achieved significant performance advantages in both test modes. Therefore, the comprehensive performance of the present invention in the cross-modal pedestrian re-identification scenario is better than the existing unsupervised methods.

[0127] Therefore, the present invention adopts the above-mentioned unsupervised pedestrian re-identification method and system based on modal invariance modeling, which solves the technical problem that the existing unsupervised cross-modal pedestrian re-identification methods are usually affected by difficult samples and noisy labels, and optimizes the comprehensive performance in cross-modal pedestrian re-identification scenarios.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. An unsupervised person re-identification method based on modal invariance modeling, characterized in that: The following steps are involved: S1, collect pedestrian image samples in visible light and infrared modalities and build a dataset; S2. Use the inter-modal style transfer model to process the image style to obtain a generated image set, and then use the triplet perceptual loss and structural consistency constraint to make the generated image have the target style and maintain the original identity; S21, inputting the visible light image set or the infrared image set into the generator to obtain a generated image set, wherein the generated images in the generated image set have the style of the infrared modality and the identity of the visible light modality or the style of the visible light modality and the identity of the infrared modality; S22, inputting the generated image set into a generator to generate a reconstructed image set, wherein the reconstructed image set has the style and identity of the visible light modality or the style and identity of the infrared modality; S23, based on triplet perceptual loss, the generated image has the target style, and based on structural consistency constraint, the generated image maintains the original identity; S3, using the online encoder to extract features from the generated image and the original image, and using the modality contrast loss to improve the robustness of the online encoder to modality differences; S4, using an average encoder to perform feature extraction, respectively obtain average features of the original image and the generated image, find the most similar k images in the original image set based on the average features of the original image, find corresponding k generated images for the selected k original images, calculate the feature similarity between the original image and the k images, and calculate the feature similarity between the corresponding generated image and the corresponding k generated images; S5, calculating the cosine similarity of the above two sets of feature similarities, calculating the fuzziness of the samples based on the cosine similarity, and selecting reliable samples from the samples based on the sample fuzziness; S6, optimizing the online encoder using a cluster contrast loss function, a difficult sample contrast loss function, and a modality contrast loss function, and then updating the average encoder using the optimized online encoder; 1) Cluster contrast loss function: Calculate the cluster centroid based on the refined results and identity pseudo-labels, for any image in the pth cluster , set the centroid of the pth cluster to the positive cluster centroid, and the centroids of other clusters to the negative cluster centroids; increase the cluster contrast loss function Similarity to the positive cluster centroid, reducing similarity to the negative cluster centroid: in, Image extracted for online encoder Online features, for The centroid of the positive cluster, is any cluster centroid, is the number of clusters in the current stage, is the cluster contrast loss function The temperature hyperparameter of 2) Difficult sample contrast loss function: During training, for each image in a batch , the positive sample with the farthest distance is the difficult positive sample, and the positive sample with the closest distance is w Negative samples are difficult negative samples. By introducing difficult samples, the contrast loss function is improved. Similarity with difficult positive samples, reducing Similarity with hard negative samples: in, Image extracted for online encoder Online features, Image extracted for the average encoder The characteristics of the difficult positive samples, Image extracted for the average encoder The characteristics of difficult negative samples, is the contrast loss function for difficult samples The temperature hyperparameter of 3) Modality contrast loss function: Calculate the modality centroid based on the refined results, identity pseudo-labels and modality labels, for any image with modality label q in the pth cluster , set the combination of modal centroids whose modal labels are not q in the p-th cluster as the positive cross-modal centroid , and the other clusters The closest front u The combination of modal centroids is called the negative modal centroid set. , the modality contrast loss function is introduced to optimize the online encoder: in, Image extracted for online encoder Online features, for The modal centroid in is the modality contrast loss function The temperature hyperparameter of 4) The overall loss function is defined as: in, and They are and The weight factor of S7. Use the updated average encoder to extract features from the test dataset and determine pedestrian images with specified identities in cross-modal scenarios based on the similarity between the features.

2. The unsupervised person re-identification method based on modal invariance modeling according to claim 1, characterized in that: The feature extraction process in step S3 specifically includes: S31, extracting features of the generated image and the original image through an online encoder to obtain online features of the generated image and online features of the original image respectively; S32, using a multi-layer perceptron to map the online features of the generated image to obtain the mapped online features of the generated image; S33, calculating the negative cosine similarity between the online features of the mapped generated image and the online features of the original image.

3. The unsupervised person re-identification method based on modal invariance modeling according to claim 2 is characterized in that: In step S5, the cosine similarity of the two sets of feature similarities is: in, is the feature similarity between the original visible light image and the selected k images, is the feature similarity between the corresponding generated image and the corresponding k generated images; Calculate fuzziness based on cosine similarity: 。 4. Unsupervised person re-identification system based on modal invariance modeling, characterized by: The unsupervised person re-identification method based on modal invariance modeling according to any one of claims 1 to 3 comprises: Acquisition module: used to collect pedestrian image samples and build training data sets; Image generation module: used to process image style and obtain generated images; Invariance learning module: used to improve the robustness of online encoders to modality differences; Similarity calculation module: used to calculate the feature similarity between the original image and the selected k images, and the feature similarity between the corresponding generated image and the corresponding k generated images; Fuzziness calculation module: used to calculate the fuzziness of samples and select reliable samples for subsequent training; Optimization module: used to optimize the online encoder and update the average encoder; Recognition module: used to identify pedestrian images with specified identities in cross-modal scenarios.

Images