Unsupervised cross-modal pedestrian re-identification method based on image-text contrast learning

Abstract

The invention discloses an unsupervised cross-modal pedestrian re-identification method based on image-text contrast learning, and the method comprises the steps: executing clustering in each modal, calculating a cluster center in the modal based on a pseudo tag in the modal, and optimizing an encoder in each modal through the identity classification loss; performing graying processing on the images in the visible mode, and setting a text sentence embedded by a learnable text for a mixed cluster of each visible image and a grayscale image; freezing an image module and a text module of the visual language model, optimizing learnable text embedding by utilizing image-text similarity loss, and storing text features; calculating the image similarity between the modals based on the cluster center, and calculating the text similarity between the modals based on the text features to obtain positive clusters between the modals; based on the positive cluster between the modals, optimizing an encoder by using cross-modal image semantic comparison loss and cross-modal text semantic consistency loss; and finishing a cross-modal pedestrian re-identification task by using the optimized encoder.

Description

Technical fields:

[0001] This invention relates to the field of visible-infrared cross-modal pedestrian re-identification, and more particularly to an unsupervised cross-modal pedestrian re-identification method based on image-text comparison learning. Background technology:

[0002] Pedestrian re-identification aims to extract discriminative feature representations from pedestrian images to accurately match multiple images of the same person. Existing surveillance cameras capture visible images during the day or under sufficient lighting conditions and automatically switch to infrared mode to capture infrared images at night or under insufficient lighting conditions. To this end, researchers have explored various visible-infrared cross-modal pedestrian re-identification methods. These methods typically use a two-stream encoder to adapt to modal differences and strive to match visible and infrared images of the same person. While existing cross-modal pedestrian re-identification methods have achieved promising performance, most rely on large-scale manual annotation. Manual annotation typically requires significant human and material resources, significantly reducing the flexibility of these cross-modal pedestrian re-identification methods.

[0003] Therefore, some researchers have begun to turn their attention to the field of unsupervised visible-infrared cross-modal person re-identification. These methods typically no longer rely on manually labeled identity tags, but instead use clustering algorithms and inter-modal matching to guide model optimization. For example, CCLNet (CLIP-Enhanced Contrastive Learning Network) introduces a visual language model and uses textual semantics as an additional supervisory signal to guide model optimization. CCLNet learns exclusive textual features for two positive clusters between modalities, and then causes the image features within each cluster to approximate the corresponding textual features. However, this method only uses textual semantics to improve intra-modal cluster compactness and cannot effectively utilize textual semantics to promote modality invariance learning. Summary of the Invention:

[0004] This invention provides an unsupervised cross-modal person re-identification method based on image-text comparison learning, in order to solve the problem that existing unsupervised cross-modal methods are unable to effectively utilize text semantics for inter-modal matching and modality invariance learning.

[0005] This invention provides an unsupervised cross-modal person re-identification method based on image-text comparison learning, comprising the following steps:

[0006] Step S1: Perform clustering within each modality and calculate the cluster centers within the modality based on the pseudo-labels within the modality. Optimize the encoder within each modality using the identity classification loss.

[0007] Step S2: Convert the images within the visible modality to grayscale and set a text sentence containing learnable text embeddings for each cluster of mixed visible and grayscale images;

[0008] Step S3: Freeze the image module and text module of the visual language model, optimize the learnable text embedding using image-text similarity loss, and preserve text features;

[0009] Step S4: Calculate the image similarity between modalities based on the cluster centers, and calculate the text similarity between modalities based on the text features, so as to obtain positive clusters between modalities;

[0010] Step S5: Based on the positive clusters between the modalities, optimize the encoder using cross-modal image semantic contrast loss and cross-modal text semantic consistency loss;

[0011] Step S6: Use the optimized encoder to extract features from the visible and infrared images to complete the cross-modal pedestrian re-identification task.

[0012] The technical solution of the present invention achieves at least the following beneficial effects:

[0013] Text features are obtained for each cluster of visible modalities based on a visual language model, and more reliable positive clusters between modalities are obtained based on image similarity and text similarity between modalities. Cross-modal image semantic contrast loss is used to effectively mitigate the impact of modal differences, and cross-modal text semantic consistency loss is used to further promote modality invariance learning. Attached image description:

[0014] The above advantages of the present invention will be more readily understood from the following description of the embodiments in conjunction with the accompanying drawings, wherein:

[0015] Figure 1 This is a flowchart of an embodiment of the present invention;

[0016] Figure 2 This is an overall structural diagram of one embodiment of the present invention;

[0017] Figure 3 It is a statistical graph of image features and text features;

[0018] Figure 4 It is an optimization graph of cross-modal image semantic contrast loss and cross-modal text semantic consistency loss. Detailed implementation method:

[0019] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0020] Figure 1 This is a flowchart of one embodiment of the present invention. Figure 2 This is an overall structural diagram of one embodiment of the present invention.

[0021] like Figure 1 and 2 As shown, an unsupervised cross-modal person re-identification method based on image-text comparison learning includes the following steps:

[0022] In step S1, clustering is performed within each modality, and cluster centers within the modality are calculated based on pseudo-labels within the modality. The encoder is optimized within each modality using identity classification loss.

[0023] Furthermore, in one embodiment of the present invention, step S1 specifically includes:

[0024] Step S101: Use the encoder to extract features from the image and perform clustering within each modality to obtain pseudo-labels within the modality;

[0025] Step S102: Calculate the cluster center within the modality based on the pseudo-labels within the modality;

[0026] Step S103: Based on the pseudo-labels within the modality, optimize the encoder within each modality using the identity classification loss.

[0027] Specifically, firstly, an encoder is used to process the infrared image. Visible image Feature extraction is performed, and clustering is carried out within each modality using the DBSCAN algorithm to obtain pseudo-labels within the modality; the cluster centers within the visible modalities are calculated based on the pseudo-labels within the visible modalities.

[0028]

[0029] in, This represents the total number of visible images in the cluster. For the visible images in this cluster Features; Calculation of cluster centers within infrared modes based on pseudo-tags within infrared modes:

[0030]

[0031] in, This represents the total number of infrared images in the cluster. Infrared images of this cluster Features; based on in-modal pseudo-labels, optimize the encoder within each modality using identity classification loss L:

[0032]

[0033]

[0034] in and The visible image features f are respectively i v and infrared image features f i ir The center of their cluster, that is, the center of their positive cluster, C v and C ir These are the sets of cluster centers in the visible and infrared modes, respectively, τ idc This is for temperature hyperparameters; ultimately, the pseudo-labels within the modality are saved.

[0035] In step S2, the images within the visible modality are converted to grayscale, and a text sentence containing learnable text embeddings is set for each mixed cluster of visible and grayscale images.

[0036] Furthermore, in one embodiment of the present invention, step S2 specifically includes:

[0037] Step S201: Perform grayscale processing on the images within the visible modality and assign the corresponding pseudo-label of the original visible image to each grayscale image;

[0038] Step S202: Set a text sentence containing learnable text embeddings for each cluster of mixed visible and grayscale images.

[0039] Specifically, the images within the visible modality are converted to grayscale, and each grayscale image is assigned a pseudo-label of the original visible image; for each cluster of visible and grayscale images, a text sentence containing the learnable text embedding “[X]1[X]2[X]3[X]4” is set, which is “A photo of a[X]1[X]2[X]3[X]4person”.

[0040] In step S3, the image module and text module of the visual language model are frozen, the learnable text embedding is optimized using image-text similarity loss, and the text features are preserved.

[0041] Furthermore, in one embodiment of the present invention, step S3 specifically includes:

[0042] Step S301: Freeze the image module and text module of the visual language model, and input any image and its corresponding text sentence from any mixture cluster into the image module and text module respectively to obtain image features and text features respectively;

[0043] Step S302: Optimize the learnable text embedding using image-text similarity loss and save the text features.

[0044] Specifically, the image and text modules of the visual language model are frozen. Any image from any mixture cluster and its corresponding text sentence are input into the image and text modules respectively to obtain image features and text features respectively; then, the image-text similarity loss L is used. clip Optimize learnable text embeddings:

[0045] L clip =L v2t +L t2v

[0046]

[0047] Where B is the number of images in a batch, and f is the image feature. i With text features t i With the same pseudo-label, U(t) i ) indicates that the text feature t is present in this batch. i Image features f with the same pseudo-label p The set; ultimately, all text features are saved.

[0048] In step S4, image similarity between modalities is calculated based on cluster centers, and text similarity between modalities is calculated based on text features to obtain positive clusters between modalities.

[0049] Furthermore, in one embodiment of the present invention, step S4 specifically includes:

[0050] Step S401: Calculate image similarity between modes based on cluster centers;

[0051] Step S402: Calculate text similarity between modalities based on text features;

[0052] Step S403: Calculate bi-semantic similarity based on image similarity and text similarity;

[0053] Step S404: Match positive clusters between modalities based on dual semantic similarity and the Hungarian algorithm.

[0054] Specifically, image similarity between modalities is calculated based on cluster centers:

[0055]

[0056] in and These are the cluster centers in the infrared and visible modes, respectively; such as Figure 3 As shown, text similarity between modalities is calculated based on text features:

[0057]

[0058] in The infrared image features are those of the i-th cluster in the infrared mode. t represents the number of image features in this cluster. j Text features in the visible modality; bi-semantic similarity is calculated based on image similarity and text similarity:

[0059]

[0060] Finally, positive clusters between modalities are matched based on bi-semantic similarity and the Hungarian algorithm.

[0061] In step S5, the encoder is optimized based on the positive clusters between modalities using cross-modal image semantic contrast loss and cross-modal text semantic consistency loss.

[0062] Furthermore, in one embodiment of the present invention, step S5 specifically includes:

[0063] Step S501: Based on the positive clusters between modalities, optimize the encoder using cross-modal image semantic contrast loss;

[0064] Step S502: Based on the positive clusters between modalities, optimize the encoder using cross-modal text semantic consistency loss.

[0065] Specifically, such as Figure 4 As shown, based on positive clusters between modalities, the cross-modal image semantic contrast loss L is utilized. cmc Optimize the encoder:

[0066]

[0067] in Image features f i a The positive cluster center between modes, a, b ∈ {ir, v} and a ≠ b, C b for The set of all cluster centers within the given mode, τ cmc The temperature hyperparameter is used; cross-modal text semantic consistency loss L is employed. tsc Optimize the encoder:

[0068]

[0069] Where t p For visible image features f i v and infrared image features f i ir The corresponding text features, N m This represents the number of positive clusters among the modes in the current phase.

[0070] In step S6, the optimized encoder is used to extract features from the visible and infrared images to complete the cross-modal pedestrian re-identification task.

[0071] The following specific embodiment further illustrates the unsupervised cross-modal person re-identification method based on image-text comparison learning proposed in this invention.

[0072] This embodiment uses Rank-1, Rank-10, Rank-20 and mean average precision (mAP) to evaluate the performance comparison between the ITFL of the present invention and existing unsupervised methods. The existing unsupervised methods selected include H2H, ADCA, CHCR, CCLNet, PGM, MIMR and SCA-RCP.

[0073] The dataset used was SYSU-MM01, which was collected by 4 visible cameras and 2 infrared cameras, and contained 22,258 visible images and 11,909 infrared images from 395 identities. Its training set contained 296 identities, the validation set contained 99 identities, and the test set contained 96 identities. The evaluation modes were All Search and Indoor Search. For All Search, the image database contained visible images captured from all visible cameras. For Indoor Search, the image database contained visible images captured by two indoor visible cameras.

[0074] The following table shows the comparison results of the present invention with existing unsupervised methods on two evaluation modes of the SYSU-MM01 dataset.

[0075]

[0076] As can be seen, this invention has achieved significant advantages over the optimal method PGM in both evaluation modes of the SYSU-MM01 dataset. Therefore, this invention is completely superior to existing unsupervised methods.

[0077] In summary, the unsupervised cross-modal person re-identification method based on image-text contrastive learning proposed in this embodiment of the invention effectively utilizes text semantics to promote modality invariance learning. Specifically, when acquiring text features, the modality independence of text features is effectively guaranteed; in the process of acquiring positive clusters between modalities, image similarity and text similarity between modalities are effectively utilized to improve the reliability of positive clusters between modalities; in addition, cross-modal image semantic contrastive loss is used to effectively mitigate the influence of modality differences, and cross-modal text semantic consistency loss is used to further promote modality invariance learning.

[0078] It should be understood that any parts not described in detail in this specification belong to the prior art.

[0079] The above description, in conjunction with the accompanying drawings, is merely a specific embodiment and process of the present invention. However, the scope of protection of the present invention is not limited thereto. Any person skilled in the art should understand that this is only an illustrative example, and various changes and substitutions can be made to this embodiment without departing from the essence of the present invention. The scope of the present invention is defined only by the appended claims.

[0080] The embodiments described with reference to the accompanying drawings are exemplary and are only used to explain the invention. They should not be construed as limiting the invention. The specific scope of the embodiments of the invention is not limited thereto. On the contrary, all embodiments of the invention include all variations and modifications that fall within the spirit and scope of the appended claims.

Claims

1. An unsupervised cross-modal person re-identification method based on image-text comparison learning, characterized in that, Includes the following steps: Step S1: Perform clustering within each modality and calculate cluster centers within the modality based on the pseudo-labels. Optimize the encoder within each modality using the identity classification loss. Specifically, this includes: Step S101: Use the encoder to extract features from the image and perform clustering within each modality to obtain pseudo-labels within the modality; Step S102: Calculate the cluster center within the modality based on the pseudo-labels within the modality; Step S103: Based on the pseudo-labels within the modality, optimize the encoder within each modality using the identity classification loss; Step S2: Convert the images within the visible modality to grayscale, and set a text sentence containing learnable text embeddings for each cluster of visible and grayscale images, specifically including: Step S201: Perform grayscale processing on the images within the visible modality and assign the corresponding pseudo-label of the original visible image to each grayscale image; Step S202: Set a text sentence containing learnable text embeddings for each cluster of mixed visible and grayscale images; Step S3: Freeze the image and text modules of the visual language model, optimize the learnable text embedding using image-text similarity loss, and preserve text features, specifically including: Step S301: Freeze the image module and text module of the visual language model, and input any image and its corresponding text sentence from any mixture cluster into the image module and text module respectively to obtain image features and text features respectively; Step S302: Optimize the learnable text embedding using image-text similarity loss and save the text features; Step S4: Calculate image similarity between modalities based on the cluster centers, and calculate text similarity between modalities based on the text features to obtain positive clusters between modalities, specifically including: Step S401: Calculate image similarity between modes based on cluster centers; Step S402: Calculate text similarity between modalities based on text features; Step S403: Calculate bi-semantic similarity based on image similarity and text similarity; Step S404: Match positive clusters between modalities based on dual semantic similarity and the Hungarian algorithm; Step S5: Based on the positive clusters between the modalities, optimize the encoder using cross-modal image semantic contrast loss and cross-modal text semantic consistency loss, specifically including: Step S501: Based on the positive clusters between modalities, optimize the encoder using cross-modal image semantic contrast loss; Step S502: Based on the positive clusters between modalities, optimize the encoder using cross-modal text semantic consistency loss; Step S6: Use the optimized encoder to extract features from the visible and infrared images to complete the cross-modal pedestrian re-identification task.

Images