A cross-modal pedestrian re-identification method, device and electronic equipment

Abstract

This application belongs to the field of computer vision and natural language processing technology, and specifically discloses a cross-modal person re-identification method, device, and electronic device. The method includes: acquiring a first positive and negative sample set and a second positive and negative sample set; and training a cross-modal person re-identification model based on the sample sets through bilateral contrastive learning. The first positive and negative sample set is constructed by forming positive sample pairs from image and text samples belonging to the same pedestrian and negative sample pairs from image and text samples belonging to different pedestrians. The second positive and negative sample set is constructed by forming positive sample pairs from fine-grained text features and fine-grained visual features of the same attribute belonging to the same pedestrian and negative sample pairs from fine-grained text features and fine-grained visual features belonging to different pedestrians or different attributes. This application enables the cross-modal person re-identification model to learn global representations while simultaneously modeling finer-grained local features.

Description

Technical Field

[0001] This application belongs to the field of computer vision and natural language processing technology, and more specifically, relates to a cross-modal pedestrian re-identification method, device and electronic device. Background Technology

[0002] Human-centered tasks, such as image / text-based person re-identification, person attribute recognition, person search, and human body analysis, are increasingly influential in a wide range of applications, including security monitoring, smart cities, virtual reality, and scene understanding. Thanks to advancements in task-specific method design, these tasks have made significant progress in recent years. However, recent research indicates that simply developing complex models based on specific modalities has reached performance bottlenecks. Meanwhile, work on general representation learning has shown great potential to further improve model performance. In particular, image / text modality representation learning can more flexibly adapt to various task requirements and has a lower barrier to entry for practical use.

[0003] Pre-training on the pure vision dataset ImageNet has proven to be an effective technique for learning strong visual representations, achieving significant performance across various visual tasks. However, due to the significant domain gap between general images and human images, this approach is suboptimal for learning human-specific representations. To address this issue, existing research attempts to learn pedestrian representations directly from large-scale unlabeled or noisy labeled pedestrian images. Some researchers have turned to designing a self-supervised learning framework that integrates contrastive learning, masked image modeling, and semantic control into a unified framework. While this pure vision-based approach is effective for some visual tasks, it tends to overlook explicit fine-grained attributes, such as a blue hat or white shirt, when learning general and discriminative representations due to the lack of descriptive metrics. These fine-grained attributes naturally provide distinctive information for human-centered tasks.

[0004] To address the ambiguity and uncertainty of fine-grained information in a single visual modality, recent self-supervised learning frameworks have introduced text as an additional modality. Since text is a logically processed modality, it often contains a large number of abstract relationships and attributes, serving as a descriptive definition for various visual information. However, recent methods focus on the global alignment of the visual and text modalities, neglecting the fine-grained pedestrian information within the visual modality. This results in learned visual representations with limited semantic information and weak robustness. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this application is to enable cross-modal pedestrian re-identification models to learn global representations while modeling finer-grained local features.

[0006] To achieve the above objectives, in a first aspect, this application provides a cross-modal pedestrian re-identification method, comprising:

[0007] Obtain the first set of positive and negative samples and the second set of positive and negative samples;

[0008] Based on the first and second sets of positive and negative samples, the cross-modal pedestrian re-identification model is trained by bilateral comparison learning.

[0009] The first set of positive and negative samples is constructed by forming positive sample pairs from text samples and image samples belonging to the same pedestrian and negative sample pairs from image samples and text samples belonging to different pedestrians.

[0010] The second set of positive and negative samples is constructed by forming positive sample pairs from fine-grained text features and fine-grained visual features of the same attribute belonging to the same pedestrian, forming negative sample pairs from fine-grained text features and fine-grained visual features of different pedestrians, and forming negative sample pairs from fine-grained text features and fine-grained visual features of different attributes.

[0011] In one possible implementation, the cross-modal person re-identification model includes: a text encoder, a visual encoder, and a fusion encoder;

[0012] A text encoder is used to extract text feature vectors corresponding to text samples;

[0013] A visual encoder is used to extract visual feature vectors corresponding to image samples;

[0014] The fusion encoder is used to fuse the text feature vectors output by the text encoder and the visual feature vectors output by the visual encoder between different modalities, and outputs the fused text feature vectors and the fused visual feature vectors.

[0015] In one possible implementation, fine-grained text features are obtained through the following steps:

[0016] Based on text samples, word positions are obtained by locating text attributes;

[0017] Based on word position, fine-grained text features are extracted from the text feature vector output by the fusion encoder.

[0018] In one possible implementation, fine-grained visual features are obtained through the following steps:

[0019] Based on image samples, bounding boxes describing the position of objects are obtained through visual semantic localization.

[0020] Based on the bounding box, fine-grained visual features are extracted from the visual feature vector output by the fusion encoder.

[0021] In one possible implementation, after training the cross-modal person re-identification model using bilateral contrastive learning based on a first set of positive and negative samples and a second set of positive and negative samples, the following is also included:

[0022] Based on the word vectors corresponding to the text samples and the word vectors corresponding to each noun phrase in the text samples, the part of the word vectors corresponding to the text samples that belongs to the noun phrases is replaced with masked word vectors to obtain the masked text word vectors;

[0023] The text encoder converts the masked word vectors into masked text feature vectors.

[0024] Input the visual feature vector without masking and the text feature vector with masking into the fusion encoder to obtain the fused text feature vector with masking.

[0025] The first decoding result is obtained by decoding the fused text feature vector under masked conditions using a text decoder.

[0026] Based on the first decoding result and the fused text feature vector without masking, calculate the first loss and obtain the first loss value;

[0027] Based on the first loss value, adjust the model parameters of the cross-modal pedestrian re-identification model and the model parameters of the text decoder.

[0028] In one possible implementation, calculating the first loss specifically involves calculating the negative log-likelihood loss.

[0029] In one possible implementation, after training the cross-modal person re-identification model using bilateral contrastive learning based on a first set of positive and negative samples and a second set of positive and negative samples, the following is also included:

[0030] Based on the block vectors of image samples, the block vectors within the bounding boxes of computational visual semantic localization are replaced with masked block vectors to obtain the masked visual block vectors.

[0031] The masked visual block vector is converted into a masked visual feature vector using a visual encoder.

[0032] Input the visual feature vector with masking and the text feature vector without masking into the fusion encoder to obtain the fused visual feature vector with masking.

[0033] The second decoding result is obtained by decoding the fused visual feature vector under the condition of masking through a visual decoder.

[0034] Based on the second decoding result and the fused visual feature vector without masking, the second loss is calculated and the second loss value is obtained.

[0035] Based on the second loss value, adjust the model parameters of the cross-modal pedestrian re-identification model and the model parameters of the visual decoder.

[0036] In one possible implementation, calculating the second loss specifically involves calculating the KL divergence loss.

[0037] Secondly, this application provides a cross-modal pedestrian re-identification device, comprising:

[0038] The positive and negative sample acquisition module is used to acquire the first positive and negative sample set and the second positive and negative sample set.

[0039] The training module is used to perform bilateral contrastive learning training on the cross-modal pedestrian re-identification model based on the first positive and negative sample set and the second positive and negative sample set;

[0040] The first set of positive and negative samples is constructed by forming positive sample pairs from text samples and image samples belonging to the same pedestrian and negative sample pairs from image samples and text samples belonging to different pedestrians.

[0041] The second set of positive and negative samples is constructed by forming positive sample pairs from fine-grained text features and fine-grained visual features of the same attribute belonging to the same pedestrian, forming negative sample pairs from fine-grained text features and fine-grained visual features of different pedestrians, and forming negative sample pairs from fine-grained text features and fine-grained visual features of different attributes.

[0042] Thirdly, this application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect or any possible implementation thereof.

[0043] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art:

[0044] (1) This application uses a reasonable fine-grained feature alignment method to enable the network to learn and model more diverse fine-grained information, thereby enhancing the information richness of global features and improving the accuracy and robustness of the model representation.

[0045] (2) The first set of positive and negative samples is constructed by forming positive sample pairs by combining text samples and image samples belonging to the same pedestrian and negative sample pairs by combining image samples and text samples belonging to different pedestrians. By using a global feature alignment method that is more in line with the characteristics of pedestrian datasets, the selection of positive sample pairs is more reasonable and effective, thereby making the training of the model more efficient and accurate.

[0046] (3) This application uses an effective attribute masking strategy, which uses noun phrase masking operation instead of random masking operation, so that the text reconstruction task contains semantics and enhances the model's ability to understand and extract text representations.

[0047] (4) This application uses an effective semantic masking strategy, which replaces random masking with semantic masking operation, so that the visual reconstruction task contains semantics and enhances the model’s ability to understand and extract visual representations. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating the cross-modal pedestrian re-identification method provided in the embodiments of this application;

[0049] Figure 2 This is a schematic diagram of global feature alignment and fine-grained feature alignment provided in the embodiments of this application;

[0050] Figure 3 This is a schematic diagram of visual block vector and text word vector extraction and corresponding modality masking operations provided in the embodiments of this application;

[0051] Figure 4 This is a schematic diagram of text attribute mask reconstruction based on noun phrases provided in an embodiment of this application;

[0052] Figure 5 This is a schematic diagram of image semantic mask reconstruction based on object position provided in an embodiment of this application;

[0053] Figure 6 This is a schematic diagram of the cross-modal pedestrian re-identification device provided in the embodiments of this application;

[0054] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of the objects. For example, "first positive and negative sample set" and "second positive and negative sample set," etc., are used to distinguish different sets of positive and negative samples, not to describe a specific order of the positive and negative sample sets.

[0057] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0058] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0059] The embodiments of this application are described below with reference to the accompanying drawings.

[0060] Figure 1 This is a flowchart illustrating the cross-modal pedestrian re-identification method provided in this application embodiment, as shown below. Figure 1 As shown, the method includes the following steps S101 and S102.

[0061] Step S101: Obtain the first set of positive and negative samples and the second set of positive and negative samples;

[0062] Step S102: Based on the first positive and negative sample set and the second positive and negative sample set, perform bilateral contrast learning training on the cross-modal pedestrian re-identification model;

[0063] The first set of positive and negative samples is constructed by forming positive sample pairs by combining text samples (samples that describe pedestrian features in textual form) and image samples (samples that describe pedestrian features in image form) belonging to the same pedestrian, and forming negative sample pairs by combining image samples and text samples belonging to different pedestrians.

[0064] The second set of positive and negative samples is constructed by forming positive sample pairs from fine-grained text features and fine-grained visual features of the same attribute belonging to the same pedestrian, forming negative sample pairs from fine-grained text features and fine-grained visual features of different pedestrians, and forming negative sample pairs from fine-grained text features and fine-grained visual features of different attributes.

[0065] For the positive sample pairs in the first set of positive and negative samples mentioned above, an example is given here. For instance, Text_1 belongs to the text sample of the pedestrian with ID 1 (named Pedestrian_1), and Image_1 belongs to the image sample of Pedestrian_1. Then Text_1 and Image_1 form a positive sample pair.

[0066] For the negative sample pairs in the first set of positive and negative samples mentioned above, an example is given here. For instance, if Text_1 is a text sample of Pedestrian_1 and Image_2 is an image sample of a pedestrian with ID 2 (named Pedestrian_2), then Text_1 and Image_2 form a negative sample pair.

[0067] Regarding the aforementioned "one attribute", it is explained here that an attribute is used to describe the characteristics of a part of a pedestrian. For example, an attribute Property_1 can be used to describe the upper body clothing of a pedestrian, another attribute Property_2 can be used to describe the head clothing of a pedestrian, and yet another attribute Property_3 can be used to describe the foot clothing of a pedestrian.

[0068] For positive sample pairs in the second set of positive and negative samples mentioned above, for example, if the fine-grained text feature Text_1_1 is the fine-grained text feature corresponding to Property_1 of Pedestrian_1, and Image_1_1 is the fine-grained visual feature corresponding to Property_1 of Pedestrian_1, then Text_1_1 and Image_1_1 form a positive sample pair.

[0069] Regarding the above-mentioned formation of negative sample pairs from fine-grained text features and fine-grained visual features belonging to different pedestrians, an example is provided below. For instance, if Text_1_1 is a fine-grained text feature corresponding to Property_1 of Pedestrian_1, and Image_2_1 is a fine-grained visual feature corresponding to Property_1 of Pedestrian_2, then Text_1_1 and Image_2_1 are fine-grained text features and fine-grained visual features belonging to different pedestrians, and Text_1_1 and Image_2_1 form a negative sample pair. Similarly, if Text_1_1 is a fine-grained text feature corresponding to Property_1 of Pedestrian_1, and Image_2_2 is a fine-grained visual feature corresponding to Property_2 of Pedestrian_2, then Text_1_1 and Image_2_2 are fine-grained text features and fine-grained visual features belonging to different pedestrians, and Text_1_1 and Image_2_2 form a negative sample pair.

[0070] To illustrate the above example of forming negative sample pairs from fine-grained text features and fine-grained visual features belonging to different property types, consider the following: For instance, if Text_1_1 is a fine-grained text feature corresponding to Property_1 of Pedestrian_1, and Image_1_2 is a fine-grained visual feature corresponding to Property_2 of Pedestrian_1, then Text_1_1 and Image_1_2 belong to fine-grained text features and fine-grained visual features belonging to different property types, and Text_1_1 and Image_1_2 form a negative sample pair. Similarly, if Text_1_1 is a fine-grained text feature corresponding to Property_1 of Pedestrian_1, and Image_2_2 is a fine-grained visual feature corresponding to Property_2 of Pedestrian_2, then Text_1_1 and Image_2_2 belong to fine-grained text features and fine-grained visual features belonging to different property types, and Text_1_1 and Image_2_2 form a negative sample pair.

[0071] Figure 2 This is a schematic diagram of global feature alignment and fine-grained feature alignment provided in the embodiments of this application, as shown below. Figure 2 As shown, it can be understood that the first positive and negative sample set is constructed by forming positive sample pairs by combining text samples and image samples belonging to the same pedestrian and negative sample pairs by combining image samples and text samples belonging to different pedestrians. By using the first positive and negative sample set to perform bilateral contrastive learning training on the cross-modal pedestrian re-identification model, the model can learn aligned global visual representations and global text representations.

[0072] Aligning global features can make global visual representations and global text representations as similar as possible in a unified feature space, but it neglects the capture and modeling of fine-grained features, resulting in limited information contained in global features and insufficient robustness. To overcome this deficiency, this application proposes a fine-grained feature alignment scheme, enabling the model to learn both global representations and finer-grained local features. Specifically, as follows... Figure 2 As shown, the second set of positive and negative samples is constructed by forming positive sample pairs from fine-grained text features and fine-grained visual features of the same attribute belonging to the same pedestrian, and forming negative sample pairs from fine-grained text features and fine-grained visual features of different attributes belonging to the same pedestrian. By using the second set of positive and negative samples to train the cross-modal pedestrian re-identification model through bilateral contrastive learning, the model can learn aligned fine-grained visual and text representations.

[0073] Therefore, by using the first and second sets of positive and negative samples to train the cross-modal pedestrian re-identification model through bilateral contrastive learning, the model can learn both global representations and model more fine-grained local features, which can effectively improve the accuracy and robustness of the model.

[0074] Furthermore, existing global feature alignment schemes only treat the current image and its corresponding text as positive sample pairs, while treating all other text as negative sample pairs. However, for pedestrian datasets, different images and text descriptions of the same pedestrian can be combined into positive sample pairs. Therefore, existing global feature alignment methods cannot adapt to the characteristics of pedestrian datasets, thus affecting the learning of pedestrian representations. To overcome this deficiency, the first set of positive and negative samples is constructed by forming positive sample pairs from text and image samples belonging to the same pedestrian and negative sample pairs from image and text samples belonging to different pedestrians. Through a global feature alignment method that better suits the characteristics of pedestrian datasets, the selection of positive sample pairs becomes more reasonable and effective, thereby making model training more efficient and accurate.

[0075] For example, image samples and text samples in a batch can be matched according to pedestrian ID (one pedestrian corresponds to one pedestrian ID). Image samples and text samples with the same pedestrian ID form positive sample pairs, and image samples and text samples with different pedestrian IDs form negative sample pairs to construct a first set of positive and negative samples. Then, these positive and negative sample pairs (positive and negative sample pairs in the first set of positive and negative samples) are used for comparative learning. By using a global feature alignment method related to pedestrian ID that is more in line with the characteristics of pedestrian datasets, the selection of positive sample pairs is more reasonable and effective, thereby making the training of the model more efficient and accurate.

[0076] In one possible implementation, the cross-modal person re-identification model includes a text encoder, a visual encoder, and a fusion encoder; the text encoder is used to extract the text feature vector corresponding to the text sample; the visual encoder is used to extract the visual feature vector corresponding to the image sample; the fusion encoder is used to perform feature fusion between different modalities on the text feature vector output by the text encoder and the visual feature vector output by the visual encoder, and output the fused text feature vector and the fused visual feature vector.

[0077] Figure 3 This is a schematic diagram illustrating the extraction of visual block vectors and text word vectors, as well as the corresponding modal masking operations, provided in the embodiments of this application. Figure 3 As shown, in one possible implementation, fine-grained text features are obtained through the following steps: based on text samples, word positions are obtained by locating text attributes; based on word positions, fine-grained text features are extracted from the text feature vector output by the fusion encoder.

[0078] Understandably, existing text masking reconstruction methods perform random masking and replacement operations on text samples. This approach does not consider word parts of speech and grammatical structure, easily leading to information leakage and insufficient model training. To overcome this deficiency, this application uses attribute masking, specifically as follows: First, part-of-speech tagging is performed on all words in the text sample using a contextual part-of-speech analysis tool. Then, noun phrases are masked as a whole. Finally, during the training phase, the text encoder and decoder are required to recover the masked noun phrases from the masked text samples. Through this attribute masking reconstruction task of pedestrian text representation, cross-modal pedestrian re-identification models can learn richer, fine-grained pedestrian text representations.

[0079] like Figure 3 As shown, in one possible implementation, fine-grained visual features are obtained through the following steps: based on image samples, bounding boxes describing the position of objects are obtained through visual semantic localization; based on the bounding boxes, fine-grained visual features are extracted from the visual feature vector output by the fusion encoder.

[0080] Understandably, existing visual representation masking reconstruction tasks are an effective method for learning visual representations. However, due to the redundancy of image information, random image masking operations cannot completely guarantee the coverage of effective information, which can also lead to information leakage and insufficient model training. To overcome this deficiency, this application uses semantic masking operations, specifically as follows: First, the noun phrases from one of the above implementations are used as descriptive words. Visual semantic localization (e.g., GroundingDINO) is input to find the bounding boxes of the objects in the image corresponding to the noun phrases. Then, the objects within the bounding boxes are masked as a whole. Finally, during the training phase, the visual encoder and decoder are required to recover the masked object parts from the masked image samples. Through the above semantic masking reconstruction task of pedestrian visual representations, cross-modal pedestrian re-identification models can learn richer, fine-grained visual representations of pedestrians.

[0081] Figure 4 This is a schematic diagram of text attribute mask reconstruction based on noun phrases provided in an embodiment of this application, such as... Figure 4 As shown, in one possible implementation, after training the cross-modal pedestrian re-identification model using bilateral contrastive learning based on the first and second positive and negative sample sets, the following is also included:

[0082] Based on the word vectors corresponding to the text samples and the word vectors corresponding to each noun phrase in the text samples, the part of the word vectors corresponding to the text samples that belongs to the noun phrases is replaced with masked word vectors to obtain the masked text word vectors;

[0083] The text encoder converts the masked word vectors into masked text feature vectors.

[0084] Input the visual feature vector without masking and the text feature vector with masking into the fusion encoder to obtain the fused text feature vector with masking.

[0085] The first decoding result is obtained by decoding the fused text feature vector under masked conditions using a text decoder.

[0086] Based on the first decoding result and the fused text feature vector without masking, calculate the first loss and obtain the first loss value;

[0087] Based on the first loss value, adjust the model parameters of the cross-modal pedestrian re-identification model and the model parameters of the text decoder.

[0088] In one possible implementation, calculating the first loss specifically involves calculating the negative log-likelihood loss.

[0089] Figure 5 This is a schematic diagram of image semantic mask reconstruction based on object location provided in an embodiment of this application, such as... Figure 5 As shown, in one possible implementation, after training the cross-modal pedestrian re-identification model using bilateral contrastive learning based on the first and second positive and negative sample sets, the following is also included:

[0090] Based on the block vectors of image samples, the block vectors within the bounding boxes of computational visual semantic localization are replaced with masked block vectors to obtain the masked visual block vectors.

[0091] The masked visual block vector is converted into a masked visual feature vector using a visual encoder.

[0092] Input the visual feature vector with masking and the text feature vector without masking into the fusion encoder to obtain the fused visual feature vector with masking.

[0093] The second decoding result is obtained by decoding the fused visual feature vector under the condition of masking through a visual decoder.

[0094] Based on the second decoding result and the fused visual feature vector without masking, the second loss is calculated and the second loss value is obtained.

[0095] Based on the second loss value, adjust the model parameters of the cross-modal pedestrian re-identification model and the model parameters of the visual decoder.

[0096] Furthermore, in one possible implementation, calculating the second loss specifically involves calculating the KL divergence loss.

[0097] Understandably, this application can leverage multimodal pedestrian attribute reconstruction and multimodal pedestrian representation alignment to achieve better pedestrian representation learning. Through multi-level alignment of global and fine-grained features, and text / visual bimodal reconstruction of fine-grained features, this application proposes an effective end-to-end algorithm for cross-modal pedestrian re-identification tasks, which can fully utilize the abstract relationships and attributes of the text modality to build better pedestrian visual representations.

[0098] The cross-modal pedestrian re-identification method provided in this application is illustrated below with an example.

[0099] First, let me introduce the overall scheme for cross-modal pedestrian re-identification, which consists of four parts: (1) extraction of pedestrian text features and visual features; (2) alignment of global features related to pedestrian ID and alignment of fine-grained pedestrian features; (3) reconstruction of text attribute masks based on noun phrases; and (4) reconstruction of image semantic masks based on object position. These four parts will be explained one by one below.

[0100] The features extracted by the first part of the model provide data support for subsequent parts. The model mainly consists of three parts: a text encoder (e.g., BERT), a visual encoder (e.g., ViT), and a fusion encoder (e.g., Transformer). The first step uses the text encoder's tokenizer (input layer) to decompose the text into independent word vectors, which are then fed into the subsequent network of the text encoder to obtain text feature vectors. Similarly, the visual encoder uses its visual embedding layer (input layer) to decompose the image into independent block vectors, which are then fed into the subsequent network of the visual encoder to obtain visual feature vectors. The second step feeds both the text and visual feature vectors into the fusion encoder, resulting in text and visual feature vectors after modal interaction and fusion. In the text feature vector, the feature vector representing the end of a sentence is designated as the text global feature, and in the visual feature vector, the class feature vector added in the visual embedding layer is the visual global feature.

[0101] The second part involves learning and aligning the visual and textual representations of pedestrians. For global features, global positive and negative sample pairs are constructed using pedestrian IDs, and bilateral contrastive learning is performed between text and image, and between image and text, enabling the model to learn aligned global visual and textual representations. For fine-grained features, fine-grained positive and negative sample pairs are constructed through text attribute localization and visual semantic localization, enabling the model to align fine-grained visual and textual representations. The alignment of global representations is described below. For image-text sample pairs in a batch, the sample pairs are first grouped using their pedestrian IDs. Sample pairs with the same pedestrian ID are assigned to one group. Then, images and text within the same group form positive sample pairs, and images and text from different groups form negative sample pairs. Under the loss effect of contrastive learning, the distance between global visual features and global textual features in positive sample pairs will decrease, while the distance between global visual features and global textual features in negative sample pairs will increase. Finally, the network can learn aligned global visual and global textual representations. The alignment of fine-grained representations is described below. For each image-text sample pair in a batch, text attribute localization (e.g., NLTK) is first used to extract word positions representing fine-grained text representations. Then, fine-grained text features are extracted from the text feature vector based on these position coordinates. Next, visual semantic localization (e.g., GroundingDINO) is used to extract bounding box positions representing fine-grained visual representations. Then, fine-grained visual features are extracted from the visual feature vector using the bounding boxes. Fine-grained features belonging to the same pedestrian and the same attribute form positive sample pairs, while fine-grained features belonging to the same pedestrian but different attributes form negative sample pairs. Contrastive learning brings the fine-grained features in positive sample pairs closer together and the fine-grained features in negative sample pairs farther apart. Finally, the network learns aligned fine-grained visual and text representations.

[0102] The third part involves learning a more accurate text representation using text attribute masking to reconstruct the text. First, the text sample is input into the word segmenter of the text encoder to obtain word vectors. Then, the corresponding noun phrases are also input into the same segmenter to obtain noun phrase word vectors. Next, the parts of the text sample word vectors belonging to noun phrases are replaced with masked word vectors, thus masking the entire noun phrase. The masked text word vectors are then fed into the text encoder to obtain masked text feature vectors. Next, the masked text feature vectors and the complete visual feature vector (i.e., the visual feature vector without masking) are fed into the fusion encoder to obtain the masked fused text feature vector (i.e., the fused text feature vector with masking). Finally, a lightweight text decoder (lightweight means the decoder has few network layers, such as one layer) decodes the masked fused text feature vector and calculates a negative log-likelihood loss with the complete fused text feature vector (i.e., the fused text feature vector without masking) to reconstruct the masked text portion.

[0103] The fourth part uses a visual semantic masking reconstruction task to learn a more accurate visual representation. First, image samples are input into the visual embedding layer of the visual encoder to obtain block vectors of the image samples. Then, the absolute position coordinates of the semantically localized bounding boxes within all block vectors are calculated. Next, the block vectors within the bounding boxes are replaced with learnable mask block vectors, thus achieving the goal of completely masking the visual semantic part. The masked visual block vectors are then fed into the subsequent network of the visual encoder to obtain masked visual feature vectors. Next, the masked visual feature vectors and the complete text feature vectors (i.e., the text feature vectors without masking) are fed into the fusion encoder to obtain the masked fused visual feature vectors (i.e., the fused visual feature vectors with masking). Finally, a lightweight visual decoder decodes the masked fused visual feature vectors and calculates the KL divergence loss with the complete fused visual feature vectors (i.e., the fused visual feature vectors without masking) to reconstruct the masked visual part. KL divergence loss is a loss function used to measure the difference between two probability distributions. In machine learning, KL divergence is often used to measure the difference between the distribution predicted by the model and the true distribution, and is used as a loss function to optimize the model's parameters.

[0104] The cross-modal pedestrian re-identification device provided in this application is described below. The cross-modal pedestrian re-identification device described below can be referred to in correspondence with the cross-modal pedestrian re-identification method described above.

[0105] Figure 6 This is a schematic diagram of the cross-modal pedestrian re-identification device provided in the embodiments of this application, as shown below. Figure 6 As shown, the device includes: a positive and negative sample acquisition module 10 and a training module 20. Wherein:

[0106] The positive and negative sample acquisition module 10 is used to acquire a first set of positive and negative samples and a second set of positive and negative samples.

[0107] Training module 20 is used to perform bilateral contrastive learning training on the cross-modal pedestrian re-identification model based on the first positive and negative sample set and the second positive and negative sample set;

[0108] The first set of positive and negative samples is constructed by forming positive sample pairs from text samples and image samples belonging to the same pedestrian and negative sample pairs from image samples and text samples belonging to different pedestrians.

[0109] The second set of positive and negative samples is constructed by forming positive sample pairs from fine-grained text features and fine-grained visual features of the same attribute belonging to the same pedestrian, forming negative sample pairs from fine-grained text features and fine-grained visual features of different pedestrians, and forming negative sample pairs from fine-grained text features and fine-grained visual features of different attributes.

[0110] It is understood that the detailed functional implementation of each of the above units / modules can be found in the description in the aforementioned method embodiments, and will not be repeated here.

[0111] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.

[0112] Based on the methods in the above embodiments, this application provides an electronic device. Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 7 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the methods in the above embodiments.

[0113] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0114] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0115] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0116] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0117] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0118] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0119] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0120] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A cross-modal pedestrian re-identification method, characterized in that, include: Obtain the first set of positive and negative samples and the second set of positive and negative samples; Based on the first set of positive and negative samples and the second set of positive and negative samples, the cross-modal pedestrian re-identification model is trained by bilateral comparison learning. The first set of positive and negative samples is constructed by forming positive sample pairs from text samples and image samples belonging to the same pedestrian and negative sample pairs from image samples and text samples belonging to different pedestrians. The second set of positive and negative samples is constructed by forming positive sample pairs from fine-grained text features and fine-grained visual features of the same attribute belonging to the same pedestrian, forming negative sample pairs from fine-grained text features and fine-grained visual features of different pedestrians, and forming negative sample pairs from fine-grained text features and fine-grained visual features of different attributes.

2. The cross-modal pedestrian re-identification method according to claim 1, characterized in that, The cross-modal pedestrian re-identification model includes: a text encoder, a visual encoder, and a fusion encoder; The text encoder is used to extract the text feature vector corresponding to the text sample; The visual encoder is used to extract the visual feature vectors corresponding to image samples; The fusion encoder is used to perform feature fusion between different modalities on the text feature vector output by the text encoder and the visual feature vector output by the visual encoder, and outputs the fused text feature vector and the fused visual feature vector.

3. The cross-modal pedestrian re-identification method according to claim 2, characterized in that, The fine-grained text features are obtained through the following steps: Based on text samples, word positions are obtained by locating text attributes; Based on the word position, fine-grained text features are extracted from the text feature vector output by the fusion encoder.

4. The cross-modal pedestrian re-identification method according to claim 2, characterized in that, The fine-grained visual features are obtained through the following steps: Based on image samples, bounding boxes describing the position of objects are obtained through visual semantic localization. Based on the bounding box, fine-grained visual features are extracted from the visual feature vector output by the fusion encoder.

5. The cross-modal pedestrian re-identification method according to claim 2, characterized in that, After training the cross-modal pedestrian re-identification model using bilateral contrastive learning based on the first and second positive and negative sample sets, the method further includes: Based on the word vectors corresponding to the text samples and the word vectors corresponding to each noun phrase in the text samples, the part of the word vectors corresponding to the text samples that belongs to the noun phrases is replaced with masked word vectors to obtain the masked text word vectors; The text encoder converts the masked word vectors into masked text feature vectors. Input the visual feature vector without masking and the text feature vector with masking into the fusion encoder to obtain the fused text feature vector with masking. The first decoding result is obtained by decoding the fused text feature vector under masked conditions using a text decoder. Based on the first decoding result and the fused text feature vector without masking, calculate the first loss and obtain the first loss value; Based on the first loss value, adjust the model parameters of the cross-modal pedestrian re-identification model and the model parameters of the text decoder.

6. The cross-modal pedestrian re-identification method according to claim 5, characterized in that, The calculation of the first loss specifically involves calculating the negative log-likelihood loss.

7. The cross-modal pedestrian re-identification method according to claim 2, characterized in that, After training the cross-modal pedestrian re-identification model using bilateral contrastive learning based on the first and second positive and negative sample sets, the method further includes: Based on the block vectors of image samples, the block vectors within the bounding boxes of computational visual semantic localization are replaced with masked block vectors to obtain the masked visual block vectors. The masked visual block vector is converted into a masked visual feature vector using a visual encoder. Input the visual feature vector with masking and the text feature vector without masking into the fusion encoder to obtain the fused visual feature vector with masking. The second decoding result is obtained by decoding the fused visual feature vector under the condition of masking through a visual decoder. Based on the second decoding result and the fused visual feature vector without masking, the second loss is calculated and the second loss value is obtained. Based on the second loss value, adjust the model parameters of the cross-modal pedestrian re-identification model and the model parameters of the visual decoder.

8. The cross-modal pedestrian re-identification method according to claim 7, characterized in that, The calculation of the second loss specifically refers to the calculation of the KL divergence loss.

9. A cross-modal pedestrian re-identification device, characterized in that, include: The positive and negative sample acquisition module is used to acquire the first positive and negative sample set and the second positive and negative sample set. The training module is used to perform bilateral contrastive learning training on the cross-modal pedestrian re-identification model based on the first positive and negative sample set and the second positive and negative sample set; The first set of positive and negative samples is constructed by forming positive sample pairs from text samples and image samples belonging to the same pedestrian and negative sample pairs from image samples and text samples belonging to different pedestrians. The second set of positive and negative samples is constructed by forming positive sample pairs from fine-grained text features and fine-grained visual features of the same attribute belonging to the same pedestrian, forming negative sample pairs from fine-grained text features and fine-grained visual features of different pedestrians, and forming negative sample pairs from fine-grained text features and fine-grained visual features of different attributes.

10. An electronic device, characterized in that, include: At least one memory for storing computer programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform the method as described in any one of claims 1-8.

Images