Cross-modal pedestrian re-identification method and device and electronic equipment

Abstract

The invention belongs to the technical field of computer vision and natural language processing, and particularly discloses a cross-modal pedestrian re-identification method and device and electronic equipment, and the method comprises the steps: obtaining a first positive and negative sample set and a second positive and negative sample set; performing bilateral contrast learning training on the cross-modal pedestrian re-identification model based on the sample set; wherein the first positive and negative sample set is constructed by forming the image-text samples belonging to the same pedestrian into positive sample pairs and forming the image-text samples belonging to different pedestrians into negative sample pairs; the second positive and negative sample set is constructed by forming a positive sample pair by the fine-grained text features and the fine-grained visual features belonging to the same attribute of the same pedestrian and forming a negative sample pair by the fine-grained text features and the fine-grained visual features belonging to different pedestrians or different attributes. According to the invention, a cross-modal pedestrian re-identification model can be realized while learning of global representation is considered, and local features with finer granularity can be modeled at the same time.

Description

Technical Field

[0001] The present application belongs to the technical field of computer vision and natural language processing, and more specifically, to a cross-modal pedestrian re-identification method, device and electronic device. Background Art

[0002] Human-centric tasks such as image / text-based person re-identification, pedestrian attribute recognition, pedestrian search, and human body parsing are becoming increasingly influential in a wide range of applications such as security monitoring, smart cities, virtual reality, and scene understanding. Thanks to the progress in the design of task-specific methods, these tasks have made significant progress in the past few years. However, recent research progress has shown that developing complex models based solely on specific modalities has encountered performance bottlenecks. At the same time, some work on general representation learning has shown great potential to further improve model performance. In particular, representation learning in image / text modalities can be more flexible to adapt to multiple task requirements and has a lower threshold in practical use.

[0003] Pre-training on the pure vision dataset ImageNet has been shown to be an effective technique for learning powerful visual representations, achieving remarkable performance on various vision tasks. However, this approach is suboptimal for learning specific representations of people due to the large domain gap between generic images and person images. To address this issue, existing studies have attempted to learn pedestrian representations directly from large-scale unlabeled or noisy labeled pedestrian images. Some researchers have turned to designing a self-supervised learning framework that integrates contrastive learning, mask image modeling, and semantic control into a unified framework. Although this pure vision-based approach is effective in some vision tasks, it is easy to ignore explicit fine-grained attributes such as blue hats, white shirts, etc. when learning generic and discriminative representations due to the lack of descriptive metrics. These fine-grained attributes naturally provide distinctive information for person-centric tasks.

[0004] In order to solve the ambiguity and uncertainty of fine-grained information in a single visual modality, recent self-supervised learning frameworks have introduced text as an additional modality. Since text is a modality that has been logically processed, it often contains a large number of abstract relationships and attributes, which can be used as a description of various visual information. However, recent methods focus on the global alignment of visual and textual modalities, while ignoring the fine-grained information of pedestrians in the visual modality, resulting in the learned visual representations having a single semantic information and weak robustness. Summary of the invention

[0005] In view of the shortcomings of the prior art, the purpose of this application is to realize a cross-modal pedestrian re-identification model that takes into account the learning of global representation while being able to model more fine-grained local features.

[0006] To achieve the above objectives, in a first aspect, the present application provides a cross-modal person re-identification method, comprising:

[0007] Obtain a first positive and negative sample set and a second positive and negative sample set;

[0008] Based on the first positive and negative sample set and the second positive and negative sample set, a bilateral contrastive learning training is performed on the cross-modal pedestrian re-identification model;

[0009] The first positive and negative sample set is constructed by combining text samples and image samples belonging to the same pedestrian into positive sample pairs and combining image samples and text samples belonging to different pedestrians into negative sample pairs;

[0010] The second positive and negative sample set is constructed by combining fine-grained text features and fine-grained visual features of the same attribute of the same pedestrian into positive sample pairs, combining fine-grained text features and fine-grained visual features belonging to different pedestrians into negative sample pairs, and combining fine-grained text features and fine-grained visual features belonging to different attributes into negative sample pairs.

[0011] In a possible implementation, the cross-modal person re-identification model includes: a text encoder, a visual encoder, and a fusion encoder;

[0012] The text encoder is used to extract the text feature vector corresponding to the text sample;

[0013] The visual encoder is used to extract the visual feature vector corresponding to the image sample;

[0014] The fusion encoder is used to perform feature fusion between different modalities on the text feature vector output by the text encoder and the visual feature vector output by the visual encoder, and output a fused text feature vector and a fused visual feature vector.

[0015] In one possible implementation, fine-grained text features are obtained through the following steps:

[0016] Based on the text sample, the word position is obtained by locating the text attributes;

[0017] Based on word positions, fine-grained text features are extracted from the text feature vector output by the fusion encoder.

[0018] In one possible implementation, fine-grained visual features are obtained by the following steps:

[0019] Based on the image samples, the bounding box used to describe the location of the object is obtained through visual semantic positioning;

[0020] Based on the bounding box, fine-grained visual features are extracted from the visual feature vector output by the fusion encoder.

[0021] In a possible implementation, after performing bilateral contrastive learning training on the cross-modal person re-identification model based on the first positive and negative sample set and the second positive and negative sample set, the method further includes:

[0022] Based on the word vector corresponding to the text sample and the word vector corresponding to each noun phrase in the text sample, the part of the word vector corresponding to the text sample belonging to the noun phrase is replaced with the mask word vector to obtain the masked text word vector;

[0023] Through the text encoder, the masked text word vector is converted into a masked text feature vector;

[0024] Input the visual feature vector without mask and the text feature vector with mask to the fusion encoder to obtain the fused text feature vector with mask;

[0025] Decoding the fused text feature vector with mask through a text decoder to obtain a first decoding result;

[0026] Based on the first decoding result and the fused text feature vector without mask, a first loss is calculated to obtain a first loss value;

[0027] Based on the first loss value, model parameters of the cross-modal person re-identification model and model parameters of the text decoder are adjusted.

[0028] In a possible implementation, calculating the first loss is specifically calculating a negative log-likelihood loss.

[0029] In a possible implementation, after performing bilateral contrastive learning training on the cross-modal person re-identification model based on the first positive and negative sample set and the second positive and negative sample set, the method further includes:

[0030] Based on the block vector of the image sample, the block vector in the bounding box for calculating the visual semantic positioning is replaced with the mask block vector to obtain the masked visual block vector;

[0031] Through the visual encoder, the masked visual block vector is converted into a visual feature vector with mask;

[0032] Input the visual feature vector with mask and the text feature vector without mask to the fusion encoder to obtain the fused visual feature vector with mask;

[0033] Decoding the fused visual feature vector with the mask through a visual decoder to obtain a second decoding result;

[0034] Calculate a second loss based on the second decoding result and the fused visual feature vector without the mask, and obtain a second loss value;

[0035] Based on the second loss value, the model parameters of the cross-modal person re-identification model and the model parameters of the visual decoder are adjusted.

[0036] In a possible implementation, calculating the second loss is specifically calculating the KL divergence loss.

[0037] In a second aspect, the present application provides a cross-modal pedestrian re-identification device, comprising:

[0038] A positive and negative sample acquisition module, used to acquire a first positive and negative sample set and a second positive and negative sample set;

[0039] A training module, configured to perform bilateral contrastive learning training on a cross-modal person re-identification model based on the first positive and negative sample set and the second positive and negative sample set;

[0040] The first positive and negative sample set is constructed by combining text samples and image samples belonging to the same pedestrian into positive sample pairs and combining image samples and text samples belonging to different pedestrians into negative sample pairs;

[0041] The second positive and negative sample set is constructed by combining fine-grained text features and fine-grained visual features of the same attribute of the same pedestrian into positive sample pairs, combining fine-grained text features and fine-grained visual features belonging to different pedestrians into negative sample pairs, and combining fine-grained text features and fine-grained visual features belonging to different attributes into negative sample pairs.

[0042] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any possible implementation of the first aspect.

[0043] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art:

[0044] (1) Through a reasonable fine-grained feature alignment method, this application can enable the network to learn and model more diverse fine-grained information, thereby enhancing the information richness of global features and improving the accuracy of the model and the robustness of representation.

[0045] (2) The first set of positive and negative samples is constructed by combining text samples and image samples belonging to the same pedestrian into positive sample pairs and combining image samples and text samples belonging to different pedestrians into negative sample pairs. Through a global feature alignment method that is more in line with the characteristics of the pedestrian dataset, the selection of positive sample pairs is more reasonable and effective, thereby making the training of the model more efficient and accurate.

[0046] (3) This application uses an effective attribute masking strategy, using noun phrase masking operations instead of random masking operations, so that the text reconstruction task contains semantics and enhances the model's ability to understand and extract text representations.

[0047] (4) This application uses an effective semantic masking strategy, using semantic masking operations instead of random masking operations, so that the visual reconstruction task contains semantics and enhances the model's ability to understand and extract visual representations. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of a cross-modal pedestrian re-identification method provided in an embodiment of the present application;

[0049] Figure 2 is a schematic diagram of global feature alignment and fine-grained feature alignment provided in an embodiment of the present application;

[0050] Figure 3 It is a schematic diagram of the visual block vector and text word vector extraction and the mask operation of the corresponding modality provided in the embodiment of the present application;

[0051] Figure 4 is a schematic diagram of text attribute mask reconstruction based on noun phrases provided in an embodiment of the present application;

[0052] Figure 5 is a schematic diagram of image semantic mask reconstruction based on object position provided by an embodiment of the present application;

[0053] Figure 6 is a schematic diagram of the structure of a cross-modal pedestrian re-identification device provided in an embodiment of the present application;

[0054] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first positive and negative sample set and a second positive and negative sample set are used to distinguish different positive and negative sample sets rather than to describe a specific order of the positive and negative sample sets.

[0057] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0058] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0059] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0060] Figure 1 is a flow chart of a cross-modal pedestrian re-identification method provided in an embodiment of the present application, such as Figure 1 As shown, the method includes the following steps S101 and S102.

[0061] Step S101, obtaining a first positive and negative sample set and a second positive and negative sample set;

[0062] Step S102, performing bilateral contrastive learning training on a cross-modal person re-identification model based on the first positive and negative sample set and the second positive and negative sample set;

[0063] The first positive and negative sample set is constructed by combining text samples (samples describing pedestrian features in a textual manner) and image samples (samples describing pedestrian features in an image manner) belonging to the same pedestrian into positive sample pairs, and combining image samples and text samples belonging to different pedestrians into negative sample pairs;

[0064] The second positive and negative sample set is constructed by combining fine-grained text features and fine-grained visual features of the same attribute of the same pedestrian into positive sample pairs, combining fine-grained text features and fine-grained visual features belonging to different pedestrians into negative sample pairs, and combining fine-grained text features and fine-grained visual features belonging to different attributes into negative sample pairs.

[0065] For the positive sample pair in the first positive and negative sample set mentioned above, an exemplary explanation is given here. For example, Text_1 belongs to the text sample of the pedestrian with ID 1 (named Pedestrian_1), and Image_1 belongs to the image sample of Pedestrian_1, then Text_1 and Image_1 constitute a positive sample pair.

[0066] For the negative sample pairs in the first positive and negative sample set mentioned above, an exemplary explanation is given here. For example, Text_1 belongs to the text sample of Pedestrian_1, and Image_2 belongs to the image sample of the pedestrian with ID 2 (named Pedestrian_2), then Text_1 and Image_2 constitute a negative sample pair.

[0067] The above-mentioned "one attribute" is explained here. One attribute is used to describe the characteristics of a part of a pedestrian. For example, one attribute Property_1 can be used to describe the upper body clothing of the pedestrian, another attribute Property_2 can be used to describe the head clothing of the pedestrian, and another attribute Property_3 can be used to describe the foot clothing of the pedestrian.

[0068] For the positive sample pairs in the second positive and negative sample set, for example, the fine-grained text feature Text_1_1 is the fine-grained text feature corresponding to Property_1 of Pedestrian_1, and Image_1_1 is the fine-grained visual feature corresponding to Property_1 of Pedestrian_1, then Text_1_1 and Image_1_1 constitute a positive sample pair.

[0069] For the above-mentioned formation of a negative sample pair of fine-grained text features and fine-grained visual features belonging to different pedestrians, an exemplary explanation is given here. For example, Text_1_1 is the fine-grained text feature corresponding to Property_1 of Pedestrian_1, and Image_2_1 is the fine-grained visual feature corresponding to Property_1 of Pedestrian_2. Then Text_1_1 and Image_2_1 are the fine-grained text features and fine-grained visual features of different pedestrians, and Text_1_1 and Image_2_1 form a negative sample pair. For another example, Text_1_1 is the fine-grained text feature corresponding to Property_1 of Pedestrian_1, and Image_2_2 is the fine-grained visual feature corresponding to Property_2 of Pedestrian_2. Then Text_1_1 and Image_2_2 are the fine-grained text features and fine-grained visual features of different pedestrians, and Text_1_1 and Image_2_2 form a negative sample pair.

[0070] For the above-mentioned formation of a negative sample pair of fine-grained text features and fine-grained visual features belonging to different types of attributes, an exemplary explanation is given here. For example, Text_1_1 is the fine-grained text feature corresponding to Property_1 of Pedestrian_1, and Image_1_2 is the fine-grained visual feature corresponding to Property_2 of Pedestrian_1. Then Text_1_1 and Image_1_2 are fine-grained text features and fine-grained visual features of different types of attributes, and Text_1_1 and Image_1_2 form a negative sample pair. For another example, Text_1_1 is the fine-grained text feature corresponding to Property_1 of Pedestrian_1, and Image_2_2 is the fine-grained visual feature corresponding to Property_2 of Pedestrian_2. Then Text_1_1 and Image_2_2 are fine-grained text features and fine-grained visual features of different types of attributes, and Text_1_1 and Image_2_2 form a negative sample pair.

[0071] Figure 2 is a schematic diagram of global feature alignment and fine-grained feature alignment provided in an embodiment of the present application, such as Figure 2 As shown, it can be understood that the first positive and negative sample sets are constructed by combining text samples and image samples belonging to the same pedestrian into positive sample pairs and combining image samples and text samples belonging to different pedestrians into negative sample pairs. Using the first positive and negative sample sets to perform bilateral contrastive learning training on the cross-modal pedestrian re-identification model can enable the model to learn aligned global visual representations and global text representations.

[0072] The alignment of global features can make the global visual representation and the global textual representation as similar as possible in a unified feature space, but it ignores the capture and modeling of fine-grained features, resulting in the global features containing a single piece of information and insufficient robustness. In order to overcome this defect, this application proposes a fine-grained feature alignment scheme, which enables the model to learn global representations while modeling finer-grained local features. Specifically, Figure 2 As shown, the second positive and negative sample sets are constructed by combining fine-grained text features and fine-grained visual features of the same attribute belonging to the same pedestrian into positive sample pairs, and combining fine-grained text features and fine-grained visual features of different attributes belonging to the same pedestrian into negative sample pairs. The cross-modal pedestrian re-identification model is trained by bilateral contrastive learning using the second positive and negative sample sets, so that the model can learn aligned fine-grained visual representations and text representations.

[0073] Therefore, the cross-modal person re-identification model is trained by bilateral contrastive learning using the first set of positive and negative samples and the second set of positive and negative samples, so that the model can model more fine-grained local features while taking into account the learning of global representation, which can effectively improve the accuracy of the model and the robustness of the representation.

[0074] In addition, the existing global feature alignment scheme only considers the current image and the corresponding text as positive sample pairs, and other texts as negative sample pairs. However, for the pedestrian dataset, different images and different text descriptions of the same pedestrian can be combined into positive sample pairs. It can be seen that the existing global feature alignment method cannot adapt to the characteristics of the pedestrian dataset, thus affecting the learning of pedestrian representation. In order to overcome this defect, the first positive and negative sample set is constructed by combining text samples and image samples belonging to the same pedestrian into positive sample pairs and image samples and text samples belonging to different pedestrians into negative sample pairs. Through the global feature alignment method that is more in line with the characteristics of the pedestrian dataset, the selection of positive sample pairs is more reasonable and effective, making the training of the model more efficient and accurate.

[0075] Exemplarily, image samples and text samples in a batch can be matched according to pedestrian ID (one pedestrian corresponds to one pedestrian ID). Image samples and text samples belonging to the same pedestrian ID constitute positive sample pairs, and image samples and text samples of different pedestrian IDs constitute negative sample pairs, so as to construct a first positive and negative sample set. Then, these positive and negative sample pairs (positive sample pairs and negative sample pairs in the first positive and negative sample set) are used for comparative learning. By using a global feature alignment method related to pedestrian ID that is more in line with the characteristics of the pedestrian dataset, the selection of positive sample pairs is more reasonable and effective, thereby making the training of the model more efficient and accurate.

[0076] In one possible implementation, the cross-modal person re-identification model includes a text encoder, a visual encoder and a fusion encoder; the text encoder is used to extract a text feature vector corresponding to a text sample; the visual encoder is used to extract a visual feature vector corresponding to an image sample; the fusion encoder is used to perform feature fusion between different modalities on the text feature vector output by the text encoder and the visual feature vector output by the visual encoder, and output a fused text feature vector and a fused visual feature vector.

[0077] Figure 3 Schematic diagram of the visual block vector and text word vector extraction and the mask operation of the corresponding modality provided by the embodiment of the present application, such as Figure 3 As shown, in a possible implementation, fine-grained text features are obtained through the following steps: based on the text sample, word positions are obtained through text attribute positioning; based on the word position, fine-grained text features are extracted from the text feature vector output by the fusion encoder.

[0078] It is understandable that the existing text mask reconstruction method performs random masking and replacement operations on text samples. This method does not take into account the word part of speech and grammatical structure, which is prone to information leakage and leads to insufficient model training. In order to overcome this shortcoming, the present application uses attribute masking operations, which are specifically done as follows: first, use a part-of-speech analysis tool to perform contextual part-of-speech analysis on all words in the text sample, then perform a comprehensive masking operation on the noun phrases therein, and finally require the text encoder and decoder to recover the masked noun phrases from the masked text samples during the training phase. Through the above-mentioned attribute mask reconstruction task of pedestrian text representation, the cross-modal pedestrian re-identification model can learn richer pedestrian fine-grained text representations.

[0079] like Figure 3 As shown, in a possible implementation, fine-grained visual features are obtained through the following steps: based on image samples, a bounding box for describing the position of the object is obtained through visual semantic positioning; based on the bounding box, fine-grained visual features are extracted from the visual feature vector output by the fusion encoder.

[0080] It is understandable that the existing visual representation mask reconstruction task is an effective method for learning visual representation, but due to the redundancy of image information, the random mask operation of the image cannot completely guarantee the covering of effective information, and will also cause information leakage, resulting in insufficient model training. In order to overcome this shortcoming, the present application uses semantic mask operations, and the specific steps are as follows: first, the noun phrase in one implementation method above is used as a descriptor, and the visual semantic positioning (such as GroundingDINO) is input to find the bounding box of the object corresponding to the noun phrase in the image, and then the object in the bounding box is masked as a whole. Finally, in the training stage, the visual encoder and decoder are required to recover the masked object part from the masked image sample. Through the above-mentioned semantic mask reconstruction task of pedestrian visual representation, the cross-modal pedestrian re-identification model can learn richer fine-grained visual representations of pedestrians.

[0081] Figure 4 is a schematic diagram of text attribute mask reconstruction based on noun phrases provided in an embodiment of the present application, such as Figure 4 As shown, in a possible implementation, after performing bilateral contrastive learning training on the cross-modal person re-identification model based on the first positive and negative sample set and the second positive and negative sample set, it also includes:

[0082] Based on the word vector corresponding to the text sample and the word vector corresponding to each noun phrase in the text sample, the part of the word vector corresponding to the text sample belonging to the noun phrase is replaced with the mask word vector to obtain the masked text word vector;

[0083] Through the text encoder, the masked text word vector is converted into a masked text feature vector;

[0084] Input the visual feature vector without mask and the text feature vector with mask to the fusion encoder to obtain the fused text feature vector with mask;

[0085] Decoding the fused text feature vector with mask through a text decoder to obtain a first decoding result;

[0086] Based on the first decoding result and the fused text feature vector without mask, a first loss is calculated to obtain a first loss value;

[0087] Based on the first loss value, model parameters of the cross-modal person re-identification model and model parameters of the text decoder are adjusted.

[0088] In a possible implementation, calculating the first loss is specifically calculating a negative log-likelihood loss.

[0089] Figure 5 is a schematic diagram of image semantic mask reconstruction based on object position provided by an embodiment of the present application, such as Figure 5 As shown, in a possible implementation, after performing bilateral contrastive learning training on the cross-modal person re-identification model based on the first positive and negative sample set and the second positive and negative sample set, it also includes:

[0090] Based on the block vector of the image sample, the block vector in the bounding box for calculating the visual semantic positioning is replaced with the mask block vector to obtain the masked visual block vector;

[0091] Through the visual encoder, the masked visual block vector is converted into a visual feature vector with mask;

[0092] Input the visual feature vector with mask and the text feature vector without mask to the fusion encoder to obtain the fused visual feature vector with mask;

[0093] Decoding the fused visual feature vector with the mask through a visual decoder to obtain a second decoding result;

[0094] Calculate a second loss based on the second decoding result and the fused visual feature vector without the mask, and obtain a second loss value;

[0095] Based on the second loss value, the model parameters of the cross-modal person re-identification model and the model parameters of the visual decoder are adjusted.

[0096] Further, in a possible implementation manner, calculating the second loss is specifically calculating the KL divergence loss.

[0097] It is understandable that this application can use multimodal pedestrian attribute reconstruction and multimodal pedestrian representation alignment to achieve better pedestrian representation learning. Through multi-level alignment of global features and fine-grained features, and text / visual bimodal reconstruction of fine-grained features, this application proposes an effective end-to-end algorithm for cross-modal pedestrian re-identification tasks, which can fully utilize the abstract relationships and attributes of text modalities to establish better pedestrian visual representations.

[0098] The cross-modal pedestrian re-identification method provided by this application is explained below with reference to an example.

[0099] First, let’s introduce the overall solution of cross-modal person re-identification. The overall solution consists of four parts: (1) extraction of textual and visual features of pedestrians; (2) global feature alignment related to pedestrian ID and fine-grained feature alignment of pedestrians; (3) reconstruction of text attribute masks based on noun phrases; and (4) reconstruction of image semantic masks based on object positions. These four parts will be explained one by one below.

[0100] The features extracted by the model in the first part provide data support for the subsequent parts. The model mainly consists of three parts: the text encoder (such as BERT), the visual encoder (such as ViT), and the fusion encoder (such as Transformer). In the first step, the word segmenter of the text encoder (the input layer of the text encoder) is used to decompose the text and map it into independent word vectors, and then the word vectors are sent to the subsequent network of the text encoder to obtain the text feature vector; the visual embedding layer of the visual encoder (the input layer of the visual encoder) is used to decompose and map the image into independent block vectors, and then the block vectors are sent to the subsequent network of the visual encoder to obtain the visual feature vector. In the second step, the text feature vector and the visual feature vector are sent to the fusion encoder together to obtain the text feature vector and the visual feature vector after modal interaction and fusion. Among them, the feature vector representing the end of the sentence in the text feature vector is designated as the text global feature, and the category feature vector added in the visual embedding layer in the visual feature vector is the visual global feature.

[0101] The second part is to learn and align the visual representation and textual representation of pedestrians. For global features, pedestrian IDs are used to construct global positive and negative sample pairs, and bilateral contrastive learning from text to image and image to text is performed, so that the model can learn aligned visual global representations and text global representations; for fine-grained features, fine-grained positive and negative sample pairs are constructed through text attribute localization and visual semantic localization, so that the model can align visual fine-grained representations and text fine-grained representations. The alignment of global representations is described below. For image and text sample pairs in a batch, the sample pairs are first grouped using the pedestrian ID of the sample pairs. Sample pairs with the same pedestrian ID are assigned to one group. Then, images and texts in the same group form positive sample pairs, and images and texts in different groups form negative sample pairs. Under the loss of contrastive learning, the distance between the global visual features and global text features in the positive sample pairs will be shortened, and the distance between the global visual features and global text features in the negative sample pairs will be extended. Finally, the network can learn aligned global visual representations and global text representations. The alignment of fine-grained representations is described below. For each pair of image-text sample pairs in a batch, first use text attribute localization (such as NLTK) to extract the word positions representing fine-grained text representations, then extract fine-grained text features in the text feature vector based on these position coordinates, then use visual semantic localization (such as GroundingDINO) to extract the bounding box positions representing fine-grained visual representations, and then use the bounding box to extract fine-grained visual features in the visual feature vector, where fine-grained features of the same attribute belonging to the same pedestrian constitute positive sample pairs, and fine-grained features belonging to the same pedestrian but different attributes constitute negative sample pairs. Contrastive learning will bring the fine-grained features in the positive sample pairs closer and the fine-grained features in the negative sample pairs farther apart, and finally the network will be able to learn aligned fine-grained visual representations and text representations.

[0102] The third part is to learn more accurate text representation using the text attribute mask reconstruction task. First, the text sample is input into the word segmenter of the text encoder to obtain the word vector of the text sample, and then the corresponding noun phrase is also input into the same word segmenter to obtain the word vector of the noun phrase. Then, the part of the text sample word vector belonging to the noun phrase is replaced with the mask word vector, so as to achieve the purpose of masking the entire noun phrase. Then, the masked text word vector is sent to the text encoder to obtain the masked text feature vector, and then the masked text feature vector and the complete visual feature vector (that is, the visual feature vector without mask) are sent to the fusion encoder to obtain the masked fused text feature vector (that is, the fused text feature vector with mask). Finally, a lightweight text decoder (lightweight means that the decoder has fewer network layers, such as one layer) is used to decode the masked fused text feature vector, and the negative log-likelihood loss is calculated with the complete fused text feature vector (that is, the fused text feature vector without mask) to reconstruct the masked text part.

[0103] The fourth part is to learn a more accurate visual representation using the visual semantic mask reconstruction task. First, the image sample is input into the visual embedding layer of the visual encoder to obtain the block vector of the image sample, and then the absolute position coordinates of the semantically localized bounding box in all block vectors are calculated. Then the block vector in the bounding box is replaced with a learnable mask block vector, so as to achieve the purpose of masking the visual semantic part as a whole. Then the masked visual block vector is sent to the subsequent network of the visual encoder to obtain the masked visual feature vector, and then the masked visual feature vector and the complete text feature vector (that is, the text feature vector without mask) are sent to the fusion encoder to obtain the masked fused visual feature vector (that is, the fused visual feature vector with mask). Finally, a lightweight visual decoder is used to decode the masked fused visual feature vector, and the KL divergence loss is calculated with the complete fused visual feature vector (that is, the fused visual feature vector without mask) to reconstruct the masked visual part. KL divergence loss (Kullback-Leibler divergence loss) is a loss function used to measure the difference between two probability distributions. In machine learning, KL divergence is often used to measure the difference between the distribution predicted by the model and the true distribution as a loss function to optimize the parameters of the model.

[0104] The cross-modal pedestrian re-identification device provided in the present application is described below. The cross-modal pedestrian re-identification device described below and the cross-modal pedestrian re-identification method described above can be referenced to each other.

[0105] Figure 6 is a schematic diagram of the structure of a cross-modal pedestrian re-identification device provided in an embodiment of the present application, such as Figure 6 As shown, the device includes: a positive and negative sample acquisition module 10 and a training module 20. Among them:

[0106] A positive and negative sample acquisition module 10, used to acquire a first positive and negative sample set and a second positive and negative sample set;

[0107] A training module 20, configured to perform bilateral contrastive learning training on a cross-modal person re-identification model based on the first positive and negative sample set and the second positive and negative sample set;

[0108] The first positive and negative sample set is constructed by combining text samples and image samples belonging to the same pedestrian into positive sample pairs and combining image samples and text samples belonging to different pedestrians into negative sample pairs;

[0109] The second positive and negative sample set is constructed by combining fine-grained text features and fine-grained visual features of the same attribute of the same pedestrian into positive sample pairs, combining fine-grained text features and fine-grained visual features belonging to different pedestrians into negative sample pairs, and combining fine-grained text features and fine-grained visual features belonging to different attributes into negative sample pairs.

[0110] It can be understood that the detailed functional implementation of each of the above-mentioned units / modules can be found in the introduction of the aforementioned method embodiment, and will not be repeated here.

[0111] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0112] Based on the method in the above embodiment, an embodiment of the present application provides an electronic device, Figure 7 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 7 As shown, the electronic device may include: a processor (Processor) 810, a communication interface (CommunicationsInterface) 820, a memory (Memory) 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logic instructions in the memory 830 to execute the method in the above embodiment.

[0113] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0114] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0115] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0116] It is understandable that the processor in the embodiment of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0117] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0118] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0119] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0120] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A cross-modal person re-identification method, characterized in that: include: Obtain a first positive and negative sample set and a second positive and negative sample set; Based on the first positive and negative sample set and the second positive and negative sample set, performing bilateral contrastive learning training on a cross-modal person re-identification model; The first positive and negative sample set is constructed by forming a positive sample pair with a text sample and an image sample belonging to the same pedestrian, and forming a negative sample pair with an image sample and a text sample belonging to different pedestrians; The second positive and negative sample sets are constructed by combining fine-grained text features and fine-grained visual features of the same attribute of the same pedestrian into positive sample pairs, combining fine-grained text features and fine-grained visual features belonging to different pedestrians into negative sample pairs, and combining fine-grained text features and fine-grained visual features belonging to different attributes into negative sample pairs.

2. The cross-modal person re-identification method according to claim 1, characterized in that: The cross-modal person re-identification model includes: a text encoder, a visual encoder and a fusion encoder; The text encoder is used to extract a text feature vector corresponding to a text sample; The visual encoder is used to extract the visual feature vector corresponding to the image sample; The fusion encoder is used to perform feature fusion between different modalities on the text feature vector output by the text encoder and the visual feature vector output by the visual encoder, and output the fused text feature vector and the fused visual feature vector.

3. The cross-modal person re-identification method according to claim 2, characterized in that: The fine-grained text features are obtained through the following steps: Based on the text sample, the word position is obtained by locating the text attributes; Based on the word positions, fine-grained text features are extracted from the text feature vector output by the fusion encoder.

4. The cross-modal person re-identification method according to claim 2, characterized in that: The fine-grained visual features are obtained by the following steps: Based on the image samples, the bounding box used to describe the location of the object is obtained through visual semantic positioning; Based on the bounding box, fine-grained visual features are extracted from the visual feature vector output by the fusion encoder.

5. The cross-modal person re-identification method according to claim 2, characterized in that: After performing bilateral contrastive learning training on the cross-modal person re-identification model based on the first positive and negative sample set and the second positive and negative sample set, the method further includes: Based on the word vector corresponding to the text sample and the word vector corresponding to each noun phrase in the text sample, the part of the word vector corresponding to the text sample belonging to the noun phrase is replaced with the mask word vector to obtain the masked text word vector; Through the text encoder, the masked text word vector is converted into a masked text feature vector; Input the visual feature vector without mask and the text feature vector with mask to the fusion encoder to obtain the fused text feature vector with mask; Decoding the fused text feature vector with mask through a text decoder to obtain a first decoding result; Based on the first decoding result and the fused text feature vector without mask, a first loss is calculated to obtain a first loss value; Based on the first loss value, model parameters of the cross-modal person re-identification model and model parameters of the text decoder are adjusted.

6. The cross-modal person re-identification method according to claim 5, characterized in that: The calculating the first loss is specifically calculating the negative log-likelihood loss.

7. The cross-modal person re-identification method according to claim 2, characterized in that: After performing bilateral contrastive learning training on the cross-modal person re-identification model based on the first positive and negative sample set and the second positive and negative sample set, the method further includes: Based on the block vector of the image sample, the block vector in the bounding box for calculating the visual semantic positioning is replaced with the mask block vector to obtain the masked visual block vector; Through the visual encoder, the masked visual block vector is converted into a visual feature vector with mask; Input the visual feature vector with mask and the text feature vector without mask to the fusion encoder to obtain the fused visual feature vector with mask; Decoding the fused visual feature vector with the mask through a visual decoder to obtain a second decoding result; Calculate a second loss based on the second decoding result and the fused visual feature vector without the mask, and obtain a second loss value; Based on the second loss value, the model parameters of the cross-modal person re-identification model and the model parameters of the visual decoder are adjusted.

8. The cross-modal person re-identification method according to claim 7, characterized in that: The calculating the second loss is specifically calculating the KL divergence loss.

9. A cross-modal person re-identification device, characterized in that: include: A positive and negative sample acquisition module, used to acquire a first positive and negative sample set and a second positive and negative sample set; A training module, configured to perform bilateral contrastive learning training on a cross-modal person re-identification model based on the first positive and negative sample set and the second positive and negative sample set; The first positive and negative sample set is constructed by forming a positive sample pair with a text sample and an image sample belonging to the same pedestrian, and forming a negative sample pair with an image sample and a text sample belonging to different pedestrians; The second positive and negative sample sets are constructed by combining fine-grained text features and fine-grained visual features of the same attribute of the same pedestrian into positive sample pairs, combining fine-grained text features and fine-grained visual features belonging to different pedestrians into negative sample pairs, and combining fine-grained text features and fine-grained visual features belonging to different attributes into negative sample pairs.

10. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 8.

Images