A pedestrian re-identification method based on text-enhanced visual language model

Abstract

The application discloses a pedestrian re-identification method based on a text-enhanced visual language model, and the method comprises the following steps: constructing a visual feature extraction module to extract visual features of an RGB pedestrian image; constructing a text feature extraction module to extract text features of a pedestrian text description; constructing a text abstract module to generate abstract features of the pedestrian text description; constructing a loss calculation module to obtain an optimal pedestrian re-identification model; and constructing a pedestrian retrieval module to obtain a pedestrian image retrieval result by using the optimal pedestrian re-identification model. The application effectively generates abstract features of the pedestrian text description with better discrimination by introducing the text abstract module, and further improves the accuracy of the pedestrian re-identification by proposing an abstract feature constraint loss function for fully utilizing the abstract features of the pedestrian text description.

Description

A Person Re-identification Method Based on Text-Enhanced Visual Language Model Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for pedestrian re-identification based on a text-enhanced visual language model. Background Technology

[0002] In recent years, text-to-image person re-identification has gradually become a research hotspot, which uses textual descriptions of pedestrians as input to retrieve pedestrian images. Compared to a single visual modality, the advantage of introducing a textual modality lies in leveraging the complementarity and robustness of modalities—textual descriptions can both fill in the information gaps in images or the lack of information in low-quality images, and enhance feature representations through semantic information. Text-to-image person re-identification methods are mainly divided into two categories. One type of method uses image encoders and text encoders, each pre-trained on different modal data. Although this type of method has strong feature extraction capabilities within its respective modality, its cross-modal retrieval capability is weak. The other type of method uses a pre-trained visual language model as the backbone network. Since the visual language model mainly adopts a contrastive learning strategy, it simultaneously pre-trains the image encoder and text encoder using multimodal data, thus possessing strong cross-modal alignment capabilities. Among them, the visual language model pre-trained using contrastive language and image models can improve the performance of text-to-image person re-identification by aligning visual features with textual features for cross-modal learning.

[0003] Recently, due to the superior performance of visual language models in various tasks, Han et al. proposed a cross-modal momentum contrastive learning framework using a contrastive language-image pre-trained model as the backbone network to handle text-to-image person re-identification tasks. Li et al. proposed a two-stage contrastive language-image pre-training-person re-identification model. In the first stage, both the text encoder and image encoder are frozen, and only learnable tokens from the text description are trained. In the second stage, both the text encoder and the tokens from the text description are frozen, and only the image encoder is trained, achieving image-text alignment without specific text labels. Jiang et al. proposed a cross-modal implicit reasoning and alignment model. This model performs global alignment through cross-modal implicit local relation learning, thereby improving the accuracy of text-visual matching in person re-identification.

[0004] The above methods rely solely on text features extracted by the text encoder as the final semantic representation, making it difficult to fully utilize the key information in the text description, which in turn affects the model's performance in cross-modal matching tasks. Summary of the Invention

[0005] The purpose of this invention is to solve the problem that pedestrian re-identification methods based on visual language models rely solely on text features extracted by text encoders, making it difficult to fully utilize key information in text descriptions. To this end, this invention provides a pedestrian re-identification method based on a text-enhanced visual language model.

[0006] Based on the above technical objectives, this invention provides a person re-identification method based on a text-enhanced visual language model, the method comprising the following steps:

[0007] Step S1: Construct a visual feature extraction module and use the visual feature extraction module to extract visual features of RGB pedestrian images;

[0008] Step S2: Construct a text feature extraction module and use the text feature extraction module to extract text features of pedestrian text descriptions;

[0009] Step S3: Construct a text summarization module. In the text summarization module, a learnable text summarization vector is randomly initialized. The learnable text summarization vector is fused with the text features of the pedestrian text description using a cross-attention mechanism to generate a summary feature of the pedestrian text description.

[0010] Step S4: Construct a loss calculation module. Input the visual features of the RGB pedestrian image, the text features of the pedestrian text description, and the summary features of the pedestrian text description into the loss calculation module. Use the obtained loss value to optimize the pedestrian re-identification model composed of the visual feature extraction module, the text feature extraction module, the text summarization module, and the loss calculation module to obtain the optimal pedestrian re-identification model.

[0011] Step S5: Construct a pedestrian retrieval module. In the pedestrian retrieval module, the visual features of the RGB pedestrian image to be retrieved and the text features of the text description of the pedestrian to be retrieved are extracted using the optimal pedestrian re-identification model. Then, the similarity between the two features is calculated, and the obtained similarity is sorted from high to low to obtain the pedestrian image retrieval results.

[0012] In one embodiment, the visual feature extraction module includes an image embedding layer and a first Transformer module, wherein the image embedding layer is used to convert an RGB pedestrian image into an image token embedding sequence; the first Transformer module includes... A Transformer block with the same structure is used to extract the visual features of the RGB pedestrian image from the image token embedding sequence.

[0013] In one embodiment, step S1 includes the following steps:

[0014] Step S11: The RGB pedestrian image is segmented using the image embedding layer and converted into an image token embedding sequence containing location information.

[0015] Step S12, utilizing the first Transformer module A Transformer block with the same structure processes the image token embedding sequence to obtain the visual features of the RGB pedestrian image.

[0016] In one embodiment, step S11 includes the following steps:

[0017] Step S111: Divide the RGB pedestrian image into several non-overlapping image blocks of fixed size;

[0018] Step S112: Use the linear mapping layer in the image embedding layer to flatten and map each image patch into an image token embedding, and obtain the image token embedding sequence.

[0019] Step S113: Randomly initialize and obtain the image position embedding of the RGB pedestrian image;

[0020] Step S114: A learnable [CLS] token embedding is concatenated at the beginning of the image token embedding sequence. The image position embedding is added element by element to the image token embedding sequence to obtain the image token embedding sequence containing position information of the RGB pedestrian image, which is used as the input of the first Transformer module.

[0021] In one embodiment, in step S12:

[0022] No. Output sequence of Transformer blocks for:

[0023] ,

[0024] in, , , , , Indicates the first The output sequence of a Transformer block, when hour, , Represents an image token embedding sequence. This indicates the layer normalization operation used in the visual feature extraction module. This indicates a multi-head self-attention mechanism. Indicates the first The output sequence of the multi-head self-attention mechanism in each Transformer block. This represents the multilayer perceptron in the first Transformer module;

[0025] Visual features of RGB pedestrian images It is calculated by the following formula:

[0026] ,in The portion of the output sequence of the last block that corresponds to the learnable token embedding.

[0027] In one embodiment, the text feature extraction module includes a text embedding layer and a second Transformer module, wherein the text embedding layer is used to convert pedestrian text descriptions into text token embedding sequences; the second Transformer module includes... A Transformer block with the same structure is used to extract text features of the pedestrian text description from the text token embedding sequence.

[0028] In one embodiment, step S2 includes the following steps:

[0029] Step S21: The pedestrian text description is segmented using the text embedding layer and converted into a text token embedding sequence containing location information.

[0030] Step S22, utilizing the second Transformer module A Transformer block with the same structure processes the text token embedding sequence to obtain the text features of the pedestrian text description.

[0031] In one embodiment, step S21 includes the following steps:

[0032] Step S211: Divide the pedestrian text description into several sub-words;

[0033] Step S212: Use the linear mapping layer in the text embedding layer to map each subword to a text token embedding to obtain a text token embedding sequence;

[0034] Step S213: Randomly initialize the text position embedding of the pedestrian text description;

[0035] Step S214: At the beginning and end of the text token embedding sequence, a learnable [SOS] token embedding and a [EOS] token embedding are respectively concatenated. The text location embedding is added element by element to the text token embedding sequence to obtain the text token embedding sequence containing location information of the pedestrian text description, which is used as the input of the second Transformer module.

[0036] In one embodiment, in step S22:

[0037] No. Output sequence of Transformer blocks for:

[0038] ,

[0039] ,

[0040] in, , , , , Indicates the first The output sequence of a Transformer block, when hour, , This indicates a sequence of text token embeddings. This indicates the layer normalization operation used in the text feature extraction module. This indicates a multi-head self-attention mechanism for the mask. Indicates the first The output sequence of the masked multi-head self-attention mechanism in each Transformer block. This represents the multilayer perceptron in the second Transformer module;

[0041] Text features of pedestrian text description It is calculated by the following formula:

[0042] ,

[0043] in, The learnable [EOS] token embedding in the output sequence of the last Transformer block. The corresponding part.

[0044] In one embodiment, step S3 includes the following steps:

[0045] Step S31: Randomly initialize learnable text summary vectors;

[0046] Step S32: The learnable text summary vector is used as the query vector, and the text features of the pedestrian text description are used as the key vector and value vector. The key vector, value vector and query vector are interacted using a cross attention mechanism to obtain the correlation between the learnable text summary vector and the text features of the pedestrian text description, thereby obtaining the summary features of the pedestrian text description.

[0047] The beneficial effects of this invention are as follows: By introducing a text summarization module, this invention utilizes learnable text summarization vectors and text features of pedestrian text descriptions to perform cross-attention interaction, generating more discriminative summary features of pedestrian text descriptions, thereby fully mining the key semantic information in the text descriptions; and by designing a summary feature constraint loss function, the summary features of pedestrian text descriptions are weighted and fused with the text features of pedestrian text descriptions and aligned with the visual features of RGB pedestrian images, thereby strengthening the feature matching ability of visual and text modalities in the shared semantic space, solving the problem of insufficient semantic representation caused by existing methods relying only on original text features, and significantly improving the accuracy and robustness of text-to-image pedestrian re-identification.

[0048] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the description, claims, and drawings. Attached Figure Description

[0049] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0050] Figure 1 is a flowchart of a pedestrian re-identification method based on a text-enhanced visual language model according to an embodiment of the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0052] Figure 1 is a flowchart of a person re-identification method based on a text-enhanced visual language model according to an embodiment of the present invention. The following uses Figure 1 as an example to illustrate some specific implementation processes of the present invention. As shown in Figure 1, the person re-identification method based on a text-enhanced visual language model includes the following steps:

[0053] Step S1: Construct a visual feature extraction module and use the visual feature extraction module to extract visual features of RGB pedestrian images;

[0054] In one embodiment of the present invention, the visual feature extraction module is constructed based on an image encoder using a contrastive language-image pre-trained model. Specifically, the visual feature extraction module includes an image embedding layer and a first Transformer module, wherein the image embedding layer is used to convert RGB pedestrian images into image token embedding sequences; the first Transformer module includes... A Transformer block with the same structure is used to extract the visual features of the RGB pedestrian image from the image token embedding sequence.

[0055] Therefore, the processing of the visual feature extraction module can be represented as follows:

[0056] ,

[0057] in, Represents an RGB pedestrian image. This represents the image embedding layer processing in the visual feature extraction module. This represents the processing of the first Transformer module in the visual feature extraction module.

[0058] In one embodiment of the present invention, to implement the visual feature extraction module, step S1 further includes the following steps:

[0059] Step S11: The RGB pedestrian image is segmented using the image embedding layer and converted into an image token embedding sequence containing location information.

[0060] Step S11 further includes the following steps:

[0061] Step S111: Divide the RGB pedestrian image into several non-overlapping image blocks of fixed size;

[0062] Step S112: Use the linear mapping layer in the image embedding layer to flatten and map each image patch into an image token embedding, and obtain the image token embedding sequence.

[0063] Step S113: Randomly initialize and obtain the image position embedding of the RGB pedestrian image;

[0064] Step S114: A learnable [CLS] token embedding is concatenated at the beginning of the image token embedding sequence. The image position embedding is added element by element to the image token embedding sequence to obtain the image token embedding sequence containing position information of the RGB pedestrian image, which is used as the input of the first Transformer module.

[0065] Therefore, the output of the image embedding layer in step S11 can be expressed as:

[0066] ,

[0067] in, Represents an image token embedding sequence. This indicates a learnable [CLS] token embedding. Indicates image location embedding. This represents the processing of the linear mapping layer in the image embedding layer. Indicates the number of image patches. Represents RGB pedestrian images The Image blocks.

[0068] In one embodiment of the present invention, the height of the RGB pedestrian image... ,Width Number of channels Number of image patches =192, where the side length and height of the image patch are... Linear mapping layer in image embedding layer From the input dimension The output dimension is A fully connected layer implementation with learnable [CLS] token embeddings. Dimensions Image location embedding Dimensions .

[0069] Step S12, utilizing the first Transformer module A Transformer block with the same structure processes the image token embedding sequence to obtain the visual features of the RGB pedestrian image.

[0070] In one embodiment of the present invention, the first Transformer module in the first Transformer module is... The output sequence of a Transformer block can be represented as:

[0071] ,

[0072] in,

[0073] ,

[0074] in, , , , , Indicates the first The output sequence of a Transformer block, when hour, , This indicates the layer normalization operation used in the visual feature extraction module. This indicates the processing of multi-head self-attention mechanisms. Indicates the first The output sequence of the multi-head self-attention mechanism in each Transformer block. This indicates the multilayer perceptron processing in the first Transformer module. Indicates the first The output sequence of a Transformer block.

[0075] So, what are the visual features of RGB pedestrian images? It can be calculated using the following formula:

[0076] ,

[0077] in, The learnable [CLS] token embedding in the output sequence of the last Transformer block The corresponding part.

[0078] In one embodiment of the present invention, the number of Transformer blocks in the visual feature extraction module... The dimension of the hidden layer in each Transformer block is Multi-head self-attention mechanisms include There are 1 attention head, and each attention head has 1 dimension. A multilayer perceptron includes an input dimension of... The output dimension is A fully connected layer and an input dimension of The output dimension is The two fully connected layers are connected by the GELU activation function.

[0079] Step S2: Construct a text feature extraction module and use the text feature extraction module to extract text features of pedestrian text descriptions;

[0080] In one embodiment of the present invention, the text feature extraction module is constructed based on a text encoder using a contrastive language-image pre-trained model. Specifically, the text feature extraction module includes a text embedding layer and a second Transformer module, wherein the text embedding layer is used to convert pedestrian text descriptions into text token embedding sequences; the second Transformer module includes... A Transformer block with the same structure is used to extract text features of the pedestrian text description from the text token embedding sequence.

[0081] Therefore, the processing of the text feature extraction module can be represented as follows:

[0082] ,

[0083] in, This represents a pedestrian text description. This indicates the processing of the text embedding layer in the text feature extraction module. This indicates the processing of the second Transformer module in the text feature extraction module.

[0084] In one embodiment of the present invention, to implement the text feature extraction module, step S2 further includes the following steps:

[0085] Step S21: The pedestrian text description is segmented using the text embedding layer and converted into a text token embedding sequence containing location information.

[0086] Step S21 further includes the following steps:

[0087] Step S211: Divide the pedestrian text description into several sub-words;

[0088] Step S212: Use the linear mapping layer in the text embedding layer to map each subword to a text token embedding to obtain a text token embedding sequence;

[0089] Step S213: Randomly initialize the text position embedding of the pedestrian text description;

[0090] Step S214: At the beginning and end of the text token embedding sequence, a learnable [SOS] token embedding and a [EOS] token embedding are respectively concatenated. The text location embedding is added element by element to the text token embedding sequence to obtain the text token embedding sequence containing location information of the pedestrian text description, which is used as the input of the second Transformer module.

[0091] Therefore, the output of the text embedding layer in step S21 can be expressed as:

[0092] ,

[0093] in, This indicates a sequence of text token embeddings. This indicates a learnable [SOS] token embedding. This indicates a learnable [EOS] token embedding. Indicates text position embedding, This represents the processing of the linear mapping layer in the text embedding layer. This indicates the maximum length of the pedestrian text description. Pedestrian text description The Individual words.

[0094] In one embodiment of the present invention, the maximum length of pedestrian text description Linear mapping layer in text embedding layer From the input dimension The output dimension is The fully connected layer implementation, the learnable [SOS] token embedding Dimensions Learnable [EOS] token embedding Dimensions Text position embedding Dimensions .

[0095] Step S22, utilizing the second Transformer module A Transformer block with the same structure processes the text token embedding sequence to obtain the text features of the pedestrian text description.

[0096] In one embodiment of the present invention, the second Transformer module... The output sequence of a Transformer block can be represented as:

[0097] ,

[0098] in,

[0099] ,

[0100] in, , , , , Indicates the first The output sequence of a Transformer block, when hour, , This indicates the layer normalization operation used in the text feature extraction module. This indicates that the mask is processed using a multi-head self-attention mechanism. Indicates the first The output sequence of the masked multi-head self-attention mechanism in each Transformer block. This indicates the multilayer perceptron processing in the second Transformer module. Indicates the first The output sequence of a Transformer block.

[0101] So, the text features of pedestrian text description It can be calculated using the following formula:

[0102] ,

[0103] in, The learnable [EOS] token embedding in the output sequence of the last Transformer block. The corresponding part.

[0104] In one embodiment of the present invention, the number of Transformer blocks in the text feature extraction module... The dimension of the hidden layer in each Transformer block is The masked multi-head self-attention mechanism includes There are 1 attention head, and each attention head has 1 dimension. A multilayer perceptron includes an input dimension of... The output dimension is A fully connected layer and an input dimension of The output dimension is The two fully connected layers are connected by the GELU activation function.

[0105] Step S3: Construct a text summarization module. In the text summarization module, a learnable text summarization vector is randomly initialized. The learnable text summarization vector is fused with the text features of the pedestrian text description using a cross-attention mechanism to generate more discriminative summary features of the pedestrian text description.

[0106] Step S3 further includes the following steps:

[0107] Step S31: Randomly initialize learnable text summary vectors;

[0108] Step S32: The learnable text summary vector is used as the query vector, and the text features of the pedestrian text description are used as the key vector and value vector. The key vector, value vector and query vector are interacted using a cross attention mechanism to obtain the correlation between the learnable text summary vector and the text features of the pedestrian text description, thereby obtaining the summary features of the pedestrian text description.

[0109] Furthermore, after obtaining the correlation between the learnable text summary vector and the text features of the pedestrian text description, residual operations can be used to process it to avoid potential gradient vanishing or exploding problems, and finally obtain the summary features of the pedestrian text description.

[0110] Therefore, the text summarization module processing in step S3 can be represented as, that is, the summarization features of pedestrian text descriptions. It can be obtained from the following formula:

[0111] ,

[0112] ,

[0113] in, Text features representing pedestrian text descriptions Dimensions Represents a learnable text summary vector. express Activation function This indicates the processing of cross-attention mechanism. This indicates the layer normalization operation used in the text summarization module.

[0114] In one embodiment of the present invention, the text features of pedestrian text description Dimensions Learnable text summarization vectors The dimension is The cross-attention mechanism employs a multi-head self-attention structure, including... There are 1 attention head, and each attention head has 1 dimension. .

[0115] Step S4: Construct a loss calculation module. Input the visual features of the RGB pedestrian image, the text features of the pedestrian text description, and the summary features of the pedestrian text description into the loss calculation module. Use the obtained loss value to optimize the pedestrian re-identification model composed of the visual feature extraction module, the text feature extraction module, the text summarization module, and the loss calculation module to obtain the optimal pedestrian re-identification model.

[0116] In one embodiment of the present invention, the loss calculation module can be represented as:

[0117] ,

[0118] in, and The similarity distribution matching loss function is as follows: and summary feature constraint loss function The weighting coefficients, The cross-entropy loss function is:

[0119] ,

[0120] in,

[0121] ,

[0122] ,

[0123] in, It is the cross-entropy loss function of the visual modality. It is the cross-entropy loss function for text modalities. It is the total number of pedestrians in a small batch. It is the number of pedestrians' identities in a small batch. For the first The true label of each sample It is the first Visual features of an RGB pedestrian image of an individual pedestrian. It is the first Textual features of pedestrian text descriptions for individual pedestrians and Visual features of RGB pedestrian images Text features of pedestrian text description Belongs to the The probability of class identity.

[0124] in, Loss function for similarity distribution matching:

[0125] ,

[0126] in,

[0127] ,

[0128] ,

[0129] in, It is a similarity distribution matching loss function from visual modality to text modality. It is a similarity distribution matching loss function from text modality to visual modality. It is the first Visual features of an RGB pedestrian image of an individual pedestrian. It is the first Textual features of pedestrian text descriptions for individual pedestrians It is the first Visual features of an RGB pedestrian image of an individual pedestrian. It is the first Textual features of pedestrian text descriptions for individual pedestrians Represents cosine similarity. It is the temperature coefficient that controls smoothness in the similarity distribution matching loss function.

[0130] in, Loss function constrained by summary features:

[0131] ,

[0132] in, It is the first Summary features of pedestrian text descriptions for each pedestrian These are the weighting coefficients of the text features and the summary features of the pedestrian text description. It is the temperature coefficient that controls smoothness in the summary feature constraint loss function.

[0133] In one embodiment of the present invention, and They are respectively and , No. individual pedestrians and the first Visual features of an RGB pedestrian image and , No. Text features of pedestrian text descriptions for individual pedestrians , No. Summary features of pedestrian text descriptions for individual pedestrians All dimensions The total number of pedestrians in a small batch Weighting coefficient Temperature coefficient controlling smoothness in similarity distribution matching loss function Temperature coefficient controlling smoothness in the summary feature constraint loss function All .

[0134] Step S5: Construct a pedestrian retrieval module. In the pedestrian retrieval module, the visual features of the RGB pedestrian image to be retrieved and the text features of the text description of the pedestrian to be retrieved are extracted using the optimal pedestrian re-identification model. Then, the similarity between the two features is calculated, and the obtained similarity is sorted from high to low to obtain the pedestrian image retrieval results. For example, the pedestrians corresponding to the features with the highest similarity can be identified as the same pedestrian.

[0135] Furthermore, the processing of the pedestrian retrieval module can be expressed as follows:

[0136] ,

[0137] ,

[0138] in, The text features of the pedestrian text description to be retrieved The search result is the first one. Visual features of an RGB pedestrian image of an individual pedestrian. This is the total number of pedestrians to be searched. For sorting functions, Sort the results of the pedestrian search.

[0139] In one embodiment of the present invention, the text features of the pedestrian text description to be retrieved are... and the first to be searched Visual features of an RGB pedestrian image All dimensions .

[0140] Using publicly available online text-to-image person re-identification databases as test subjects, such as the CUHK-PEDES database, the present invention achieved a Rank-1 accuracy and mean average precision (mAP) of [amount missing]. and The pedestrian re-identification method of this invention effectively generates more discriminative summary features of pedestrian text descriptions by introducing a text summarization module. To fully utilize the summary features of the pedestrian text descriptions, a summary feature constraint loss function is proposed, which greatly improves the accuracy of pedestrian re-identification, thus demonstrating the effectiveness of the method of this invention.

[0141] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A person re-identification method based on a text-enhanced visual language model, characterized in that: The method includes the following steps: Step S1, constructing a visual feature extraction module to extract visual features of RGB pedestrian images; Step S2, constructing a text feature extraction module to extract text features of pedestrian text descriptions; Step S3, constructing a text summarization module, in which learnable text summarization vectors are randomly initialized, and a cross-attention mechanism is used to fuse the learnable text summarization vectors with the text features of pedestrian text descriptions to generate summary features of pedestrian text descriptions; Step S4, constructing a loss calculation module, inputting the visual features of the RGB pedestrian images, the text features of pedestrian text descriptions, and the summary features of pedestrian text descriptions into the loss calculation module, and using the obtained loss value to optimize the pedestrian re-identification model composed of the visual feature extraction module, text feature extraction module, text summarization module, and loss calculation module to obtain the optimal pedestrian re-identification model; wherein, the loss calculation module is represented as: ,in, and The similarity distribution matching loss function is as follows: and summary feature constraint loss function The weighting coefficients, The cross-entropy loss function consists of the cross-entropy loss function for the visual modality and the cross-entropy loss function for the text modality, along with the similarity distribution matching loss function. It consists of a similarity distribution matching loss function from visual modality to text modality and a similarity distribution matching loss function from text modality to visual modality, plus a summary feature constraint loss function. for: ,in, It is the total number of pedestrians in a small batch. It is the first Visual features of an RGB pedestrian image of an individual pedestrian. It is the first Textual features of pedestrian text descriptions for individual pedestrians It is the first Summary features of pedestrian text descriptions for each pedestrian These are the weighting coefficients of the text features and the summary features of the pedestrian text description. It is the temperature coefficient that controls smoothness in the summary feature constraint loss function; Step S5, construct a pedestrian retrieval module. In the pedestrian retrieval module, the visual features of the RGB pedestrian image to be retrieved and the text features of the text description of the pedestrian to be retrieved are extracted using the optimal pedestrian re-identification model. Then, the similarity between the two features is calculated, and the obtained similarity is sorted from high to low to obtain the pedestrian image retrieval results.

2. The method according to claim 1, characterized in that, The visual feature extraction module includes an image embedding layer and a first Transformer module. The image embedding layer is used to convert RGB pedestrian images into image token embedding sequences. The first Transformer module includes L Transformer blocks with identical structures, used to extract the visual features of the RGB pedestrian images from the image token embedding sequences.

3. The method according to claim 2, characterized in that, Step S1 includes the following steps: Step S11, using the image embedding layer to segment the RGB pedestrian image into blocks and convert it into an image token embedding sequence containing location information; Step S12, using L Transformer blocks with the same structure in the first Transformer module to process the image token embedding sequence to obtain the visual features of the RGB pedestrian image.

4. The method according to claim 3, characterized in that, Step S11 includes the following steps: Step S111, dividing the RGB pedestrian image into several non-overlapping image blocks of fixed size; Step S112, using the linear mapping layer in the image embedding layer to flatten and map each image block into an image token embedding, obtaining an image token embedding sequence; Step S113, randomly initializing to obtain the image position embedding of the RGB pedestrian image; Step S114, concatenating a learnable [CLS] token embedding at the beginning of the image token embedding sequence, adding the image position embedding to the image token embedding sequence element by element, obtaining the image token embedding sequence of the RGB pedestrian image containing position information, which serves as the input of the first Transformer module.

5. The method according to claim 4, characterized in that, In step S12: Output sequence of Transformer blocks for: ， ,in, ， ， ， ， Indicates the first The output sequence of a Transformer block, when hour, ， This represents an image token embedding sequence. This indicates the layer normalization operation used in the visual feature extraction module. This indicates a multi-head self-attention mechanism. Indicates the first The output sequence of the multi-head self-attention mechanism in each Transformer block Represents the multilayer perceptron in the first Transformer module; visual features of RGB pedestrian images. It is calculated by the following formula: ,in, The learnable [CLS] token embedding in the output sequence of the last Transformer block The corresponding part.

6. The method according to claim 1, characterized in that, The text feature extraction module includes a text embedding layer and a second Transformer module. The text embedding layer is used to convert pedestrian text descriptions into text token embedding sequences. The second Transformer module includes L Transformer blocks with identical structures, used to extract text features of the pedestrian text descriptions from the text token embedding sequences.

7. The method according to claim 6, characterized in that, Step S2 includes the following steps: Step S21, using the text embedding layer to perform word segmentation on the pedestrian text description and convert it into a text token embedding sequence containing location information; Step S22, using L Transformer blocks with the same structure in the second Transformer module to process the text token embedding sequence to obtain the text features of the pedestrian text description.

8. The method according to claim 7, characterized in that, Step S21 includes the following steps: Step S211, dividing the pedestrian text description into several sub-words; Step S212, using the linear mapping layer in the text embedding layer to map each sub-word into a text token embedding, obtaining a text token embedding sequence; Step S213, randomly initializing to obtain the text position embedding of the pedestrian text description; Step S214, concatenating the learnable [SOS] token embedding and [EOS] token embedding at the beginning and end of the text token embedding sequence respectively, adding the text position embedding to the text token embedding sequence element by element to obtain the text token embedding sequence of the pedestrian text description containing position information, which serves as the input of the second Transformer module.

9. The method according to claim 8, characterized in that, In step S22: the first Output sequence of Transformer blocks for: ， ,in, ， ， ， ， Indicates the first The output sequence of a Transformer block, when hour, ， This indicates a sequence of text token embeddings. This indicates the layer normalization operation used in the text feature extraction module. This indicates a multi-head self-attention mechanism for the mask. Indicates the first The output sequence of the masked multi-head self-attention mechanism in each Transformer block. This represents the multilayer perceptron in the second Transformer module; text features in pedestrian text description. It is calculated by the following formula: ,in, The learnable [EOS] token embedding in the output sequence of the last Transformer block. The corresponding part.

10. The method according to claim 1, characterized in that, Step S3 includes the following steps: Step S31, randomly initialize a learnable text summary vector; Step S32, use the learnable text summary vector as a query vector, use the text features of the pedestrian text description as key vector and value vector, and use a cross-attention mechanism to interact the key vector, value vector and query vector to obtain the correlation between the learnable text summary vector and the text features of the pedestrian text description, thereby obtaining the summary features of the pedestrian text description.

Images