A text-to-image pedestrian retrieval method based on an auxiliary text alignment paradigm

Abstract

The application discloses a text-to-image pedestrian retrieval method based on an auxiliary text alignment paradigm, which comprises the following steps: generating corresponding auxiliary text based on images and interactive instructions by using a multi-modal large language model; extracting global features and local features of the images by an image encoder, and extracting global features and local features of the text and the auxiliary text by a text encoder; transferring the global feature knowledge of the auxiliary text to the image and text modalities, dynamically fusing the transferred image global features and the transferred text global features based on a hybrid expert knowledge learning module, and obtaining corresponding image final feature representations and text final feature representations; dynamically selecting the most similar negative sample pair for each image and text, using multi-head cross-attention contrast learning to compare the fine-grained differences between the positive and negative sample pairs, realizing cross-modal matching of the text-to-image, and completing pedestrian retrieval. The application can realize more accurate text-to-image pedestrian retrieval.

Description

Technical Field

[0001] This invention belongs to the field of cross-modal pedestrian re-identification technology, and particularly relates to a text-to-image pedestrian retrieval method based on an auxiliary text alignment paradigm. Background Technology

[0002] In the field of intelligent security, text-to-image pedestrian retrieval technology has significant application value. This technology aims to accurately locate target pedestrians from massive amounts of surveillance images using natural language descriptions, and can be widely applied to security monitoring systems in public places such as shopping malls, stations, and streets. Traditional methods of manually reviewing surveillance videos are inefficient and struggle to meet the demands of rapid retrieval of massive amounts of data. In contrast, deep learning-based pedestrian retrieval technology can quickly match target pedestrian images with text descriptions, significantly improving the efficiency and accuracy of security monitoring and providing strong support for tasks such as suspicious person tracking and criminal investigation.

[0003] Currently, the technical solutions for text-to-image pedestrian retrieval are mainly divided into two categories: global feature alignment and local feature alignment.

[0004] Global feature alignment methods typically utilize deep learning models to extract global features from images and text separately, and achieve cross-modal feature alignment through loss functions such as contrastive loss and triplet loss. However, due to insufficient interaction between image and text modalities, this method struggles to achieve fine-grained feature matching, resulting in limited retrieval accuracy in complex scenarios. For example, when different pedestrians wear similar clothing but have differences in local features, relying solely on global feature alignment can easily lead to false positives or false negatives.

[0005] Local feature alignment methods improve retrieval accuracy by decomposing images and text into multiple attribute components to achieve finer cross-modal matching. However, current local-based methods, in the process of fine-grained alignment, mainly focus on the current paired sample, ignoring information from other negative samples in the batch. They lack a global perspective and struggle to effectively focus on subtle but discriminative fine-grained features, resulting in inaccurate retrieval when dealing with similar negative samples.

[0006] In addition, existing public datasets contain paired data of images and corresponding natural language, but the text descriptions in them mostly only provide a rough description of pedestrians. This makes the model only able to establish basic mappings between different modalities and unable to capture deep semantic knowledge, resulting in insufficient reasoning ability of the model when faced with complex sentences or words.

[0007] To address the aforementioned issues, there is an urgent need to propose a text-to-image pedestrian retrieval method based on an auxiliary text alignment paradigm to improve retrieval accuracy in complex scenarios. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention proposes a text-to-image pedestrian retrieval method based on an auxiliary text alignment paradigm, thereby resolving the issues present in the prior art.

[0009] To achieve the above objectives, this invention provides a text-to-image pedestrian retrieval method based on an auxiliary text alignment paradigm, comprising the following steps:

[0010] Acquire pedestrian image and text modal data, and obtain interactive instructions by analyzing the image data;

[0011] Using a multimodal large language model, corresponding auxiliary text is generated based on images and interactive commands;

[0012] Global and local features of the image are extracted using an image encoder, and global and local features of the text and auxiliary text are extracted using a text encoder, respectively.

[0013] By leveraging the feature interactions between auxiliary text and images and text, global feature knowledge of the auxiliary text is transferred to image and text modalities, resulting in transferred global image features and transferred global text features.

[0014] A hybrid expert knowledge learning module is constructed. Based on the hybrid expert knowledge learning module, the transferred global features of the image and the transferred global features of the text are dynamically fused to obtain the corresponding final feature representations of the image and the final feature representations of the text.

[0015] Based on the final feature representations of the image and the final feature representation of the text, the most similar negative sample pair is dynamically selected for each image and text. Multi-head cross-attention is used to compare and learn the fine-grained differences between positive and negative sample pairs, thereby achieving cross-modal matching from text to image and completing pedestrian retrieval.

[0016] Optionally, the process of extracting features from the image based on the image encoder includes:

[0017] Use a pre-trained VIT model as the image encoder;

[0018] The image is divided into several non-overlapping tile sequences. A learnable image tag is added to the tile sequence. The tile sequence with the tag is then fed into the image encoder after position embedding, and the local and global features of the image are output.

[0019] Optionally, the process of feature extraction from the text and auxiliary text based on the text encoder includes:

[0020] Use the BERT model as a text encoder;

[0021] Each word in the text or auxiliary text is mapped to its corresponding word embedding to generate a word embedding sequence. A learnable text tag is added to the word embedding sequence, and then a padding tag is used to pad the word embedding sequence until the word embedding sequence reaches a fixed length. The padded word embedding sequence is then fed into the text encoder along with a positional embedding to output the local and global features of the text or auxiliary text.

[0022] Optionally, by leveraging the feature interactions between auxiliary text and images and text, the global feature knowledge of the auxiliary text is transferred to the image and text modalities to obtain the transferred global image features and the transferred global text features.

[0023] The global features of the image, text, and auxiliary text are projected into a latent subspace to obtain the projected global features of the image, the projected global features of the text, and the projected global features of the auxiliary text.

[0024] The interaction information between the global features of the projected auxiliary text and the global features of the projected image is obtained. Based on the corresponding interaction information, the global features of the projected image are adjusted using a transformation factor to transfer the knowledge in the auxiliary text to the global features of the image, thereby obtaining the transferred global features of the image.

[0025] The interaction information between the global features of the projected auxiliary text and the global features of the projected text is obtained. Based on the corresponding interaction information, the global features of the projected text are adjusted using a transformation factor to transfer the knowledge in the auxiliary text to the global features of the text, thereby obtaining the transferred global features of the text.

[0026] Optionally, a hybrid expert knowledge learning module is constructed. The process of dynamically fusing the transferred global image features and transferred global text features based on this module to obtain the corresponding final image feature representation and final text feature representation includes:

[0027] The hybrid expert learning module includes two expert networks. One expert network is used to capture local patterns and spatial features of the transferred global image features or the transferred global text features through a one-dimensional convolutional network. The other expert network is used to learn global dependencies and contextual relationships of the transferred global image features or the transferred global text features through a multi-head self-attention mechanism.

[0028] By dynamically fusing the knowledge of two expert networks using learnable factors, the corresponding final image feature representation and final text feature representation are obtained.

[0029] Optionally, based on the final feature representations of the image and the final feature representations of the text, the most similar negative sample pair is dynamically selected for each image and text. Fine-grained differences between positive and negative sample pairs are learned through multi-head cross-attention comparison. The process of achieving cross-modal matching from text to image includes:

[0030] Based on the final feature representation of the image and the final feature representation of the text, the similarity between the image and the text and the similarity between the text and the image are calculated respectively.

[0031] For each text's local features, based on the text-to-image similarity, the most similar unpaired image's local features are selected as negative samples. Then, multi-head cross-attention is used to compare and learn the fine-grained differences between positive and negative sample pairs.

[0032] For each image's local features, based on the image-to-text similarity, the most similar unpaired text local features are selected as negative samples. Then, multi-head cross-attention is used to compare and learn the fine-grained differences between positive and negative sample pairs.

[0033] The present invention also provides a text-to-image pedestrian retrieval system based on an auxiliary text alignment paradigm, for implementing the method, comprising: an auxiliary text generation module, a feature extraction module, a latent subspace knowledge transfer module, a hybrid expert knowledge learning module, and a fine-grained matching module;

[0034] The auxiliary text generation module is used to analyze existing image and text modal data to obtain interactive instructions, and uses a multimodal large language model to generate corresponding auxiliary text based on images and interactive instructions;

[0035] The feature extraction module is used to extract global and local features of an image through an image encoder, and to extract global and local features of text and auxiliary text through a text encoder, respectively.

[0036] The latent subspace knowledge transfer module is used to utilize the feature interaction between auxiliary text and image and text to transfer the global feature knowledge of auxiliary text to image and text modalities, thereby obtaining the transferred global features of the image and the transferred global features of the text.

[0037] The hybrid expert knowledge learning module is used to dynamically fuse the transferred global features of the image and the transferred global features of the text respectively, to obtain the corresponding final feature representations of the image and the final feature representations of the text.

[0038] The fine-grained matching module is used to dynamically select the most similar negative sample pair for each image and text based on the final feature representation of the image and the final feature representation of the text. It uses multi-head cross-attention to compare and learn the fine-grained differences between positive and negative sample pairs, thereby achieving cross-modal matching from text to image and completing pedestrian retrieval.

[0039] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method.

[0040] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method.

[0041] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method.

[0042] Compared with the prior art, the present invention has the following advantages and technical effects:

[0043] (1) This invention is the first in the field of text-image pedestrian retrieval to utilize a multimodal large model to generate more detailed and information-rich auxiliary text from pedestrian images, thus overcoming the shortcomings of overly coarse text descriptions in currently available datasets. Furthermore, the use of a large model effectively reduces the significant overhead of manual labeling and achieves excellent scalability.

[0044] (2) The present invention designs a knowledge transfer module based on latent subspace, which transfers knowledge from auxiliary text to image and text modalities, deepens the model’s understanding of complex semantics and effectively promotes cross-modal alignment.

[0045] (3) The fine-grained matching module based on the most difficult sample proposed in this invention can help the model effectively utilize the information of unpaired samples in the current batch of data. In a broader contextual perspective, by distinguishing between paired samples and the most difficult unpaired samples, the model can better capture discriminative fine-grained features, thereby achieving more accurate retrieval. Attached Figure Description

[0046] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0047] Figure 1 This is a structural framework diagram of text-to-image pedestrian retrieval according to an embodiment of the present invention. Detailed Implementation

[0048] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0049] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0050] Example 1

[0051] like Figure 1 As shown, this embodiment provides a text-to-image pedestrian retrieval method based on an auxiliary text alignment paradigm, including the following steps:

[0052] Acquire pedestrian image and text modal data, and preset interactive commands for images and text;

[0053] Using a multimodal large language model, corresponding auxiliary text is generated based on images and interactive commands;

[0054] Global and local features of the image are extracted using an image encoder, and global and local features of the text and auxiliary text are extracted using a text encoder, respectively.

[0055] By leveraging the feature interactions between auxiliary text and images and text, global feature knowledge of the auxiliary text is transferred to image and text modalities, resulting in transferred global image features and transferred global text features.

[0056] A hybrid expert knowledge learning module is constructed. Based on the hybrid expert knowledge learning module, the transferred global features of the image and the transferred global features of the text are dynamically fused to obtain the corresponding final feature representations of the image and the final feature representations of the text.

[0057] Based on the final feature representations of the image and the final feature representation of the text, the most similar negative sample pair is dynamically selected for each image and text. Multi-head cross-attention is used to compare and learn the fine-grained differences between positive and negative sample pairs, thereby achieving cross-modal matching from text to image and completing pedestrian retrieval.

[0058] As a feasible approach, the process of acquiring pedestrian image and text modal data, pre-setting interactive commands for the images and text, and generating corresponding auxiliary text based on the images and interactive commands using a multimodal large language model includes:

[0059] Most text descriptions in existing datasets only present basic information about people, posing a significant challenge to models in fully understanding deep semantic information. Furthermore, manually annotating pedestrian images to obtain richer text descriptions is not only time-consuming but also has poor scalability. Fortunately, recent advances in Multimodal Large Language Models (MLLMs) have made it possible to generate accurate and detailed image descriptions. Therefore, this embodiment uses MLLM to generate auxiliary text to enhance the training data and leverages additional semantic knowledge to promote cross-modal alignment. It is important to note that this embodiment only introduces auxiliary text during the model training phase; during the inference phase, the experimental configuration remains consistent with other methods. In this study, the publicly available Generalized Questions model was selected; however, any MLLM capable of providing detailed descriptions of person images can be used as an alternative.

[0060] To generate more comprehensive text descriptions, this embodiment designs interactive instructions covering six key aspects: gender, hairstyle, clothing, shoes, accessories, and posture. The instructions are as follows: "Provide a detailed description of the person in the image, including aspects like gender, hairstyle, clothing, shoes, accessories, and posture. If any attribute is not visible, you may leave it out. Avoid adding any information not shown in the image." For each pedestrian image in the dataset, this embodiment feeds the image and the above instructions into the MLLM model to obtain an auxiliary text description of the image. The auxiliary text generated using MLLM contains richer information and finer-grained knowledge, providing more valuable semantic support for model training.

[0061] As an implementable approach, the process of extracting global and local features of an image using an image encoder, and extracting global and local features of the text and auxiliary text respectively using a text encoder, includes:

[0062] For the n image-text pairs contained in the current batch, this embodiment defines them as follows: The corresponding generated auxiliary text is defined as

[0063] Image encoder: Given an input image Where H, W, and C represent height, width, and the number of input channels, respectively. This embodiment uses a pre-trained image encoder (Vision Transformer, VIT) to extract image features. First, the image is divided into... Each non-overlapping patch is assigned a value, where p represents the patch size. A learnable [CLS] label is then added to the patch sequence to capture global features. The patch sequence, along with its location embedding, is then passed through an image encoder. Finally, the output features are... Representative image I i The feature representation. For clarity, this embodiment defines global features as follows: Local features are defined as

[0064] Text encoder: Given an input text description T i In this embodiment, the BERT model is used as the text encoder. First, T... i Each word is mapped to its corresponding word embedding, and a learnable [CLS] tag is added to the sequence to represent global features. Then, special tags [PAD] are used to pad the word embeddings until a fixed length is reached. After adding positional embeddings, the entire sequence is fed into a text encoder. The final text representation is as follows: Where M represents the sentence length. For text T i This embodiment defines As a global feature representation This represents local features. For auxiliary text generated by MLLM... This embodiment uses a shared text encoder to extract features. And define Global features as auxiliary text.

[0065] As a specific implementation method, the process of transferring global feature knowledge of the auxiliary text to the image and text modalities by utilizing the feature interaction between the auxiliary text and the image and text modalities to obtain the transferred global features of the image and the transferred global features of the text includes:

[0066] While the auxiliary text generated by MLLM provides deep semantic information, it may also contain inaccurate descriptions of individuals. For example, a dark-haired man in an image might be incorrectly described by the MLLM-generated text as wearing a black hat. To minimize this impact, this embodiment does not directly perform interactions between individual samples, but instead analyzes and integrates the overall sample.

[0067] Specifically, for the n images contained in the current batch Features, as defined in this embodiment Similarly, for the n texts contained in the current batch Features, as defined in this embodiment And for the n auxiliary texts contained in the current batch Features, as defined in this embodiment These features are then projected into a latent subspace using the operation of formula (1).

[0068]

[0069] Where the projection matrix Map the features to a common subspace of dimension d'.

[0070] In the knowledge transfer process, the first step is to calculate... and The interaction information between them. Then, based on the calculated interaction information, the image features are processed using a transformation factor φ. Adjustments are made to transfer knowledge from the auxiliary text to the image features. Similarly, for Knowledge is transferred in the same way from Transferred to This process embeds knowledge from the auxiliary text into the image and text modalities, making their feature distribution in the feature space closer to that of the auxiliary text, thereby reducing the cross-modal distance. The overall operation can be represented by formulas (2) and (3).

[0071]

[0072] Where ⊙ represents the Hadamard product and φ is the softmax operation.

[0073] As a specific implementation method, a hybrid expert knowledge learning module is constructed. Based on this module, the transferred global image features and transferred global text features are dynamically fused to obtain the corresponding final image feature representation and final text feature representation. The process includes:

[0074] In order to effectively help and After knowledge transfer, from auxiliary text To learn knowledge, this embodiment proposes a hybrid expert knowledge learning module. This module uses two parallel expert networks to process the features of the current batch of images. These expert networks are dynamically fused to obtain the final feature representation.

[0075] Specifically, an expert network uses a 1D convolutional network to allow... Capturing local patterns and spatial features, another expert used a multi-head self-attention mechanism (MHSA) to help. The system learns global dependencies and contextual relationships. Finally, it dynamically fuses the knowledge from the two expert networks using learnable factors α and β to obtain the final feature representation. For the text features of the current batch Applying the same steps as above, we obtain... The formula can be expressed as:

[0076]

[0077] Here, Linear represents a linear mapping, LN represents layer normalization, and Conv refers to a one-dimensional convolution operation. Φ1 and Φ2 are the outputs of the two expert networks, and α and β are learnable parameters. Finally, the SDM loss function is used to... and Perform cross-modal matching.

[0078] As a specific implementation method, based on the final feature representation of the image and the final feature representation of the text, the most similar negative sample pair is dynamically selected for each image and text. Multi-head cross-attention is used to contrastively learn the fine-grained differences between positive and negative sample pairs, achieving cross-modal matching from text to image. The process of completing pedestrian retrieval includes:

[0079] Extracting fine-grained discriminative information is crucial for achieving accurate text-to-image retrieval. Traditional methods often design complex local interaction strategies to facilitate fine-grained matching. However, these methods typically focus only on the current positive sample pair, ignoring the learning of other similar negative samples within the current batch. This limitation restricts the model's ability to effectively distinguish between positive and negative samples in a broader context. To address this issue, this embodiment proposes a fine-grained matching module based on the hardest sample. This module selects the most similar negative text for each image and the most similar negative image for each text, forming a new negative pair. By distinguishing between paired samples and the most similar negative samples during training, the network can uncover more fine-grained discriminative information, thereby improving the overall performance of text-to-image retrieval.

[0080] Specifically, this embodiment uses the previously obtained and To calculate the pairwise similarity between each image and text in the current batch of n images, the formula is as follows:

[0081]

[0082] Where τ is the temperature scaling factor. For each text's local features... The corresponding local image features are Based on S t2i In this embodiment, the most similar unpaired local features of the image can be selected as negative samples, denoted as... Then as well as Interaction is achieved through multi-head cross-attention (HMCA). The specific formula is shown below:

[0083]

[0084]

[0085]

[0086] head i =Att(TW Q VW K VW V ), (12)

[0087]

[0088] in This represents the result after the interaction between paired samples. This is the result after interactions between unpaired samples. W Q W K W V , It is a learnable linear transformation, where h is the number of heads in the multi-head attention mechanism, and d is the dimension of the feature. After obtaining... and Next, in this embodiment, average pooling is first applied to obtain... Then, using a linear classifier and softmax operation, the calculation is performed. Matching probability as well as Matching probability Finally, the classification loss is calculated using cross-entropy. The specific steps are as follows.

[0089]

[0090] Similarly, for the local features of each image and its corresponding local text features Based on S i2t The most similar unpaired text local features are selected as negative samples, defined as follows: Then perform the same operations as described above.

[0091] On the other hand, based on the same inventive concept as the above embodiments, this embodiment also provides a text-to-image pedestrian retrieval system based on an auxiliary text alignment paradigm. The implementation method of this pedestrian retrieval system can be mutually referenced with the pedestrian retrieval method provided in the above embodiments in terms of effect. The pedestrian retrieval system includes: an auxiliary text generation module, a feature extraction module, a latent subspace knowledge transfer module, a hybrid expert knowledge learning module, and a fine-grained matching module.

[0092] The auxiliary text generation module is used to acquire pedestrian image and text modal data, preset interactive instructions for images and text, and generate corresponding auxiliary text based on images and interactive instructions using a multimodal large language model.

[0093] The feature extraction module is used to extract global and local features of an image through an image encoder, and to extract global and local features of text and auxiliary text through a text encoder, respectively.

[0094] The latent subspace knowledge transfer module is used to utilize the feature interaction between auxiliary text and image and text to transfer the global feature knowledge of auxiliary text to image and text modalities, thereby obtaining the transferred global features of the image and the transferred global features of the text.

[0095] The hybrid expert knowledge learning module is used to dynamically fuse the transferred global features of the image and the transferred global features of the text respectively, to obtain the corresponding final feature representations of the image and the final feature representations of the text.

[0096] The fine-grained matching module is used to dynamically select the most similar negative sample pair for each image and text based on the final feature representation of the image and the final feature representation of the text. It uses multi-head cross-attention to compare and learn the fine-grained differences between positive and negative sample pairs, thereby achieving cross-modal matching from text to image and completing pedestrian retrieval.

[0097] As a specific implementation, the auxiliary text generation module in this embodiment utilizes a multimodal large language model (MLLM) to generate auxiliary text based on images, thereby providing the model with richer semantic knowledge. The feature extraction module extracts global and local feature representations of images and text. The latent subspace knowledge transfer module transfers knowledge from the auxiliary text to the image and text modalities, helping the network capture more complex semantic relationships and promoting cross-modal alignment. The hybrid expert knowledge learning module helps images and text better learn the knowledge from the auxiliary text. Finally, the fine-grained matching module based on the most difficult sample utilizes the most similar negative samples to help the model deeply explore and extract discriminative fine-grained features.

[0098] Example 2

[0099] This embodiment also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method.

[0100] Example 3

[0101] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method.

[0102] Example 4

[0103] This embodiment also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method.

[0104] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A text-to-image pedestrian retrieval method based on an auxiliary text alignment paradigm, characterized in that, Includes the following steps: Acquire pedestrian image and text modal data, and obtain interactive instructions by analyzing the image data; The interactive instructions include the pedestrian's gender, hairstyle, clothing, shoes, accessories, and posture; Using a multimodal large language model, corresponding auxiliary text is generated based on images and interactive commands; Global and local features of the image are extracted using an image encoder, and global and local features of the text and auxiliary text are extracted using a text encoder, respectively. By leveraging the feature interactions between auxiliary text and images and text, global feature knowledge of the auxiliary text is transferred to image and text modalities, resulting in transferred global image features and transferred global text features. A hybrid expert knowledge learning module is constructed. Based on the hybrid expert knowledge learning module, the transferred global features of the image and the transferred global features of the text are dynamically fused to obtain the corresponding final feature representations of the image and the final feature representations of the text. Based on the final feature representation of the image and the final feature representation of the text, the most similar negative sample pair is dynamically selected for each image and text. Multi-head cross-attention is used to compare and learn the fine-grained differences between positive and negative sample pairs, thereby achieving cross-modal matching from text to image and completing pedestrian retrieval. The process of constructing a hybrid expert knowledge learning module, and dynamically fusing the transferred global image features and transferred global text features based on the hybrid expert knowledge learning module to obtain the corresponding final image feature representation and final text feature representation includes: The learning module of the hybrid expert includes two expert network branches. One expert network branch uses a one-dimensional convolutional network to capture local patterns and spatial features, and the other expert network branch uses a multi-head self-attention mechanism to learn global dependencies and contextual relationships. By dynamically fusing the knowledge of two expert network branches using learnable factors, the transferred global features of the image are input into the two expert network branches to obtain the corresponding final feature representation of the image. The transferred global text features are input into two expert network branches to obtain the corresponding final text feature representations.

2. The method according to claim 1, characterized in that, The process of feature extraction from an image based on the image encoder includes: Use a pre-trained VIT model as the image encoder; The image is divided into several non-overlapping tile sequences. A learnable image tag is added to the tile sequence. The tile sequence with the tag is then fed into the image encoder after position embedding, and the local and global features of the image are output.

3. The method according to claim 1, characterized in that, The process of feature extraction from the text and auxiliary text based on the text encoder includes: Use the BERT model as a text encoder; Each word in the text or auxiliary text is mapped to its corresponding word embedding to generate a word embedding sequence. A learnable text tag is added to the word embedding sequence, and then a padding tag is used to pad the word embedding sequence until the word embedding sequence reaches a fixed length. The padded word embedding sequence is then fed into the text encoder along with a positional embedding to output the local and global features of the text or auxiliary text.

4. The method according to claim 1, characterized in that, The process of transferring global feature knowledge from auxiliary text to image and text modalities by leveraging feature interactions between auxiliary text and images / text, and obtaining transferred global image features and global text features, includes: The global features of the image, text, and auxiliary text are projected into a latent subspace to obtain the projected global features of the image, the projected global features of the text, and the projected global features of the auxiliary text. The interaction information between the global features of the projected auxiliary text and the global features of the projected image is obtained. Based on the corresponding interaction information, the global features of the projected image are adjusted using a transformation factor to transfer the knowledge in the auxiliary text to the global features of the image, thereby obtaining the transferred global features of the image. The interaction information between the global features of the projected auxiliary text and the global features of the projected text is obtained. Based on the corresponding interaction information, the global features of the projected text are adjusted using a transformation factor to transfer the knowledge in the auxiliary text to the global features of the text, thereby obtaining the transferred global features of the text.

5. The method according to claim 1, characterized in that, Based on the final feature representations of the image and text, the most similar negative sample pair is dynamically selected for each image and text. Multi-head cross-attention is used to contrastively learn the fine-grained differences between positive and negative sample pairs. The process of achieving cross-modal matching from text to image includes: Based on the final feature representation of the image and the final feature representation of the text, the similarity between the image and the text and the similarity between the text and the image are calculated respectively. For each text's local features, based on the text-to-image similarity, the most similar unpaired image's local features are selected as negative samples. Then, multi-head cross-attention is used to compare and learn the fine-grained differences between positive and negative sample pairs. For each image's local features, based on the image-to-text similarity, the most similar unpaired text local features are selected as negative samples. Then, multi-head cross-attention is used to compare and learn the fine-grained differences between positive and negative sample pairs.

6. A text-to-image pedestrian retrieval system based on an auxiliary text alignment paradigm, characterized in that, The method for implementing any one of claims 1-5 includes: an auxiliary text generation module, a feature extraction module, a latent subspace knowledge transfer module, a hybrid expert knowledge learning module, and a fine-grained matching module; The auxiliary text generation module is used to analyze existing image and text modal data to obtain interactive instructions, and uses a multimodal large language model to generate corresponding auxiliary text based on images and interactive instructions; The feature extraction module is used to extract global and local features of an image through an image encoder, and to extract global and local features of text and auxiliary text through a text encoder, respectively. The latent subspace knowledge transfer module is used to utilize the feature interaction between auxiliary text and image and text to transfer the global feature knowledge of auxiliary text to image and text modalities, thereby obtaining the transferred global features of the image and the transferred global features of the text. The hybrid expert knowledge learning module is used to dynamically fuse the transferred global features of the image and the transferred global features of the text respectively, to obtain the corresponding final feature representations of the image and the final feature representations of the text. The fine-grained matching module is used to dynamically select the most similar negative sample pair for each image and text based on the final feature representation of the image and the final feature representation of the text. It uses multi-head cross-attention to compare and learn the fine-grained differences between positive and negative sample pairs, thereby achieving cross-modal matching from text to image and completing pedestrian retrieval.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-5.

9. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-5.

Images