Text-to-image pedestrian retrieval method based on auxiliary text alignment normal form

Abstract

The invention discloses a text-to-image pedestrian retrieval method based on an auxiliary text alignment normal form, which comprises the following steps of: generating a corresponding auxiliary text based on an image and an interactive instruction by utilizing a multi-modal large language model; extracting global features and local features of the image through an image encoder, and respectively extracting global features and local features of the text and the auxiliary text through a text encoder; the global feature knowledge of the auxiliary text is migrated to an image and text mode, the migrated image global features and the migrated text global features are dynamically fused based on a hybrid expert knowledge learning module, and corresponding image final feature representation and text final feature representation are obtained; the most similar negative sample pair is dynamically selected for each image and text, and the fine-grained difference of the positive and negative sample pairs is learned by utilizing multi-head cross attention contrast, so that cross-modal matching from the text to the image is realized, and pedestrian retrieval is completed. According to the method, more accurate text-to-image pedestrian retrieval can be realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of cross-modal person re-identification, and in particular relates to a text-to-image pedestrian retrieval method based on an auxiliary text alignment paradigm. Background Art

[0002] Text-to-image pedestrian retrieval technology holds significant application value in the field of intelligent security. This technology aims to accurately locate target pedestrians from massive amounts of surveillance imagery using natural language descriptions. It can be widely applied to security surveillance systems in public places such as shopping malls, train stations, and streets. Traditional manual review of surveillance video is inefficient and struggles to rapidly retrieve massive amounts of data. Deep learning-based pedestrian retrieval technology, however, can rapidly match target pedestrian images with text descriptions, significantly improving the efficiency and accuracy of security surveillance and providing powerful support for tasks such as tracking suspicious individuals and investigating crimes.

[0003] At present, the technical solutions for text-to-image pedestrian retrieval are mainly divided into two categories: global feature alignment and local feature alignment.

[0004] Global feature alignment methods typically use deep learning models to extract global features from images and text, respectively, and then achieve cross-modal feature alignment using loss functions such as contrastive loss and triplet loss. However, due to the lack of interaction between image and text modalities, this method struggles to achieve fine-grained feature matching, limiting retrieval accuracy in complex scenarios. For example, when different pedestrians may be wearing similar clothing but have different local features, relying solely on global feature alignment can easily lead to false or missed detections.

[0005] Local feature alignment methods improve retrieval accuracy by decomposing images and text into multiple attribute components, enabling more refined cross-modal matching. However, current local-based methods focus primarily on the current paired sample during fine-grained alignment, ignoring information about other negative samples in the batch. This lacks a global perspective and makes it difficult to effectively focus on subtle yet discriminative fine-grained features, resulting in inaccurate retrieval when dealing with similar negative samples.

[0006] In addition, existing public datasets contain paired data of images and corresponding natural language, but the text descriptions mostly only provide a rough description of pedestrians. This allows the model to only establish basic mappings between different modalities and cannot capture deep semantic knowledge, resulting in insufficient reasoning ability when faced with complex sentences or vocabulary.

[0007] To address the above problems, it is urgent to propose a text-to-image pedestrian retrieval method based on the auxiliary text alignment paradigm to improve the retrieval accuracy in complex scenarios. Summary of the Invention

[0008] To solve the above technical problems, the present invention proposes a text-to-image pedestrian retrieval method based on an auxiliary text alignment paradigm to solve the problems existing in the above-mentioned prior art.

[0009] To achieve the above objectives, the present invention provides a text-to-image pedestrian retrieval method based on an auxiliary text alignment paradigm, comprising the following steps:

[0010] Obtain pedestrian image and text modality data, and obtain interactive instructions by analyzing the image data;

[0011] Using a multimodal large language model, corresponding auxiliary text is generated based on images and interactive instructions;

[0012] The global features and local features of the image are extracted through the image encoder, and the global features and local features of the text and auxiliary text are extracted through the text encoder respectively;

[0013] By utilizing the feature interaction between auxiliary text, image and text, the global feature knowledge of auxiliary text is transferred to the image and text modalities to obtain the transferred global features of the image and the transferred global features of the text;

[0014] Construct a hybrid expert knowledge learning module, based on which the migrated global image features and migrated global text features are dynamically fused to obtain the corresponding final image feature representation and final text feature representation;

[0015] Based on the final feature representation of the image and the final feature representation of the text, the most similar negative sample pairs are dynamically selected for each image and text. The fine-grained differences between positive and negative sample pairs are learned using multi-head cross-attention comparison to achieve cross-modal matching from text to image and complete pedestrian retrieval.

[0016] Optionally, the process of extracting features from an image based on the image encoder includes:

[0017] Use the pre-trained VIT model as the image encoder;

[0018] The image is divided into several non-overlapping tile sequences, a preset learnable image marker is added to the tile sequence, and then the labeled tile sequence is embedded with a position and input into the image encoder to output the local and global features of the image.

[0019] Optionally, the process of extracting features from the text and auxiliary text based on the text encoder includes:

[0020] Use the BERT model as a text encoder;

[0021] Each word in the text or auxiliary text is mapped to the corresponding word embedding to generate a word embedding sequence; a preset learnable text tag is added to the word embedding sequence, and then a preset padding tag is preset to fill the word embedding sequence until the word embedding sequence reaches a fixed length; the padded word embedding sequence is added with the position embedding and input into the text encoder, thereby outputting the local features and global features of the text or auxiliary text.

[0022] Optionally, the global feature knowledge of the auxiliary text is transferred to the image and text modalities by utilizing the feature interaction between the auxiliary text and the image and text. The process of obtaining the transferred global features of the image and the transferred global features of the text includes:

[0023] Projecting the global features of the image, text, and auxiliary text into a latent subspace to obtain the projected image global features, the projected text global features, and the projected auxiliary text global features;

[0024] Obtaining the interaction information between the projected auxiliary text global features and the projected image global features, adjusting the projected image global features using a conversion factor based on the corresponding interaction information, migrating the knowledge in the auxiliary text to the image global features, and obtaining the migrated image global features;

[0025] The interaction information between the projected auxiliary text global features and the projected text global features is obtained, and the projected text global features are adjusted using a conversion factor based on the corresponding interaction information, so that the knowledge in the auxiliary text is transferred to the text global features to obtain the transferred text global features.

[0026] Optionally, a hybrid expert knowledge learning module is constructed, and a process of dynamically fusing the migrated image global features and the migrated text global features based on the hybrid expert knowledge learning module to obtain the corresponding final image feature representation and final text feature representation includes:

[0027] The hybrid expert learning module includes two expert networks, one expert network is used to allow the transferred global image features or the transferred global text features to capture local patterns and spatial features through a one-dimensional convolutional network, and the other expert network is used to allow the transferred global image features or the transferred global text features to learn global dependencies and contextual relationships through a multi-head self-attention mechanism;

[0028] The knowledge of the two expert networks is dynamically fused using learnable factors to obtain the corresponding final feature representation of the image and the final feature representation of the text.

[0029] Optionally, based on the final feature representation of the image and the final feature representation of the text, the most similar negative sample pair is dynamically selected for each image and text, and the fine-grained differences between the positive and negative sample pairs are learned by multi-head cross-attention comparison. The process of achieving cross-modal matching of text to image includes:

[0030] Based on the final feature representation of the image and the final feature representation of the text, the similarity between the image and the text and the similarity between the text and the image are calculated respectively;

[0031] For each local feature of text, based on the similarity between text and image, the most similar unpaired local features of the image are selected as negative samples. Then, multi-head cross-attention is used to compare and learn the fine-grained differences between positive and negative sample pairs.

[0032] For each image’s local features, based on the image-to-text similarity, the most similar unpaired text local features are selected as negative samples, and then multi-head cross-attention is used to compare and learn the fine-grained differences between positive and negative sample pairs.

[0033] The present invention also provides a text-to-image pedestrian retrieval system based on an auxiliary text alignment paradigm for implementing the method described, comprising: an auxiliary text generation module, a feature extraction module, a latent subspace knowledge transfer module, a hybrid expert knowledge learning module, and a fine-grained matching module;

[0034] The auxiliary text generation module is used to analyze the existing image and text modal data to obtain interactive instructions, and use the multimodal large language model to generate corresponding auxiliary text based on the image and interactive instructions;

[0035] The feature extraction module is used to extract global features and local features of the image through the image encoder, and to extract global features and local features of the text and auxiliary text respectively through the text encoder;

[0036] The latent subspace knowledge transfer module is used to utilize the feature interaction between the auxiliary text and the image and text to transfer the global feature knowledge of the auxiliary text to the image and text modalities, thereby obtaining the transferred global features of the image and the transferred global features of the text;

[0037] The hybrid expert knowledge learning module is used to dynamically fuse the migrated image global features and the migrated text global features to obtain the corresponding image final feature representation and text final feature representation;

[0038] The fine-grained matching module is used to dynamically select the most similar negative sample pairs for each image and text based on the final feature representation of the image and the final feature representation of the text, and use multi-head cross-attention to compare and learn the fine-grained differences between positive and negative sample pairs to achieve cross-modal matching from text to image and complete pedestrian retrieval.

[0039] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method.

[0040] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method when the computer program is executed by a processor.

[0041] The present invention also provides a computer program product, comprising a computer program, which implements the steps of the method when executed by a processor.

[0042] Compared with the prior art, the present invention has the following advantages and technical effects:

[0043] (1) This paper is the first in the field of text-image person retrieval to utilize a large multimodal model to generate more informative and fine-grained auxiliary text from person images, addressing the overly coarse filtering of text descriptions in existing public datasets. Furthermore, the use of a large model effectively reduces the significant overhead of manual labeling and achieves excellent scalability.

[0044] (2) The present invention designs a knowledge transfer module based on latent subspace to transfer the knowledge of auxiliary text to image and text modalities, deepen the model's understanding of complex semantics and effectively promote cross-modal alignment.

[0045] (3) The proposed fine-grained matching module based on the most difficult sample can help the model effectively utilize the information of unpaired samples in the current batch of data. By distinguishing paired samples from the most difficult unpaired samples in a broader context, the model can better capture discriminative fine-grained features, thereby achieving more accurate retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0047] Figure 1 This is a structural framework diagram of text-to-image pedestrian retrieval in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0049] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0050] Example 1

[0051] like Figure 1 As shown, this embodiment provides a text-to-image pedestrian retrieval method based on an auxiliary text alignment paradigm, including the following steps:

[0052] Obtain pedestrian image and text modal data, and preset interactive instructions for images and text;

[0053] Using a multimodal large language model, corresponding auxiliary text is generated based on images and interactive instructions;

[0054] The global features and local features of the image are extracted through the image encoder, and the global features and local features of the text and auxiliary text are extracted through the text encoder respectively;

[0055] By utilizing the feature interaction between auxiliary text, image and text, the global feature knowledge of auxiliary text is transferred to the image and text modalities to obtain the transferred global features of the image and the transferred global features of the text;

[0056] Construct a hybrid expert knowledge learning module, based on which the migrated global image features and migrated global text features are dynamically fused to obtain the corresponding final image feature representation and final text feature representation;

[0057] Based on the final feature representation of the image and the final feature representation of the text, the most similar negative sample pairs are dynamically selected for each image and text. The fine-grained differences between positive and negative sample pairs are learned using multi-head cross-attention comparison to achieve cross-modal matching from text to image and complete pedestrian retrieval.

[0058] As an implementable approach, the process of obtaining pedestrian image and text modal data and presetting interactive instructions for the image and text is carried out. The process of generating corresponding auxiliary text based on the image and interactive instructions using a multimodal large language model includes the following:

[0059] Most text descriptions in existing data sets only present basic information about the characters, which makes it difficult for the model to fully understand deep semantic information. Moreover, if pedestrian images are manually annotated to obtain richer text descriptions, it will not only take a lot of time, but also have poor scalability. Fortunately, the latest progress in the field of multimodal large language models (MLLM) has made it possible to generate accurate and detailed image descriptions. Therefore, this embodiment uses MLLM to generate auxiliary text to enhance the training data and use additional semantic knowledge to promote cross-modal alignment. It should be emphasized that this embodiment only introduces auxiliary text in the model training stage. In the reasoning stage, the experimental configuration is consistent with other methods. In the study of this embodiment, this embodiment selected the publicly available Tongyi Qianwen model. Of course, any MLLM that can provide a detailed description of a character image can be used as an alternative.

[0060] In order to generate a more comprehensive text description, this embodiment designs interactive instructions covering six key aspects: gender, hairstyle, clothing, shoes, accessories and posture. The instructions are as follows: "Provide adetailed description of the person in the image, including aspects like gender, hairstyle, clothing, shoes, accessories and posture. If any attribute is not visible, you may leave it out. Avoid adding any information not shown in the image." For each pedestrian image in the dataset, this embodiment feeds the image and the above instructions into the MLLM model to obtain an auxiliary text description of this image. The auxiliary text generated by MLLM contains richer information and fine-grained knowledge, which can provide more valuable semantic support for model training.

[0061] As an implementable approach, the process of extracting global features and local features of an image through an image encoder and extracting global features and local features of text and auxiliary text through a text encoder includes:

[0062] For the n pairs of images and texts contained in the current batch, this embodiment is defined as The corresponding generated auxiliary text is defined as

[0063] Image encoder: Given an input image Where H, W, and C represent the height, width, and number of input channels, respectively. This embodiment uses a pre-trained image encoder (Vision Transformer, VIT) to extract image features. First, the image is divided into non-overlapping tiles, where p represents the tile size. Then a learnable [CLS] tag is added to the tile sequence to capture global features. The tile sequence is embedded with the position and passed through the image encoder together. Finally, the output feature Representative image I i For the sake of clarity, this embodiment defines the global feature as The local features are defined as

[0064] Text Encoder: Given an input text description T i , this embodiment uses the BERT model as the text encoder. First, T i Each word in is mapped to its corresponding word embedding, and a learnable [CLS] tag is added to the sequence to represent the global feature. Then a special tag [PAD] is used to pad the word embedding until it reaches a fixed length. After adding the position embedding, the entire sequence is fed into the text encoder. The final text representation is Where M represents the sentence length. For text T i , this embodiment defines As a global feature representation, For local feature representation. For auxiliary text generated by MLLM This example uses a shared text encoder to extract features And define As a global feature of auxiliary text.

[0065] As a specific implementation method, the global feature knowledge of the auxiliary text is transferred to the image and text modalities by utilizing the feature interaction between the auxiliary text and the image and text. The process of obtaining the transferred global features of the image and the transferred global features of the text includes:

[0066] While the auxiliary text generated by MLLM provides deep semantic information, it may also contain inaccurate descriptions of individuals. For example, a man with dark hair in an image may be incorrectly described as wearing a black hat by the text generated by MLLM. To minimize this effect, this example does not directly perform interactions between individual samples, but instead analyzes and integrates the entire sample.

[0067] Specifically, for the n images contained in the current batch Features, this embodiment is defined as Similarly, for the n texts contained in the current batch Features, this embodiment is defined as And for the n auxiliary texts included in the current batch Features, this embodiment is defined as These features are then projected into a latent subspace using the operation of formula (1).

[0068]

[0069] The projection matrix Map the features to a common subspace of dimension d'.

[0070] In the knowledge transfer process, first calculate and Then, the image features are transformed using the conversion factor φ based on the calculated interaction information. Adjust to transfer the knowledge in the auxiliary text to the image features. Similarly, for In the same way, knowledge is transferred from Transfer to This process embeds the knowledge in the auxiliary text into the image and text modalities, making their feature distribution in the feature space closer to the auxiliary text, thereby narrowing the cross-modal distance. The overall operation can be expressed as Equation (2) and Equation (3).

[0071]

[0072] where ⊙ represents the Hadamard product and φ is the softmax operation.

[0073] As a specific implementation method, a hybrid expert knowledge learning module is constructed. Based on the hybrid expert knowledge learning module, the process of dynamically fusing the migrated image global features and the migrated text global features to obtain the corresponding final image feature representation and final text feature representation includes:

[0074] To effectively help and After knowledge transfer from auxiliary text To learn knowledge, this embodiment proposes a hybrid expert knowledge learning module. This module uses two parallel expert networks to process the current batch of image features. These expert networks are dynamically integrated to obtain the final feature representation

[0075] Specifically, an expert network uses a 1D convolutional network to make capture local patterns and spatial features, while another expert uses the Multi-Head Self-Attention Mechanism (MHSA) to help Learn global dependencies and contextual relationships. Finally, the knowledge of the two expert networks is dynamically integrated using the learnable factors α and β to obtain the final feature representation. For the text features of the current batch Applying the same operation as above, we get The formula can be expressed as:

[0076]

[0077] Where Linear represents linear mapping, LN represents layer normalization, and Conv refers to a one-dimensional convolution operation. Φ1 and Φ2 are the outputs of the two expert networks, and α and β are learnable parameters. Finally, the SDM loss function is used to and Perform cross-modal matching.

[0078] As a specific implementation, based on the final feature representations of the image and text, the most similar negative sample pairs are dynamically selected for each image and text. Multi-head cross-attention is used to compare and learn the fine-grained differences between positive and negative sample pairs, achieving cross-modal matching from text to image. The process of completing pedestrian retrieval includes:

[0079] Extracting fine-grained discriminative information is crucial for achieving accurate text-to-image retrieval. Traditional methods often design complex local interaction strategies to facilitate fine-grained matching. However, these methods usually only focus on the current positive sample pair and ignore the learning of other similar negative samples in the current batch. This limitation restricts the model's ability to effectively distinguish between positive and negative samples in a wider context. To address this problem, this embodiment proposes a fine-grained matching module based on the most difficult samples. This module selects the most similar negative text for each image and the most similar negative image for each text to form a new negative pair. By distinguishing between paired samples and the most similar negative samples during training, the network can discover more fine-grained discriminative information, thereby improving the overall performance of text-to-image retrieval.

[0080] Specifically, this embodiment uses the previously obtained and To calculate the similarity between the two images and texts in the current batch n. The calculation formula is as follows:

[0081]

[0082] Where τ is the temperature scaling factor. For each local feature of text The corresponding local image features are Based on S t2i , this embodiment can select the most similar unpaired local features of the image as negative samples, denoted as Then as well as Interaction is achieved through multi-head cross attention (HMCA). The specific formula is as follows:

[0083]

[0084]

[0085]

[0086] head i =Att(TW Q ,VW K ,VW V ), (12)

[0087]

[0088] in represents the result after the interaction between paired samples, is the result after the interaction between unpaired samples. Q 、W K 、W V 、 is a linear transformation that can be learned, h is the number of heads in the multi-head attention mechanism, and d is the dimension of the feature. and Afterwards, this embodiment first applies the average pooling operation to obtain Then use the linear classifier and softmax operation to calculate The matching probability as well as The matching probability Finally, the classification loss is calculated using cross entropy The specific operations are as follows.

[0089]

[0090] Similarly, for the local features of each image and the corresponding local features of the text Based on S i2t Select the most similar unpaired text local features as negative samples, defined as Then perform the above operations.

[0091] On the other hand, based on the same inventive concept as the above embodiment, this embodiment also provides a text-to-image pedestrian retrieval system based on an auxiliary text alignment paradigm. The implementation method of this pedestrian retrieval system and the pedestrian retrieval method provided in the above embodiment can be mutually referenced in terms of effect. The pedestrian retrieval system includes: an auxiliary text generation module, a feature extraction module, a latent subspace knowledge transfer module, a hybrid expert knowledge learning module, and a fine-grained matching module;

[0092] The auxiliary text generation module is used to obtain the image and text modal data of the pedestrian, preset interactive instructions for the image and text, and use the multimodal large language model to generate corresponding auxiliary text based on the image and interactive instructions;

[0093] The feature extraction module is used to extract global features and local features of the image through the image encoder, and to extract global features and local features of the text and auxiliary text respectively through the text encoder;

[0094] The latent subspace knowledge transfer module is used to utilize the feature interaction between the auxiliary text and the image and text to transfer the global feature knowledge of the auxiliary text to the image and text modalities, thereby obtaining the transferred global features of the image and the transferred global features of the text;

[0095] The hybrid expert knowledge learning module is used to dynamically fuse the migrated image global features and the migrated text global features to obtain the corresponding image final feature representation and text final feature representation;

[0096] The fine-grained matching module is used to dynamically select the most similar negative sample pairs for each image and text based on the final feature representation of the image and the final feature representation of the text, and use multi-head cross-attention to compare and learn the fine-grained differences between positive and negative sample pairs to achieve cross-modal matching from text to image and complete pedestrian retrieval.

[0097] As a specific implementation method, the auxiliary text generation module of this embodiment uses a multimodal large language model (MLLM) to generate auxiliary text based on images, thereby providing the model with richer semantic knowledge. The feature extraction module is used to extract global and local feature representations of images and texts. The latent subspace knowledge transfer module transfers the knowledge of auxiliary text to image and text modalities, helping the network capture more complex semantic relationships and promoting cross-modal alignment. The hybrid expert knowledge learning module helps images and texts better learn the knowledge of auxiliary text. Finally, the fine-grained matching module based on the most difficult samples utilizes the most similar negative samples to help the model deeply explore and extract discriminative fine-grained features.

[0098] Example 2

[0099] This embodiment further provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method.

[0100] Example 3

[0101] This embodiment further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method when executed by a processor.

[0102] Example 4

[0103] This embodiment also provides a computer program product, including a computer program, which implements the steps of the method when executed by a processor.

[0104] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A text-to-image pedestrian retrieval method based on auxiliary text alignment paradigm, characterized by: The following steps are involved: Obtain pedestrian image and text modality data, and obtain interactive instructions by analyzing the image data; Using a multimodal large language model, corresponding auxiliary text is generated based on images and interactive instructions; The global features and local features of the image are extracted through the image encoder, and the global features and local features of the text and auxiliary text are extracted through the text encoder respectively; By utilizing the feature interaction between auxiliary text, image and text, the global feature knowledge of auxiliary text is transferred to the image and text modalities to obtain the transferred global features of the image and the transferred global features of the text; Construct a hybrid expert knowledge learning module, based on which the migrated global image features and migrated global text features are dynamically fused to obtain the corresponding final image feature representation and final text feature representation; Based on the final feature representation of the image and the final feature representation of the text, the most similar negative sample pairs are dynamically selected for each image and text. The fine-grained differences between positive and negative sample pairs are learned using multi-head cross-attention comparison to achieve cross-modal matching from text to image and complete pedestrian retrieval.

2. The method according to claim 1, characterized in that The process of extracting features from an image based on the image encoder includes: Use the pre-trained VIT model as the image encoder; The image is divided into several non-overlapping tile sequences, a preset learnable image marker is added to the tile sequence, and then the labeled tile sequence is embedded with a position and input into the image encoder to output the local and global features of the image.

3. The method according to claim 1, characterized in that The process of extracting features from the text and auxiliary text based on the text encoder includes: Use the BERT model as a text encoder; Each word in the text or auxiliary text is mapped to the corresponding word embedding to generate a word embedding sequence; a preset learnable text tag is added to the word embedding sequence, and then a preset padding tag is preset to fill the word embedding sequence until the word embedding sequence reaches a fixed length; the padded word embedding sequence is added with the position embedding and input into the text encoder, thereby outputting the local features and global features of the text or auxiliary text.

4. The method according to claim 1, wherein By utilizing the feature interaction between auxiliary text, image, and text, the global feature knowledge of auxiliary text is transferred to the image and text modalities. The process of obtaining the transferred image global features and the transferred text global features includes: Projecting the global features of the image, text, and auxiliary text into a latent subspace to obtain the projected image global features, the projected text global features, and the projected auxiliary text global features; Obtaining the interaction information between the projected auxiliary text global features and the projected image global features, adjusting the projected image global features using a conversion factor based on the corresponding interaction information, migrating the knowledge in the auxiliary text to the image global features, and obtaining the migrated image global features; The interaction information between the projected auxiliary text global features and the projected text global features is obtained, and the projected text global features are adjusted using a conversion factor based on the corresponding interaction information, so that the knowledge in the auxiliary text is transferred to the text global features to obtain the transferred text global features.

5. The method according to claim 4, characterized in that Constructing a hybrid expert knowledge learning module, based on which the migrated global image features and migrated global text features are dynamically fused to obtain the corresponding final image feature representation and final text feature representation. The process includes: The hybrid expert learning module includes two expert networks, one expert network uses a one-dimensional convolutional network to allow the transferred global image features or the transferred global text features to capture local patterns and spatial features, and the other expert network uses a multi-head self-attention mechanism to allow the transferred global image features or the transferred global text features to learn global dependencies and contextual relationships; The knowledge of the two expert networks is dynamically fused using learnable factors to obtain the corresponding final feature representation of the image and the final feature representation of the text.

6. The method according to claim 1, characterized in that Based on the final feature representation of the image and text, the most similar negative sample pairs are dynamically selected for each image and text. The fine-grained differences between the positive and negative sample pairs are learned using multi-head cross-attention comparison. The process of achieving cross-modal text-to-image matching includes: Based on the final feature representation of the image and the final feature representation of the text, the similarity between the image and the text and the similarity between the text and the image are calculated respectively; For each local feature of text, based on the similarity between text and image, the most similar unpaired local features of the image are selected as negative samples. Then, multi-head cross-attention is used to compare and learn the fine-grained differences between positive and negative sample pairs. For each image’s local features, based on the image-to-text similarity, the most similar unpaired text local features are selected as negative samples, and then multi-head cross-attention is used to compare and learn the fine-grained differences between positive and negative sample pairs.

7. A text-to-image pedestrian retrieval system based on auxiliary text alignment paradigm, characterized by: Used to implement the method according to any one of claims 1 to 6, comprising: an auxiliary text generation module, a feature extraction module, a latent subspace knowledge transfer module, a hybrid expert knowledge learning module and a fine-grained matching module; The auxiliary text generation module is used to analyze the existing image and text modal data to obtain interactive instructions, and use the multimodal large language model to generate corresponding auxiliary text based on the image and interactive instructions; The feature extraction module is used to extract global features and local features of the image through the image encoder, and to extract global features and local features of the text and auxiliary text respectively through the text encoder; The latent subspace knowledge transfer module is used to utilize the feature interaction between the auxiliary text and the image and text to transfer the global feature knowledge of the auxiliary text to the image and text modalities, thereby obtaining the transferred global features of the image and the transferred global features of the text; The hybrid expert knowledge learning module is used to dynamically fuse the migrated image global features and the migrated text global features to obtain the corresponding image final feature representation and text final feature representation; The fine-grained matching module is used to dynamically select the most similar negative sample pairs for each image and text based on the final feature representation of the image and the final feature representation of the text, and use multi-head cross-attention to compare and learn the fine-grained differences between positive and negative sample pairs to achieve cross-modal matching from text to image and complete pedestrian retrieval.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images