Cross-modal image-text pedestrian retrieval method based on generative model

Abstract

The invention relates to a cross-modal retrieval technology, in particular to a cross-modal image-text pedestrian retrieval method based on a generative model, which comprises the following steps: acquiring text description information and a corresponding original pedestrian image; calling a diffusion generation model to generate an intermediate image based on the text description; original image features and text features are extracted through an image encoder and a text encoder respectively, and semantic-fused text feature representation is constructed; calculating the similarity between the image feature representation and the text feature representation to obtain an image-text matching score; introducing and generating an intermediate image as a semantic bridge, and realizing multi-modal fusion among the image, the text and the intermediate image through a cross attention mechanism to obtain fused image and text features; and using the fusion features to train a recognition model, and finally outputting a pedestrian image retrieval result corresponding to the text description. According to the method, the image-text alignment precision can be improved, the robustness of a scene with incomplete text description is enhanced, and the recognition accuracy in a cross-modal retrieval task is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to computer vision and natural language processing technology, in particular to a kind of combination diffusion model and cross-modal feature alignment text-person retrieval method, belong to artificial intelligence and image retrieval field. BACKGROUND

[0002] TIPR text-person retrieval (Text-to-Image Person Retrieval) is a key task in the field of cross-modal retrieval, which aims to retrieve the target person image consistent with the semantics from a large number of image libraries according to the input natural language description. This task can directly locate and identify the person through the text without providing image query samples by human, has high practical application value, and has been widely used in intelligent security monitoring, criminal investigation tracking, smart city traffic and other scenes.

[0003] In recent years, with the development of visual language pre-training model (such as CLIP), TIPR method has made significant progress. Typical methods usually use text and image feature encoders to embed text and image into the same feature space, and complete retrieval through similarity measurement. However, the existing methods still have the following outstanding problems in practical application:

[0004] Incomplete text semantic expression: the natural language description provided by the user is usually subjective and abstract, which is difficult to cover the key details in the image, especially when it comes to fine semantics such as clothing, accessories and color, which leads to a decrease in text-image matching accuracy;

[0005] Modality difference is difficult to align: there is a natural modality difference between image and text in the perception structure. Existing models mainly use deep feature alignment to alleviate the difference, but it is still difficult to effectively associate when facing semantic missing or description ambiguity;

[0006] Performance degradation is serious in low-resource environment: when the sample of the data set is limited or the text information is insufficient, the existing method is prone to overfitting, and lacks robustness in complex scenes and weak supervision conditions. SUMMARY

[0007] The technical problem to be solved by the present application is that the existing text-person retrieval method has the defects of insufficient generalization ability and unstable alignment effect when facing the actual problems of sparse text expression, large modality difference and limited training samples, and provides a text-person retrieval solution that can enhance text semantic expression, improve text-image matching alignment and improve retrieval robustness.

[0008] The technical solution adopted by the present application to solve the above technical problems is a cross-modal text-person retrieval method based on a generative model, comprising

[0009] The method comprises a training phase and an inference phase:

[0010] 1) The training phase comprises the following steps:

[0011] Obtain a set of image-text pairs formed by text description information and corresponding original images containing pedestrians;

[0012] Input the original images in the set of image-text pairs into a first image encoder to extract original image feature vectors; at the same time, input the text description information into a first text encoder to extract original text feature vectors, and input the text description information into a diffusion generation model as conditional information, which is used to control the diffusion generation model to generate and output intermediate images consistent with the semantics of the text description;

[0013] Input the intermediate images into an intermediate image encoder to extract intermediate image feature vectors; in the training process, the intermediate image encoder performs weighted fusion on the text feature vectors and the intermediate image feature vectors to obtain text representation features; and the text representation features and the original image feature vectors in the set of image-text pairs are used to calculate alignment loss, so as to improve the similarity between the text representation features and the matched original image feature vectors, and reduce the similarity between the text representation features and other original image feature vectors as the training target;

[0014] Input the original image feature vectors, the intermediate image feature vectors and the text feature vectors into a generative intermediate fusion module, which comprises a second image encoder and a second text encoder; the generative intermediate fusion module performs fusion processing on the original image feature vectors and the intermediate image feature vectors to obtain fusion image features, and then inputs the fusion image features into the second image encoder to obtain re-encoded image features; at the same time, the generative intermediate fusion module performs fusion processing on the original text feature vectors and the intermediate image feature vectors to obtain fusion text features, and then inputs the fusion text features into the second text encoder to obtain re-encoded text features; the re-encoded image features and the re-encoded text features are used to calculate fusion loss, so as to improve the similarity between the re-encoded text features and the matched re-encoded image features, and reduce the similarity between the re-encoded text features and other re-encoded image features as the training target;

[0015] 2) The inference phase comprises the following steps:

[0016] Obtain text description information and a set of original images containing pedestrians to be matched, and input them into an image-text pedestrian retrieval model composed of a first image encoder, a first text encoder, an intermediate image encoder and a diffusion generation model; the image-text pedestrian retrieval model outputs the original images matched as retrieval results. Based on the same inventive concept, the present application provides a computer program product, which, when executed by a processor, realizes all steps of the image-text pedestrian retrieval method.

[0017] The application introduces a diffusion model as a generation module to generate an intermediate image for enhancing text semantic expression, and when original text information is incomplete or semantic is sparse, key description content can still be completed to improve the accuracy of image-text alignment. A GIF module fusing an intermediate semantic image is proposed to realize fine-grained fusion among images, text and generated images, and to alleviate the problem of large modal difference in traditional image-text retrieval methods. Experiments show that the application has better performance than existing methods on multiple mainstream TIPR datasets (such as CUHK-PEDES, RSTPReid and ICFG-PEDES), verifying the universality and high accuracy of the method. The model structure is flexible and can be deployed in actual video monitoring, intelligent retrieval and security scenarios, and has engineering landing ability.

[0018] The application has the beneficial effect of realizing text semantic completion and enhancement, improving the robustness and generalization ability of image-text matching, and further improving the accuracy and application value of image-text pedestrian retrieval. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 FIG. 1 is a structural schematic diagram of a training process of a cross-modal image-text pedestrian retrieval based on a generation model in an embodiment of the application. DETAILED DESCRIPTION

[0020] An image-text pedestrian retrieval method includes a training phase and an inference phase:

[0021] 1) The training phase, as shown in part (a), is used to realize a TGTE Text-Guided Token Enhancement model of an image-text pedestrian retrieval method, which includes a first image encoder, a diffusion generation model, an intermediate image encoder, a first text encoder and a generative intermediate fusion module GIF (Generative intermediate Fusion): Figure 1

[0022] Obtain a set of image-text pairs formed by text description information and corresponding original images containing pedestrians;

[0023] Input the original image Image into the first image encoder to extract the original image feature vector Image Feature; at the same time, input the text description information Text into the first text encoder to extract the original text feature vector Text Feature, and input the text description information Text into the diffusion generation model as conditional information, which is used to control the diffusion generation model to generate and output an intermediate image Generated Image consistent with the text description semantic; the intermediate image serves as a visual supplement to the text semantic;

[0024] ​The intermediate image is input to an intermediate image encoder to extract an intermediate image feature vector Diffusion Feature; the intermediate image encoder shares the weight Share weight of the image encoder, that is, the intermediate image encoder adopts a model with the same parameters as the first image encoder; in the training process, the text feature vector and the intermediate image feature vector are weighted and fused to obtain a text representation feature; the text representation feature and each original image feature vector in the image-text pair set are used to calculate the alignment loss Align Loss to improve the similarity between the text representation feature and the matching original image feature vector, and reduce the similarity between the text representation feature and other original image feature vectors as the pedestrian retrieval model training target; for example, in a batch of training set, 64 text description information and matching original images form an image-text pair, in a batch, the weighted and fused text representation feature A corresponding to the text description information a and the generated intermediate image is calculated with the original 64 original images to improve the similarity with the matching original image, and the similarity of the other 63 original images which do not match is weighted and summed in the Align Loss, so that their similarity is lower, and the difficult examples (original images which do not correspond but have high similarity) are better distinguished; the similarity adopted in the embodiment is cosine similarity;

[0025] As shown in part (b) of Figure 1 The original image feature vector, the intermediate image feature vector and the text feature vector are input into the generative intermediate fusion module, and the generative intermediate fusion module includes a second image encoder and a second text encoder;

[0026] The generative intermediate fusion module performs cross attention fusion processing on the original image feature vector and the intermediate image feature vector through a first cross attention path to obtain a fused image feature Fused Image Feature; since the fused feature contains a lot of useless information, the fused image feature is input into the second image encoder to obtain a re-encoded image feature; at the same time, the original text feature vector and the intermediate image feature vector are fused through a second cross attention path to obtain a fused text feature Fused Text Feature; the fused text feature is input into the second text encoder to obtain a re-encoded text feature; the re-encoded image feature and the re-encoded text feature are used to calculate the fusion loss Fusion Loss to improve the similarity between the re-encoded text feature and the matching re-encoded image feature, and reduce the similarity between the re-encoded text feature and other re-encoded image features as the pedestrian retrieval model training target;

[0027] The alignment loss and the fusion loss are used to jointly constrain the image-text matching process to complete the pedestrian retrieval model training;

[0028] 2) Inference stage, the pedestrian retrieval model uses the trained first image encoder, diffusion generation model, intermediate image encoder and first text encoder to perform image-text pedestrian retrieval:

[0029] Obtain the text description information and the original image set containing pedestrians to be matched;

[0030] Input each original image in the original image set into the first image encoder to extract the original image feature vector; at the same time, input the text description information into the first text encoder to extract the original text feature vector, and input the text description information into the diffusion generation model as conditional information, and the conditional information output is an intermediate image.

[0031] Input the intermediate image into the intermediate image encoder to extract the intermediate image feature vector; and perform weighted fusion on the text feature vector and the intermediate image feature vector to obtain a text representation feature.

[0032] Calculate the similarity between the text representation feature and each original image in the original image set containing pedestrians to be matched, and output the original image with the highest similarity as the retrieval result.

[0033] Specifically, the weighted fusion method of the text feature vector and the intermediate image feature vector is:

[0034] t cls =(1-ω)t eos +ωg cls , 0 < ω < 1.

[0035] Where t cls is the fused text representation feature, t eos is the original text feature vector, g cls is the intermediate image feature vector, and ω is the fusion weight.

[0036] Specifically, the diffusion generation model is a cross-modal generation model, and the hyperparameters of the model include a cross-modal mixing parameter and a temperature parameter. More specifically, the diffusion generation model is a pre-trained stable diffusion model Stable Diffusion 3, which is based on a frozen contrastive language-image pre-training model CLIP, embeds a text into a conditional vector, and iteratively denoises to generate a corresponding intermediate image in a hidden space.

[0037] Specifically, the generated intermediate fusion module GIF includes a multi-layer cross-attention structure, adopts a Transformer as a basic network unit, and is used to realize fine-grained feature fusion between the intermediate image and the original image and the text. The first text encoder and the first image encoder are both shared visual language encoders based on a CLIP model, and are used to extract modal feature vectors of the image and the text, respectively. The first text encoder and the first image encoder include 12 intermediate layers composed of a self-attention mechanism and a feed-forward network structure. The second text encoder and the second image encoder include 6 intermediate layers composed of a self-attention mechanism and a feed-forward network structure, and are used to realize fine alignment between the intermediate semantics and the original modalities.

[0038] Embodiment one

[0039] The text-person image retrieval method described in the embodiment is used to perform a text-person image retrieval experiment in a text-person retrieval scene to verify the effectiveness and generalization performance thereof.

[0040] The specific settings are as follows:

[0041] The CUHK-PEDES, RSTPReid and ICFG-PEDES three text-person retrieval data sets are used to train and test the model described in the application.

[0042] As shown in Table 1, compared with other text-person retrieval methods in different data set scenarios:

[0043] Table 1

[0044]

[0045]

[0046] The experimental results show that the generated enhanced alignment method (GEA) proposed in the application achieves a performance significantly better than that of the prior art on the above three data sets. Especially on the CUHK-PEDES data set, the Rank-1 retrieval accuracy reaches 80.56%, which is improved by 7.18% and 4.05% compared with the existing representative methods (such as IRRA and RaSa); at the same time, on the RSTPReid and ICFG-PEDES data sets, the Rank-1 accuracy reaches 67.60% and 65.56%, respectively, which are optimal levels under the current unsupervised setting.

[0047] On the RSTPReid dataset, our method achieved 67.60% on R-1 and 54.03% on mAP, which are 0.70% and 1.72% higher than RaSa, respectively. These results show that our method can provide accuracy and robustness in various retrieval scenarios, especially in aligning semantic representations across modalities.

[0048] On the ICFG-PEDES dataset, our method performs competitively. It achieves the highest mAP score of 43.08%, ranking first in R-5 and R-10. Although the R-1 score (65.56%) is slightly lower than the best-performing method, our model still demonstrates strong retrieval capability and overall balanced performance.

[0049] Embodiment Two

[0050] Based on the same inventive concept, the second embodiment of the present application further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the pedestrian re-identification method described above when executing the program.

[0051] Embodiment Three

[0052] Based on the same inventive concept, the third embodiment of the present application further provides a computer program product, comprising computer programs / instructions, which, when executed by a processor, implement the method steps described above.

[0053] Those skilled in the art will appreciate that embodiments of the present application can be provided as a method, a pedestrian retrieval model, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

Claims

1. A cross-modal image-text pedestrian retrieval method based on a generative model, characterized by: Including training phase and inference phase: 1) Training phase: Obtain a set of image-text pairs consisting of text description information and the corresponding original image containing the pedestrian; The original image in the image-text pair set is input into the first image encoder to extract the original image feature vector; at the same time, the text description information is input into the first text encoder to extract the original text feature vector, and the text description information is input into the diffusion generation model as conditional information. The conditional information is used to control the diffusion generation model to generate and output an intermediate image that is semantically consistent with the text description; The intermediate image is input and the intermediate image encoder is used to extract the intermediate image feature vector. During the training process, the intermediate image encoder performs weighted fusion on the text feature vector and the intermediate image feature vector to obtain the text representation feature. The text representation feature and the image-text pair are used to calculate the alignment loss of each original image feature vector in the set, with the training goal of improving the similarity between the text representation feature and the matching original image feature vector and reducing the similarity between the text representation feature and the other original image feature vectors. The original image feature vector, the intermediate image feature vector, and the text feature vector are input into a generative intermediate fusion module, which includes a second image encoder and a second text encoder; the generative intermediate fusion module fuses the original image feature vector and the intermediate image feature vector to obtain a fused image feature, and then inputs the fused image feature into the second image encoder to obtain a re-encoded image feature; at the same time, the original text feature vector and the intermediate image feature vector are fused to obtain a fused text feature, and then inputs the fused text feature into the second text encoder to obtain a re-encoded text feature; The fusion loss is calculated using the re-encoded image features and the re-encoded text features, with the training goal of increasing the similarity between the re-encoded text features and the matching re-encoded image features, and decreasing the similarity between the re-encoded text features and other re-encoded image features. 2) Reasoning stage: The text description information and the set of original images containing pedestrians to be matched are obtained and input into the image-text pedestrian retrieval model composed of the trained first image encoder, first text encoder, intermediate image encoder and diffusion generation model. The image-text pedestrian retrieval model outputs the matching original image as the retrieval result.

2. The method according to claim 1, wherein: The specific steps of the reasoning phase are: Obtain text description information and a set of original images containing pedestrians to be matched; Input each original image in the original image set into a first image encoder to extract the original image feature vector; at the same time, input the text description information into the first text encoder to extract the original text feature vector, and input the text description information into the diffusion generation model as conditional information, and the conditional information outputs an intermediate image; The intermediate image is input and the intermediate image encoder is used to extract the intermediate image feature vector; the text feature vector and the intermediate image feature vector are weightedly fused to obtain the text representation feature; The similarity between the text representation feature and each original image in the original image set containing pedestrians to be matched is calculated, and the original image with the highest similarity is output as the matched retrieval result.

3. The method according to claim 1, wherein: The weighted fusion method of text feature vector and intermediate image feature vector is: t cls =(1-ω)t eos +ωg cls ,0<ω<1。 where t cls is the fused text representation feature, t eos is the original text feature vector, g cls is the intermediate image feature vector, and ω is the fusion weight.

4. The method according to claim 1, wherein: The diffusion generation model is the pre-trained stable diffusion model StableDiffusion 3, which is based on the text encoder of the frozen contrastive language-image pre-training model CLIP. It embeds the text into the conditional vector and iteratively denoises the latent space to generate the corresponding intermediate image.

5. The method according to claim 1, wherein: The generated intermediate fusion module includes a multi-layer cross-attention structure and uses Transformer as the basic network unit to achieve fine-grained feature fusion between intermediate images, original images and text.

6. The method according to claim 5, wherein: The first text encoder and the first image encoder are both shared visual language encoders based on the CLIP model; the first text encoder and the first image encoder both contain 12 intermediate layers consisting of a self-attention mechanism and a feedforward network structure; the second text encoder and the second image encoder both contain 6 intermediate layers consisting of a self-attention mechanism and a feedforward network structure.

7. The method according to claim 1, wherein: The model parameters of the intermediate image encoder are the same as those of the first image encoder. During the training process, the intermediate image encoder shares the model parameters of the first image encoder.

8. The method according to claim 1, wherein: The similarity is cosine similarity.

9. The method according to claim 1, wherein: The fusion process in the generative intermediate fusion module is achieved through a fork attention path.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method steps described in any one of claims 1 to 9 are implemented.

Images