Pedestrian image-text pair generation method based on large language model and diffusion model

Abstract

The invention relates to a pedestrian image-text pair generation method based on a large language model and a diffusion model, and belongs to the field of computer vision and multi-modal learning. Comprising the following steps: analyzing text description in an existing data set by utilizing a large language model, extracting attributes and templates, and constructing an attribute library and a template library; generating recombined text description by randomly recombining attributes and templates and combining with a large language model; generating a high-quality pedestrian image according to the recombined text by using a diffusion model subjected to field fine tuning, and pairing the pedestrian image with the text to form a large-scale image-text pair data set; and pre-training the CLIP model by using the data set. The system is composed of a data preprocessing module, an attribute and template library construction module, a text recombination module, a diffusion model fine tuning module, an image generation module, a data pair synthesis module and a performance evaluation module. According to the method, massive and diversified training samples can be automatically and efficiently generated, the manual labeling cost is remarkably reduced, the data scale is expanded, and the generalization ability of a downstream model is effectively improved.

Description

Technical Field

[0001] This invention relates to a method for generating pedestrian image-text pairs based on a large language model and a diffusion model, belonging to the field of computer vision and multimodal learning technology. Background Technology

[0002] Person re-identification (ReID) is one of the core tasks in the field of computer vision, aiming to retrieve specific target pedestrians at different times and from different camera perspectives. With the development of deep learning technology, text-based person re-identification (ReID) has attracted widespread attention due to its ability to utilize flexible natural language descriptions for retrieval, and has great application potential in fields such as intelligent surveillance and public safety.

[0003] However, deep learning-based text-based person re-identification models are data-driven, and their performance heavily relies on training with large-scale, high-quality pedestrian image-text pairing data. Currently, the construction of such datasets mainly depends on manual collection and annotation, a tedious and costly process. For example, it requires hiring a large number of annotators to provide detailed textual descriptions of pedestrian appearances in images (such as clothing color, trouser style, and items carried). This manual annotation method has the following inherent drawbacks:

[0004] 1. High cost: The labeling process requires a lot of manpower, time and money;

[0005] 2. Scale limitation: Due to the cost of annotation, the size of the dataset is difficult to increase;

[0006] 3. Insufficient diversity: The limited variety of manually collected image scenes, pedestrian postures, and clothing combinations results in insufficient diversity of training data, which restricts the improvement of the model's generalization ability.

[0007] Therefore, developing a method that can automatically and cost-effectively generate large-scale, high-quality, and diverse pedestrian image-text pair data is crucial for advancing text-based pedestrian re-identification technology. Summary of the Invention

[0008] To overcome the aforementioned deficiencies in existing technologies, this invention provides a pedestrian image-text pair generation method based on a large language model and a diffusion model. This invention can automatically and efficiently generate massive and diverse pedestrian image-text pair data at a low cost, which can be used to train high-performance text pedestrian re-identification models, thereby effectively reducing the reliance on manually labeled data.

[0009] The technical solution of this invention is: a pedestrian image-text pair generation method based on a large language model and a diffusion model, the method comprising:

[0010] S1. Use a large language model to parse the text descriptions in the existing dataset and build an attribute library and a template library;

[0011] S2. Randomly recombine text templates and attributes, and use a large language model to generate grammatically fluent recombined text descriptions to construct a recombined text dataset.

[0012] S3. Use the diffusion model to generate corresponding images based on the recombined text descriptions, and construct an image-text pair dataset.

[0013] S4. Performance evaluation of the text pedestrian re-identification task using the image-text pair dataset.

[0014] Further, S1 includes:

[0015] S1.1 Select a large language model as the parsing tool;

[0016] S1.2 Guide the large language model to parse the text description through prompt words, extract the attribute type and attribute content, and generate the corresponding text template;

[0017] S1.3 Clean the text template and attribute data;

[0018] S1.4 Summarize the cleaned text templates and attributes to build a template library and an attribute library.

[0019] Further, S2 includes:

[0020] S2.1. Randomly select a text template from the template library;

[0021] S2.2 For each variable attribute of the selected text template, randomly select an attribute from the attribute library that matches its attribute type;

[0022] S2.3 Input the selected text template and attributes into the large language model to generate a reconstructed text description while ensuring grammatical fluency;

[0023] S2.4: Repeat steps S2.1 to S2.3 to construct a reconstructed text dataset.

[0024] Further, S3 includes:

[0025] S3.1 Fine-tuning the pre-trained text-image diffusion model on the existing pedestrian re-identification dataset;

[0026] S3.2 Input the recombined text description generated in S2 into the fine-tuned diffusion model to generate the corresponding image;

[0027] S3.3 Pair the generated images with the reconstructed text descriptions to construct an image-text pair dataset.

[0028] Further, S4 includes:

[0029] S4.1. Pre-train the CLIP model using the constructed image-text pair dataset;

[0030] S4.2. Further fine-tune the pre-trained CLIP model on the training set of the original dataset;

[0031] S4.3. Evaluate the performance of the CLIP model on the test set of the original dataset.

[0032] This invention also provides a pedestrian image-text pair generation system based on a large language model and a diffusion model, the system comprising:

[0033] The data preprocessing module is used to load and process existing pedestrian re-identification datasets;

[0034] The attribute and template library building module is used to call the large language model to parse the text description, extract attributes and templates, and build attribute libraries and template libraries respectively. The text recombination module is used to randomly recombine attributes and templates using a large language model to generate recombined text descriptions. The diffusion model fine-tuning module is used to fine-tune a pre-trained text-image diffusion model on an existing dataset; The image generation module is used to generate corresponding images based on the reconstructed text description; The data pair synthesis module is used to pair reconstructed text descriptions with corresponding generated images to construct a standardized image-text pair dataset;

[0035] The performance evaluation module is used to evaluate the performance of the image-text pair dataset in the text pedestrian re-identification task.

[0036] Furthermore, the attribute and template library construction module includes functions for implementing the following:

[0037] 1) Call the large language model to parse the text description, extract the attribute types and attribute content, and generate the corresponding text template;

[0038] 2) Clean the generated text templates and attributes to ensure they are formatted correctly; 3) Compile the text templates to build a template library; compile the attribute types and attribute contents to build an attribute library.

[0039] The pedestrian image-text pair generation method or system based on large language model and diffusion model is applicable not only to text pedestrian retrieval tasks, but also to other target retrieval tasks based on image-text matching.

[0040] The beneficial effects of this invention are:

[0041] 1. This invention significantly reduces data construction costs: by automatically generating image-text pairs, it avoids the expensive and tedious manual annotation process;

[0042] 2. This invention enables large-scale dataset expansion: This invention can efficiently generate a huge number of diverse training samples, thereby significantly reducing the cost of manual annotation and effectively expanding the scale of the dataset;

[0043] 3. This invention can improve model performance: Pre-training with the generated dataset before the fine-tuning stage can significantly improve the generalization ability of the downstream pedestrian retrieval model. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating an embodiment of the method of the present invention. Detailed Implementation

[0045] Example 1: As Figure 1 As shown, a pedestrian image-text pair generation method based on a large language model and a diffusion model is described, the method comprising:

[0046] S1. Use a large language model to parse the text descriptions in the CUHK-PEDES dataset and build an attribute library and a template library.

[0047] Further, S1 includes:

[0048] S1.1 Select ChatGPT-3.5-Turbo as the parsing tool;

[0049] S1.2 Guide the large language model to parse the text description through prompt words, extract the attribute type and attribute content, and generate the corresponding text template;

[0050] S1.3 Clean the text templates and attribute data generated in the previous step: unify capitalization, remove extra spaces, remove data with incorrect formatting, etc.

[0051] S1.4 Store the cleaned templates and attributes into the template library and attribute library respectively; store different types of attributes separately in the attribute library, and build the template library and attribute library for subsequent operations.

[0052] S2. Randomly recombine text templates and attributes, and use a large language model to generate grammatically fluent recombined text descriptions to construct a recombined text dataset.

[0053] Further, S2 includes:

[0054] S2.1. Randomly select a text template from the template library;

[0055] S2.2 Use regular expressions to find all "[attribute type]" parts. For each "[attribute type]", randomly select an attribute of the same type from the attribute library.

[0056] S2.3 Input the selected text template and attributes into the large language model, and use prompt words to guide the model to fill the attributes into the template while ensuring grammatical fluency, and generate a reconstructed text description;

[0057] Prompt template: "Given a pedestrian description template and specific attributes {attribute1 type: attribute1 content, attribute2 type: attribute2 content...}, fill in the specific attributes into the template while keeping the general structure of the template and ensuring grammatical fluency."

[0058] Use functional calls to make the model output standardized answers;

[0059] The translation of the prompt word template is as follows: Given a pedestrian description template and specific attributes, fill in the specific attributes into the template while preserving the overall structure of the template and ensuring grammatical fluency.

[0060] S2.4: Repeat steps S2.1 to S2.3 to build a sufficiently large reconstructed text dataset.

[0061] S3. Use the diffusion model to generate corresponding images based on the recombined text descriptions, and construct an image-text pair dataset.

[0062] Further, S3 includes:

[0063] S3.1 Select Stable-Diffusion-v1-5 as the base model for image generation. Fine-tune the selected text-image diffusion model on the CUHK-PEDES dataset, specifying its output size as 384 pixels high and 128 pixels wide. This step is used to enable the diffusion model to learn the feature distribution in the pedestrian image domain, thereby enabling it to generate high-quality images that conform to the real appearance details of pedestrians based on the input text description.

[0064] S3.2 Input the recombined text description generated in S2 into the fine-tuned diffusion model to generate the corresponding image;

[0065] S3.3 Pair the generated images with the reconstructed text descriptions to construct an image-text pair dataset.

[0066] S4. Performance evaluation of the text pedestrian re-identification task using the image-text pair dataset.

[0067] Further, S4 includes:

[0068] S4.1. Pre-train the CLIP model using the constructed image-text pair dataset;

[0069] S4.2. Further fine-tune the pre-trained CLIP model on the training set of the CUHK-PEDES dataset;

[0070] S4.3. Evaluate the performance of the CLIP model on the test set of the CUHK-PEDES dataset.

[0071] Example 3: This invention also provides a pedestrian image-text pair generation system based on a large language model and a diffusion model, the system comprising:

[0072] The data preprocessing module is used to load and process existing pedestrian re-identification datasets;

[0073] The attribute and template library building module is used to call the large language model to parse the text description, extract attributes and templates, and build attribute libraries and template libraries respectively. The text recombination module is used to randomly recombine attributes and templates using a large language model to generate recombined text descriptions. The diffusion model fine-tuning module is used to fine-tune a pre-trained text-image diffusion model on an existing dataset; The image generation module is used to generate corresponding images based on the reconstructed text description; The data pair synthesis module is used to pair reconstructed text descriptions with corresponding generated images to construct a standardized image-text pair dataset;

[0074] The performance evaluation module is used to evaluate the performance of the image-text pair dataset in the text pedestrian re-identification task.

[0075] Furthermore, the attribute and template library construction module includes functions for implementing the following:

[0076] 1) Call the large language model to parse the text description, extract the attribute types and attribute content, and generate the corresponding text template;

[0077] 2) Clean the generated text templates and attributes to ensure they are formatted correctly; 3) Compile the text templates to build a template library; compile the attribute types and attribute contents to build an attribute library.

[0078] The pedestrian image-text pair generation method or system based on large language model and diffusion model is applicable not only to text pedestrian retrieval tasks, but also to other target retrieval tasks based on image-text matching.

[0079] The core of this invention lies in expanding the diversity of text descriptions through the understanding and creativity of a large language model, and using a diffusion model to transform text semantics into visual features. This general process can be applied to any computer vision task that requires image-text pairing data. In the e-commerce field, this invention can be used to generate product descriptions such as "a red dress with lace trim and knee-length" and their corresponding images for training product retrieval models. In the field of intelligent transportation, it can be used to generate vehicle descriptions and images such as "a black SUV with roof rack and scratches on the front bumper." This invention also has application potential in fields such as security monitoring, content moderation, and medical image analysis.

[0080] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A pedestrian image-text pair generation method based on a large language model and a diffusion model, characterized by: The method includes: S1. Use a large language model to parse the text descriptions in the existing dataset and build an attribute library and a template library; S2. Randomly recombine text templates and attributes, and use a large language model to generate grammatically fluent recombined text descriptions to construct a recombined text dataset. S3. Use the diffusion model to generate corresponding images based on the recombined text descriptions, and construct an image-text pair dataset. S4. Performance evaluation of the text pedestrian re-identification task using the image-text pair dataset.

2. The pedestrian image-text pair generation method based on a large language model and a diffusion model according to claim 1, characterized in that: S1 includes: S1.1 Select a large language model as the parsing tool; S1.2 Guide the large language model to parse the text description through prompt words, extract the attribute type and attribute content, and generate the corresponding text template; S1.3 Clean the text template and attribute data; S1.4 Summarize the cleaned text templates and attributes to build a template library and an attribute library.

3. The pedestrian image-text pair generation method based on a large language model and a diffusion model according to claim 1, characterized in that: S2 includes: S2.

1. Randomly select a text template from the template library; S2.2 For each variable attribute of the selected text template, randomly select an attribute from the attribute library that matches its attribute type; S2.3 Input the selected text template and attributes into the large language model to generate a reconstructed text description while ensuring grammatical fluency; S2.4: Repeat steps S2.1 to S2.3 to construct a reconstructed text dataset.

4. The pedestrian image-text pair generation method based on a large language model and a diffusion model according to claim 1, characterized in that: S3 includes: S3.1 Fine-tuning the pre-trained text-image diffusion model on the existing pedestrian re-identification dataset; S3.2 Input the recombined text description generated in S2 into the fine-tuned diffusion model to generate the corresponding image; S3.3 Pair the generated images with the reconstructed text descriptions to construct an image-text pair dataset.

5. The pedestrian image-text pair generation method based on a large language model and a diffusion model according to claim 1, characterized in that: S4 includes: S4.

1. Pre-train the CLIP model using the constructed image-text pair dataset; S4.

2. Further fine-tune the pre-trained CLIP model on the training set of the original dataset; S4.3 Evaluate the performance of the CLIP model on the test set of the original dataset.

6. A pedestrian image-text pair generation system based on a large language model and a diffusion model, characterized in that, The system includes: The data preprocessing module is used to load and process existing pedestrian re-identification datasets; The attribute and template library building module is used to call the large language model to parse the text description, extract attributes and templates, and build attribute libraries and template libraries respectively. The text recombination module is used to randomly recombine attributes and templates using a large language model to generate recombined text descriptions. The diffusion model fine-tuning module is used to fine-tune a pre-trained text-image diffusion model on an existing dataset; The image generation module is used to generate corresponding images based on the reconstructed text description; The data pair synthesis module is used to pair reconstructed text descriptions with corresponding generated images to construct a standardized image-text pair dataset; The performance evaluation module is used to evaluate the performance of the image-text pair dataset in the text pedestrian re-identification task.

7. The pedestrian image-text pair generation system based on a large language model and a diffusion model according to claim 6, characterized in that, The attribute and template library construction module includes functions to implement the following: 1) Call the large language model to parse the text description, extract the attribute types and attribute content, and generate the corresponding text template; 2) Clean the generated text templates and attributes to ensure they are formatted correctly; 3) Compile the text templates to build a template library; compile the attribute types and attribute contents to build an attribute library.

8. The pedestrian image-text pair generation method or system based on a large language model and a diffusion model as described in any one of claims 1-7, characterized in that, The method or system described herein is applicable not only to text-based pedestrian retrieval tasks, but also to other target retrieval tasks based on image-text matching.

Images