High-quality downstream task training data generation method
By constructing object lists and large language models to generate layouts and descriptions, combining controllable image models to generate pictures, and using comprehensive filters, the problems of data scale expansion and quality matching in the existing technology are solved, and high-quality downstream task training is achieved Data generation is suitable for a variety of tasks and improves model training effect.
Patent Information
- Application Number
- CN202410965884.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
When generating downstream task training data, the existing technology lacks the rationality of layout considerations and relies on real text and visual annotation information, which leads to difficulty in expanding the data scale, low-quality data exists and is difficult to match multiple downstream tasks, affecting the model training effect.
By constructing object lists, using large language models to generate layouts and descriptions, combining controllable image models to generate pictures, and using comprehensive filters of layout and pictures, CLIS indicators to filter high-quality data, reduce dependence on real annotations, and generate suitable perceptions and training data for inference tasks.
It realizes the generation of high-quality training data, is suitable for a variety of downstream tasks, improves model training effect, reduces dependence on real annotations, the data generation process is scalable, and data quality is improved through comprehensive indicator screening.
Smart Images

Figure CN119089195A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of training data generation, and in particular to a method for generating high-quality downstream task training data. Background Art
[0002] At present, the generation of training data for downstream tasks such as perception task scenarios and multimodal reasoning tasks generally adopts generative models. However, the method of generating training data based on copying and pasting does not consider the rationality of the layout. It simply pastes the foreground of the generated object onto a random background, or pieces together several generated objects into a new picture. Such training data is only suitable for perception tasks, not for reasoning tasks, and there is a lack of appropriate relationships between different objects. The method of first using a large language model to generate a layout based on the description of the picture, and then generating pictures based on a controllable picture generation model and the generation method based on real data annotations all rely on a large number of real annotations, which makes it difficult to expand the scale of generated data and control the proportion of generated data.
[0003] In summary, the current methods for generating high-quality training data for downstream tasks rely on real text and visual annotation information, such as text descriptions of images and annotation information for detection and segmentation, which makes it difficult to expand the scale of generated data. In addition, the current methods lack a comprehensive evaluation index to screen training data, and the generated training data contains low-quality data that cannot match a variety of downstream tasks, resulting in poor training effects on the models of downstream tasks. Summary of the invention
[0004] The purpose of the present invention is to reduce the dependence on real text, improve the efficiency of training data generation, and provide a high-quality downstream task training data generation method by matching downstream tasks to improve training effects.
[0005] The purpose of the present invention can be achieved by the following technical solutions:
[0006] A method for generating high-quality downstream task training data, the method comprising the following steps:
[0007] S1. Build an object list, input the object list into the large language model of the original data generator, generate an original layout and its related description, input the related description and the original layout into the controllable text-based image model of the original data generator to generate an original image;
[0008] S2, inputting the original picture, the relevant description and the original layout into a data filter, wherein the data filter includes a layout filter and a picture filter, wherein the layout filter calculates the similarity between the size, direction and distance of the original layout and the sample layout, and calculates the total similarity based on the similarity between the size, direction and distance and the sample layout, and the picture filter calculates the description similarity between the original description corresponding to the original picture and the relevant description;
[0009] S3, selecting original layouts whose total similarity exceeds a first threshold as high-quality layouts, and selecting original pictures whose description similarity exceeds a second threshold as high-quality pictures;
[0010] S4. Use high-quality layouts and high-quality images together as training data for perception and reasoning tasks in downstream tasks.
[0011] Furthermore, the size similarity between the original layout and the sample layout is:
[0012]
[0013] Among them, s and o are two different objects, r is the relationship between s and o, E is the sample pool, s e and e are two different objects in the sample layout in the sample pool, S sim is the basic formula for similarity, specifically:
[0014]
[0015] A and B are two different variables. n and B n are the normalization of A and B respectively.
[0016] Furthermore, the similarity between the orientation of the original layout and the sample layout is:
[0017]
[0018] Among them, Di r Represents the direction vector between two objects.
[0019] Furthermore, the distance between the original layout and the similarity between the sample layout is:
[0020] S IoU (s,o,s e , o e )=S sim (IoU(s,o),IoU(s e , o e ))
[0021] S RelDist(s,o,s e , o e )=S sim (RelDist(s,o),RelDist(s e , o e ))
[0022]
[0023] Among them, S Dist (s, o, r) represents the distance between the original layout and the similarity between the sample layout, IoU represents the overlap function, RelDist is the relative distance function, and α and β are weight parameters.
[0024] Furthermore, the total similarity is:
[0025] CLIS-L(s,o,r)=w1S size (s,o,r)+w2,S Dist (s,o,r)+w3S Dir (s, o, r)
[0026] Among them, w1, w2 and w3 are weight parameters.
[0027] Furthermore, the specific steps of calculating the description similarity between the original description corresponding to the original picture and the related description are:
[0028] The original layout includes one or more sub-layout areas, the original description corresponding to the original image includes a first layout description corresponding to the one or more sub-layout areas and a first overall description for the original image, the related description includes a second layout description corresponding to the one or more sub-layout areas and a second overall description for the original image, the similarities between the first layout description and the second layout description of the corresponding sub-layout areas are calculated respectively, and the similarity between the first overall description and the second overall description is calculated, the similarities between the one or more first layout descriptions and the second layout description are multiplied by the corresponding weight coefficients, and the similarities between the first overall description and the second overall description are multiplied by the corresponding weight coefficients, and the above products are added to obtain the description similarity between the original description corresponding to the original image and the related description.
[0029] Furthermore, the object list includes a plurality of objects arranged in sequence.
[0030] Furthermore, the related description is specifically a description of the attributes of each of the objects, the relationship between the objects, and a related description of the original layout.
[0031] Furthermore, the layout specifically refers to the position of each object in the corresponding image, and the object is represented in the layout in the form of a bounding box.
[0032] Furthermore, the original description is obtained by describing the original image using a multimodal model.
[0033] Compared with the prior art, the present invention has the following beneficial effects:
[0034] The present invention adopts a self-designed generation paradigm, uses the contextual learning ability of a large language model to generate relevant descriptions and original layouts, generates a list of objects described in natural language, and generates training images based on the object list. Less real information is required, and the data has a detailed description and a reasonable layout. It can be applied not only to the training of perception task scenarios, but also to the training of multimodal reasoning tasks, and is suitable for a variety of downstream tasks. At the same time, the present invention proposes a comprehensive index, and uses a data filter constructed based on the comprehensive index to filter and obtain high-quality data, which is more helpful for downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 is a flow chart of the present invention;
[0036] Figure 2 The structure diagram of the original data generator and the structure diagram of the data filter used in the present invention, wherein Figure 2 (a) is the structural diagram of the original data generator. Figure 2 (b) is the structural diagram of the data filter. DETAILED DESCRIPTION
[0037] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.
[0038] The present invention proposes a method for generating high-quality downstream task training data. The flowchart of the method is as follows: Figure 1 As shown. The method of the present invention uses a large language model to plan appropriate layout and description information based on a simple list of objects, and then uses a generative model to combine this information to generate a set of pictures. This design reduces the dependence on existing annotation information and takes into account the rationality of the layout. At the same time, the framework of the present invention also includes a data filter, which uses a designed comprehensive evaluation index (similarity) to filter suitable data to improve the effect of downstream tasks. The method of the present invention includes the following steps:
[0039] S1. Build an object list, input the object list into the large language model of the original data generator, generate an original layout and its related description, input the related description and the original layout into the controllable text-based image model of the original data generator to generate an original image;
[0040] S2, inputting the original picture, the relevant description and the original layout into a data filter, wherein the data filter includes a layout filter and a picture filter, wherein the layout filter calculates the similarity between the size, direction and distance of the original layout and the sample layout, and calculates the total similarity based on the similarity between the size, direction and distance and the sample layout, and the picture filter calculates the description similarity between the original description corresponding to the original picture and the relevant description;
[0041] S3, selecting original layouts whose total similarity exceeds a first threshold as high-quality layouts, and selecting original pictures whose description similarity exceeds a second threshold as high-quality pictures;
[0042] S4. Use high-quality layouts and high-quality images together as training data for perception and reasoning tasks in downstream tasks.
[0043] The present invention has the following effects:
[0044] Current methods rely on real text and visual annotation information, such as text descriptions of images and annotation information for detection and segmentation. This invention improves the paradigm of generating training data, reduces the reliance on real annotations (text image descriptions, or visual annotation information), and only relies on a natural language object list to generate data, that is, training data can be generated based on only a concept list, making the generation process scalable.
[0045] The current method lacks a comprehensive evaluation index to screen training data. This paper proposes an index for comprehensive evaluation of layout and images to screen data, taking into account the importance of reasonable layout that conforms to real scenes, without directly adopting the copy-paste method, and studying its correlation with the improvement of the effect on downstream tasks.
[0046] The core of the present invention includes two contents:
[0047] 1. Raw data generator: From a simple list of objects, use a large language model to generate a reasonable layout and corresponding descriptions, and then use a controllable image generation model to generate a set of images.
[0048] 2. Data filter: Use the designed CLIS index to filter suitable training data. This index includes the evaluation of images and layout.
[0049] The structure diagram of the original data generator and data filter is as follows Figure 2 shown.
[0050] This solution adopts a self-designed generation paradigm, starting from an object list (Object List), for example, [dog, frisbee], and using the contextual learning ability of the large language model to generate relevant descriptions (Description) (including detailed attribute descriptions of each object, the relationship between different objects, and a detailed description of the entire picture composed of all objects) and the original layout (Raw Layout), specifically, the position of each object in the picture, where the position is represented by a bounding box, and the description includes the attributes of the object, the relationship between different objects, and the text description of the picture. Then, the generated information is used to generate a set of raw images (Raw Image) through a controllable text-generated image model.
[0051] The generated raw data is filtered using a self-designed data filter to obtain high-quality data. The screening process takes into account the perspectives of images and layouts. For layout, consider screening from the perspectives of size, direction, and distance. First, a sample pool is built based on real data. This sample pool includes two objects, the relationship between objects, and their real layout. Then, under the premise of the same objects and relationships, consider the similarity of the size, direction, and distance of the sample layout currently evaluated with the sample layout from the sample pool. For images, a multimodal model is first used to describe the layout area and the overall area, and then the similarity between the description and the description in the generation stage is calculated. Finally, a certain threshold (default 0.6) is set to screen suitable layouts and images.
[0052] The calculation process of the similarity described is:
[0053] 1) If a piece of data contains two layout areas, there will be three descriptions (two local and one overall).
[0054] 2) For the above three descriptions, the generation stage already has corresponding descriptions. Customize a prompt word so that the large language model can calculate the similarity between them separately (that is, calculate the similarity of the three descriptions separately, and then integrate them through different weights)
[0055] The specific calculation method of similarity is:
[0056] Basic formula, the similarity between A and B, where An and Bn are the standards of A and B respectively:
[0057]
[0058] For objects s, o and the relationship r between them, E is the sample pool.
[0059] Relative size:
[0060]
[0061] Where Area is the formula for finding the area.
[0062] Relative distance:
[0063] S IoU (s,o,s e , o e )=S sim (IoU(s,o),IoU(s e , o e ))
[0064] S RelDist (s,o,s e , o e )=S sim (RelDist(s,o),RelDist(s e , o e ))
[0065]
[0066] Among them, IoU and RelDist are functions for calculating the IoU and relative distance of two objects.
[0067] Relative direction:
[0068]
[0069] Among them, r Calculates the direction vector between two objects.
[0070] The present invention aims to improve the current method of generating training data and generate data suitable for improving the performance of downstream tasks. The advantages are:
[0071] 1. Changed the generation paradigm, the entire architecture relies only on a list of objects described in natural language.
[0072] 2. A new evaluation metric for generated data is proposed to screen the original generated data, and the correlation between the effect on the generated metric and the improvement of downstream tasks is proved.
[0073] 3. The generated data comes with detailed descriptions and reasonable layouts, which can be used not only for training in perception task scenarios, but also for training in multimodal reasoning tasks.
[0074] The present invention requires less real information and reduces the dependencies to only a list of objects composed of natural language; it takes into account the generation of training data for downstream tasks and includes a comprehensive indicator to screen data that is more helpful for downstream tasks.
[0075] The data generated by the present invention is used to demonstrate the effectiveness of LVIS long-tail instance segmentation, open domain object detection, and multimodal MME and GQABenchmark.
[0076] The LVIS long-tail instance segmentation results are shown in Table 1. On the LVIS long-tail instance segmentation task, based on the Mask R-CNN baseline model, an improvement of 5.2APr and 1.1AP was achieved.
[0077] Table 1 LVIS long-tail instance segmentation results
[0078]
[0079] The results of open domain object detection are shown in Table 2. In the task of open domain object detection, based on the Grounding-DINO benchmark model, an APnovel improvement of LVIS 1.3 and an APnovel improvement of COCO 0.4 were achieved.
[0080] Table 2 Results of LVIS open domain object detection
[0081]
[0082] The results of multimodal task detection are shown in Table 3. In the multimodal task, LLaVA-1.5 improves MME by 80.1% and GQA by 0.4%.
[0083] Table 3 Results of multimodal task detection
[0084]
[0085] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.
Claims
1. A method for generating high-quality downstream task training data, characterized in that: The method comprises the following steps: S1. Build an object list, input the object list into the large language model of the original data generator, generate an original layout and its related description, input the related description and the original layout into the controllable text-based image model of the original data generator to generate an original image; S2, inputting the original picture, the relevant description and the original layout into a data filter, wherein the data filter includes a layout filter and a picture filter, wherein the layout filter calculates the similarity between the size, direction and distance of the original layout and the sample layout, and calculates the total similarity based on the similarity between the size, direction and distance and the sample layout, and the picture filter calculates the description similarity between the original description corresponding to the original picture and the relevant description; S3, selecting original layouts whose total similarity exceeds a first threshold as high-quality layouts, and selecting original pictures whose description similarity exceeds a second threshold as high-quality pictures; S4. Use high-quality layouts and high-quality images together as training data for perception and reasoning tasks in downstream tasks.
2. A method for generating high-quality downstream task training data according to claim 1, characterized in that: The size similarity between the original layout and the sample layout is: Among them, s and o are two different objects, r is the relationship between s and o, E is the sample pool, s e and e are two different objects in the sample layout in the sample pool, S sim is the basic formula for similarity, specifically: A and B are two different variables, and An and Bn are the standardizations of A and B respectively.
3. A method for generating high-quality downstream task training data according to claim 2, characterized in that: The similarity between the orientation of the original layout and the sample layout is: Among them, Dir represents the direction vector between two objects.
4. The method for generating high-quality downstream task training data according to claim 3, characterized in that: The distance of the original layout and the similarity of the sample layout is: S IoU (SOS e ,the e )=S sim (IoU(s,o),IoU(s e ,the e )) S RelDist (s,o,s e ,o e )=S sim (RelDist(s,o),RelDist(s e ,o e )) Among them, S Dist (s, o, r) represents the distance between the original layout and the similarity between the sample layout, IoU represents the overlap function, RelDist is the relative distance function, and α and β are weight parameters.
5. A method for generating high-quality downstream task training data according to claim 4, characterized in that: The total similarity is: CLIS-L(s,o,r)=w1S size (s,o,r)+w2S Dist (s,o,r)+w3S Dir (s, o, r) Among them, w1, w2 and w3 are weight parameters.
6. A method for generating high-quality downstream task training data according to claim 5, characterized in that: The specific steps of calculating the description similarity between the original description corresponding to the original picture and the related description are: The original layout includes one or more sub-layout areas, the original description corresponding to the original image includes a first layout description corresponding to the one or more sub-layout areas and a first overall description for the original image, the related description includes a second layout description corresponding to the one or more sub-layout areas and a second overall description for the original image, the similarities between the first layout description and the second layout description of the corresponding sub-layout areas are calculated respectively, and the similarity between the first overall description and the second overall description is calculated, the similarities between the one or more first layout descriptions and the second layout description are multiplied by the corresponding weight coefficients, and the similarities between the first overall description and the second overall description are multiplied by the corresponding weight coefficients, and the above products are added to obtain the description similarity between the original description corresponding to the original image and the related description.
7. A method for generating high-quality downstream task training data according to claim 6, characterized in that: The object list includes a plurality of objects arranged in sequence.
8. A method for generating high-quality downstream task training data according to claim 7, characterized in that: The related description is specifically a description of the attributes of each of the objects, the relationship between the objects, and a related description of the original layout.
9. A method for generating high-quality downstream task training data according to claim 8, characterized in that: The layout is specifically the position of each object in the corresponding image, and the object is represented in the layout in the form of a bounding box.
10. A method for generating high-quality downstream task training data according to claim 9, characterized in that: The original description is obtained by describing the original image using a multimodal model.
Citation Information
Cited By
Document classification method and system based on template matching
CN119672744A
