Controllable data generation method and system based on potential diffusion model, and medium

Abstract

The invention relates to a controllable data generation method and system based on a potential diffusion model, and a medium, and the method comprises the steps: U1, carrying out the image coding of a noise picture and a target ROI picture through an image encoder of the potential diffusion model, obtaining the embedded data information of the noise picture and the embedded data information of the ROI, inputting the target box information into an MLP model, and carrying out the recognition, target box embedded data information is obtained, the scene cue words are encoded through a natural language editor of the potential diffusion model, and prompt embedded data information is obtained; and U2, on the basis of the prompt embedded data information, the target box embedded data information and the ROI embedded data information, splicing and embedding the data by adopting a cross attention splicing algorithm based on adaptive learning. According to the method, the problem that extreme samples are difficult to collect in a target detection part scene is solved, and the problem that the controllability of diffusion model generation content is not accurate enough is solved by introducing target ROI information and box information.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a controllable data generation method, system and medium based on a potential diffusion model. Background Art

[0002] Generative models have made significant progress in the field of image generation, and the quality of their output images is close to that of images captured by real cameras. Based on this technology, a variety of generation methods have been developed, such as image-to-image generation, text-to-image generation, and more complex combined generation techniques. Under the guidance of reasonable generation strategies, these models can controllably generate images of specified categories and rich in semantic elements. If the accuracy of control factors such as target concepts and position constraints can be further improved, generative models will show a broader application prospect.

[0003] The following is an example of the application of target detection data enhancement:

[0004] As a basic task in the field of computer vision, object detection is widely used in many industries. Although object detection technology is relatively mature, there are still challenges in collecting data of different types and scenarios. Taking traffic light detection in autonomous driving as an example, the following are two main collection problems:

[0005] (1) Traffic lights usually only appear at intersections, resulting in a single collection scenario. In conventional collection routes, vehicles only pass through these intersections within a limited time.

[0006] (2) Traffic lights convey specific semantic information through different combinations of colors, shapes, etc. However, it is difficult to achieve balanced collection of these different types of traffic lights, especially some extreme samples, such as yellow lights in rainy, snowy and foggy weather, because their appearance is uncontrollable and their duration is short, it is almost impossible to collect a large number of them.

[0007] Therefore, the improvement of generative models in terms of precise controllability is crucial for data enhancement of target detection. Although traditional data enhancement methods, such as rotation, scaling, cropping, and color adjustment, can enrich the dataset to a certain extent, they cannot generate categories and semantic elements that do not originally exist in the dataset. Generative models can make up for this shortcoming and provide new possibilities for data enhancement. Summary of the invention

[0008] In view of the above problems, the present invention provides a controllable data generation method, system and medium based on a potential diffusion model, which not only solves the problem that some scenes and extreme samples of target detection are difficult to collect, but also introduces target ROI information and box information to solve the problem that the controllability of content generated by the diffusion model is not accurate enough.

[0009] In order to achieve the above-mentioned purpose and other related purposes, the technical solution provided by the present invention is as follows:

[0010] A controllable data generation method based on a potential diffusion model, the method comprising:

[0011] U1. Use the image encoder of the latent diffusion model to encode the noise image and the target ROI image, obtain the noise image embedding data information and ROI embedding data information, input the target box information into the MLP model for recognition, obtain the target box embedding data information, and encode the scene prompt word through the natural language editor of the latent diffusion model to obtain the prompt embedding data information;

[0012] U2. Based on the prompt embedded data information, the target box embedded data information and the ROI embedded data information, the cross-attention splicing algorithm based on adaptive learning is used to splice and embed the data to obtain the control information embedded data information;

[0013] U3. Input the data information embedded in the control information and the data information embedded in the noise image into the diffusion model backbone network model for training and learning to obtain a trained diffusion model backbone network model;

[0014] U4. Based on the trained diffusion model backbone network model, the data information embedded in the control information and the data information embedded in the noise image are input, the result image is predicted, and the predicted result image data information is obtained.

[0015] U5. Based on the predicted result image data information, a loss function G is constructed to evaluate the true value of the image to obtain the image true value evaluation data information.

[0016] Furthermore, the loss function G is:

[0017]

[0018] Among them, x is the predicted result image data information, y is the original image data information, g1 is the overall similarity loss function, g2 is the target similarity loss function, and α1, α2 and α3 are hyperparameters.

[0019] Furthermore, the constraints of the hyperparameters α1, α2 and α3 are:

[0020]

[0021] The overall similarity loss function g1 is:

[0022]

[0023] The target similarity loss function g2 is:

[0024]

[0025] Among them, x is the predicted result image data information, y is the original image data information, and a i is the target ROI embedding of the original image, b i is the target ROI embedding of the predicted result image, and n is the sample capacity.

[0026] Furthermore, the target ROI of the original image is embedded in a i The original picture corresponding to the i-th target is obtained by cropping the target box information and inputting it into the image encoder of the potential diffusion model to obtain the target ROI embedding of the original picture. The target ROI embedding of the predicted result picture is b i The predicted result image of the i-th target is cropped by cropping the target box information and input into the image encoder of the potential diffusion model to obtain the target ROI embedding of the predicted result image.

[0027] Furthermore, the method further comprises:

[0028] U6. Based on the picture truth value evaluation data information, set a preset threshold. If the picture truth value evaluation data information is less than the preset threshold, the requirement is met. If the picture truth value evaluation data information is greater than the preset threshold, the requirement is not met, and return to step U3.

[0029] Furthermore, in step U2, the cross-attention splicing algorithm based on adaptive learning is used to splice and embed the data, including:

[0030] U21. Based on the ROI embedded data information and the target box embedded data information, establish a data fusion function Q,

[0031]

[0032] Among them, z1 is the ROI embedded data information, z2 is the target box embedded data information, β1, β2 and β3 are data fusion difference factors, and the ROI embedded data and the target box embedded data are fused to obtain the fused target embedded data information;

[0033] U22. Based on the fused target embedding data information and the prompt embedding data information, a cross attention splicing embedding function W is established.

[0034]

[0035] Among them, r is the fused target embedded data information, z3 is the prompt embedded data information, δ1, δ2 and δ3 are adaptive learning factors;

[0036] U23. Based on the cross-attention splicing embedding function W, the data is spliced ​​and embedded to obtain data information in which the control information is embedded.

[0037] Furthermore, the adaptive learning factors δ1, δ2 and δ3 are,

[0038]

[0039] Among them, r is the fused target embedded data information, and z3 is the prompt embedded data information.

[0040] Further, in step U3, the step of inputting the data information embedded with the control information and the data information embedded with the noise picture into the diffusion model backbone network model for training and learning includes:

[0041] U31. Constructing a training data set based on the data information embedded in the control information and the data information embedded in the noise picture;

[0042] U32. Input the training data set into the diffusion model backbone network model for training and learning, and determine the result image prediction function S,

[0043]

[0044] Among them, h is the training data set, η1, η2 and η3 are the result image prediction factors, and the trained diffusion backbone network model is obtained.

[0045] In order to achieve the above-mentioned purpose and other related purposes, the present invention also provides a controllable data generation system based on a potential diffusion model, including a computer device, which is programmed or configured to execute any step of the controllable data generation method based on a potential diffusion model.

[0046] In order to achieve the above-mentioned purpose and other related purposes, the present invention also provides a computer-readable storage medium, on which is stored a computer program programmed or configured to execute any one of the controllable data generation methods based on the potential diffusion model.

[0047] The present invention has the following positive effects:

[0048] 1. The present invention adopts a cross-attention splicing algorithm based on adaptive learning to splice and embed data, and combines it with the diffusion model backbone network model, inputs the data information embedded in the control information and the data information embedded in the noise image, predicts the result image, and obtains the predicted result image data information, which not only expands the sample diversity of the existing data set, but also introduces target ROI information and box information to solve the problem that the controllability of the content generated by the diffusion model is not accurate enough.

[0049] 2. The present invention constructs a loss function G to evaluate the true value of the image, which not only realizes the overall conceptual control of the generated content and the fine control of the local generated target content, but also solves the problem of fusing control information from different sources. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0051] Figure 2 It is a flow chart of the cross-attention splicing algorithm based on adaptive learning of the present invention;

[0052] Figure 3 It is a flow chart of the diffusion model backbone network model of the present invention;

[0053] Figure 4 is a schematic diagram of the potential diffusion model of the present invention;

[0054] Figure 5 Schematic diagram of the loss function of the present invention. DETAILED DESCRIPTION

[0055] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0056] Example 1: Figure 1 or Figure 4 or Figure 5 As shown, a controllable data generation method based on a potential diffusion model comprises:

[0057] U1. Use the image encoder of the latent diffusion model to encode the noise image and the target ROI image, obtain the noise image embedding data information and ROI embedding data information, input the target box information into the MLP model for recognition, obtain the target box embedding data information, and encode the scene prompt word through the natural language editor of the latent diffusion model to obtain the prompt embedding data information;

[0058] U2. Based on the prompt embedded data information, the target box embedded data information and the ROI embedded data information, the cross-attention splicing algorithm based on adaptive learning is used to splice and embed the data to obtain the control information embedded data information;

[0059] U3. Input the data information embedded in the control information and the data information embedded in the noise image into the diffusion model backbone network model for training and learning to obtain a trained diffusion model backbone network model;

[0060] U4. Based on the trained diffusion model backbone network model, the data information embedded in the control information and the data information embedded in the noise image are input, the result image is predicted, and the predicted result image data information is obtained.

[0061] U5. Based on the predicted result image data information, a loss function G is constructed to evaluate the true value of the image to obtain the image true value evaluation data information.

[0062] In this embodiment, the loss function G is:

[0063]

[0064] Among them, x is the predicted result image data information, y is the original image data information, g1 is the overall similarity loss function, g2 is the target similarity loss function, and α1, α2 and α3 are hyperparameters.

[0065] In this embodiment, the constraints of the hyperparameters α1, α2 and α3 are:

[0066]

[0067] The overall similarity loss function g1 is:

[0068]

[0069] The target similarity loss function g2 is:

[0070]

[0071] Among them, x is the predicted result image data information, y is the original image data information, and a i is the target ROI embedding of the original image, b i is the target ROI embedding of the predicted result image, and n is the sample capacity.

[0072] In this embodiment, the target ROI of the original image is embedded in a i The original picture corresponding to the i-th target is obtained by cropping the target box information and inputting it into the image encoder of the potential diffusion model to obtain the target ROI embedding of the original picture. The target ROI embedding of the predicted result picture is b i The predicted result image of the i-th target is cropped by cropping the target box information and input into the image encoder of the potential diffusion model to obtain the target ROI embedding of the predicted result image.

[0073] In this embodiment, the method further includes:

[0074] U6. Based on the picture truth value evaluation data information, set a preset threshold. If the picture truth value evaluation data information is less than the preset threshold, the requirement is met. If the picture truth value evaluation data information is greater than the preset threshold, the requirement is not met, and return to step U3.

[0075] In this embodiment, the function is as follows: specify N target frame information that needs to be generated (generally expressed as pixel coordinates x, y of the center point of the outer frame, width and height w, h of the outer frame), target image samples, and text description prompts of the overall environment of the image that needs to be generated, so as to realize automatic generation of images, whose overall scene conforms to the text description prompts of the environment, and the generated image generates a target type similar to the input target image sample at the specified position of the target frame.

[0076] In this embodiment, the implementation method is as follows: first, supplementary annotation is performed based on the existing target detection dataset; the data enhancement model is trained using the supplementary annotated dataset; and customized data samples are generated using the data enhancement model and specified control information for downstream task training.

[0077] In this embodiment, a training data set is prepared: assuming that there is an existing target detection data set, the annotation information is the pixel coordinate position and size of the target outer box, each annotated image may contain multiple annotated targets, and the current image has N annotation boxes, that is, each image in the training data set needs to correspond to the center point pixel coordinates, width and height of the N annotated targets; in addition, each image is annotated with a text description prompt word for the overall environment.

[0078] In this embodiment, the model design is based on the Stable Diffusion pre-trained model, mainly including an image encoder, a natural language encoder, a diffusion model backbone network, an image decoder, and a control information encoding process, wherein the image encoder, the natural language encoder, the diffusion model backbone network, and the image decoder are usually Stable Diffusion pre-trained modules. The weights of these modules are frozen during training and prediction of this method to give full play to the powerful diversity generation ability of the diffusion model and avoid overfitting. The overall idea is to embed the generated result image through the noise image and the control information designed by this method, so that the result image satisfies: the noise image input is used to ensure the generation diversity; the target ROI image and the target box information are used to ensure that the specified target type appears at the specified position of the generated image; the overall prompt word is used to ensure the overall style and environment of the generated image.

[0079] Embodiment 2: Based on the controllable data generation method based on the potential diffusion model in Embodiment 1, the present invention is further illustrated and described below.

[0080] like Figure 1 or Figure 4 or Figure 5 As shown, a controllable data generation method based on a potential diffusion model comprises:

[0081] U1. Use the image encoder of the latent diffusion model to encode the noise image and the target ROI image, obtain the noise image embedding data information and ROI embedding data information, input the target box information into the MLP model for recognition, obtain the target box embedding data information, and encode the scene prompt word through the natural language editor of the latent diffusion model to obtain the prompt embedding data information;

[0082] U2. Based on the prompt embedded data information, the target box embedded data information and the ROI embedded data information, the cross-attention splicing algorithm based on adaptive learning is used to splice and embed the data to obtain the control information embedded data information;

[0083] U3. Input the data information embedded in the control information and the data information embedded in the noise image into the diffusion model backbone network model for training and learning to obtain a trained diffusion model backbone network model;

[0084] U4. Based on the trained diffusion model backbone network model, the data information embedded in the control information and the data information embedded in the noise image are input, the result image is predicted, and the predicted result image data information is obtained.

[0085] U5. Based on the predicted result image data information, a loss function G is constructed to evaluate the true value of the image to obtain the image true value evaluation data information.

[0086] In this embodiment, in step U2, the use of the cross-attention splicing algorithm based on adaptive learning to splice and embed the data includes:

[0087] U21. Based on the ROI embedded data information and the target box embedded data information, establish a data fusion function Q,

[0088]

[0089] Among them, z1 is the ROI embedded data information, z2 is the target box embedded data information, β1, β2 and β3 are data fusion difference factors, and the ROI embedded data and the target box embedded data are fused to obtain the fused target embedded data information;

[0090] U22. Based on the fused target embedding data information and the prompt embedding data information, establish a cross-attention splicing embedding function W,

[0091]

[0092] Among them, r is the fused target embedded data information, z3 is the prompt embedded data information, δ1, δ2 and δ3 are adaptive learning factors;

[0093] U23. Based on the cross-attention splicing embedding function W, the data is spliced ​​and embedded to obtain data information in which the control information is embedded.

[0094] In this embodiment, the adaptive learning factors δ1, δ2 and δ3 are,

[0095]

[0096] Among them, r is the fused target embedded data information, and z3 is the prompt embedded data information.

[0097] In this embodiment, the noise image is generated by random sampling without manual preparation. By inputting the target ROI image sample to be generated, the box information of the target specified generation, and the overall environment prompt words, the image is automatically generated so that the box specified position can be controlled to generate a target image with the same concept and position as the sample target.

[0098] In this embodiment, in step U3, inputting the data information embedded with the control information and the data information embedded with the noise picture into the diffusion model backbone network model for training and learning includes:

[0099] U31. Constructing a training data set based on the data information embedded in the control information and the data information embedded in the noise picture;

[0100] U32. Input the training data set into the diffusion model backbone network model for training and learning, and determine the result image prediction function S,

[0101]

[0102] Among them, h is the training data set, η1, η2 and η3 are the result image prediction factors, and the trained diffusion backbone network model is obtained.

[0103] In this embodiment, the present invention provides a controllable data generation system based on a potential diffusion model, including a computer device, which is programmed or configured to execute any one of the steps of the controllable data generation method based on a potential diffusion model.

[0104] In this embodiment, the present invention provides a computer-readable storage medium, on which is stored a computer program programmed or configured to execute any one of the controllable data generation methods based on a potential diffusion model.

[0105] Any reference to memory, storage, database or other media used in the embodiments provided in the present application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM).

[0106] In summary, the present invention not only solves the problem that some scenes and extreme samples of target detection are difficult to collect, but also introduces target ROI information and box information to solve the problem that the controllability of content generated by the diffusion model is not accurate enough.

[0107] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A controllable data generation method based on a potential diffusion model, characterized in that: The method comprises: U1. Use the image encoder of the latent diffusion model to encode the noise image and the target ROI image, obtain the noise image embedding data information and ROI embedding data information, input the target box information into the MLP model for recognition, obtain the target box embedding data information, and encode the scene prompt word through the natural language editor of the latent diffusion model to obtain the prompt embedding data information; U2. Based on the prompt embedded data information, the target box embedded data information and the ROI embedded data information, the cross-attention splicing algorithm based on adaptive learning is used to splice and embed the data to obtain the control information embedded data information; U3. Input the data information embedded in the control information and the data information embedded in the noise image into the diffusion model backbone network model for training and learning to obtain a trained diffusion model backbone network model; U4. Based on the trained diffusion model backbone network model, the data information embedded in the control information and the data information embedded in the noise image are input, the result image is predicted, and the predicted result image data information is obtained. U5. Based on the predicted result image data information, a loss function G is constructed to evaluate the true value of the image to obtain the image true value evaluation data information.

2. The controllable data generation method based on the potential diffusion model according to claim 1, characterized in that: The loss function G is, Among them, x is the predicted result image data information, y is the original image data information, g1 is the overall similarity loss function, g2 is the target similarity loss function, and α1, α2 and α3 are hyperparameters.

3. The controllable data generation method based on the potential diffusion model according to claim 2 is characterized in that: The constraints of the hyperparameters α1, α2 and α3 are: The overall similarity loss function g1 is: The target similarity loss function g2 is: Among them, x is the predicted result image data information, y is the original image data information, and a i is the target ROI embedding of the original image, b i is the target ROI embedding of the predicted result image, and n is the sample capacity.

4. The controllable data generation method based on the potential diffusion model according to claim 3 is characterized in that: The target ROI of the original image is embedded in a i The original picture corresponding to the i-th target is obtained by cropping the target box information and inputting it into the image encoder of the potential diffusion model to obtain the target ROI embedding of the original picture. The target ROI embedding of the predicted result picture is b i The predicted result image of the i-th target is cropped by cropping the target box information and input into the image encoder of the potential diffusion model to obtain the target ROI embedding of the predicted result image.

5. The controllable data generation method based on the potential diffusion model according to claim 1 is characterized in that: The method further comprises: U6. Based on the picture truth value evaluation data information, set a preset threshold. If the picture truth value evaluation data information is less than the preset threshold, the requirement is met. If the picture truth value evaluation data information is greater than the preset threshold, the requirement is not met, and return to step U3.

6. The controllable data generation method based on the potential diffusion model according to claim 1 is characterized in that: In step U2, the cross-attention splicing algorithm based on adaptive learning is used to splice and embed the data, including: U21. Based on the ROI embedded data information and the target box embedded data information, establish a data fusion function Q, Among them, z1 is the ROI embedded data information, z2 is the target box embedded data information, β1, β2 and β3 are data fusion difference factors, and the ROI embedded data and the target box embedded data are fused to obtain the fused target embedded data information; U22. Based on the fused target embedding data information and the prompt embedding data information, establish a cross-attention splicing embedding function W, Among them, r is the fused target embedded data information, z3 is the prompt embedded data information, δ1, δ2 and δ3 are adaptive learning factors; U23. Based on the cross-attention splicing embedding function W, the data is spliced ​​and embedded to obtain data information in which the control information is embedded.

7. The controllable data generation method based on the potential diffusion model according to claim 6 is characterized in that: The adaptive learning factors δ1, δ2 and δ3 are, Among them, r is the fused target embedded data information, and z3 is the prompt embedded data information.

8. The controllable data generation method based on the potential diffusion model according to claim 1 is characterized in that: In step U3, inputting the data information embedded with the control information and the data information embedded with the noise picture into the diffusion model backbone network model for training and learning includes: U31. Constructing a training data set based on the data information embedded in the control information and the data information embedded in the noise picture; U32. Input the training data set into the diffusion model backbone network for training and learning, and determine the result image prediction function S, Among them, h is the training data set, η1, η2 and η3 are the result image prediction factors, and the trained diffusion backbone network is obtained.

9. A controllable data generation system based on a potential diffusion model, comprising a computer device, characterized in that: The computer device is programmed or configured to execute the steps of the controllable data generation method based on the potential diffusion model as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program programmed or configured to execute the controllable data generation method based on a potential diffusion model as described in any one of claims 1 to 8.

Images