Method for carrying out data enhancement by utilizing text prompt and image generation technology

Abstract

The invention discloses a method for carrying out data enhancement by utilizing a text prompt and image generation technology, which is used for automatically generating a new image meeting task requirements, reducing the dependence on manual annotation and reducing the data annotation cost. By generating diversified background and target object images, the diversity of training data is increased, and the data enhancement process is simplified, so that the robustness and generalization ability of the model are remarkably improved. The method is not only suitable for target detection and instance segmentation, but also can be expanded and applied to various computer vision tasks such as image classification and semantic segmentation, and is helpful for promoting the development of the computer vision field.

Description

Technical Field

[0001] The present invention relates to a method for data enhancement, and more particularly to a method for data enhancement using text prompts and image generation technology. Background Art

[0002] Image augmentation technology plays a vital role in the field of computer vision, providing models with richer and more diverse training data, thereby improving their performance and generalization capabilities.

[0003] Therefore, in the current prior art, there is an invention patent with patent number CN202410218732.1, which is named a text-driven data enhancement method. It discloses that the image enhancement effect is achieved by extracting visual features from the source image, converting the visual features into a text description sequence, and then repeating the text description sequence by constructing a corpus. Then, new image data is generated based on the repeated text description sequence. However, the above method has problems such as insufficient data, high annotation cost, and poor model generalization ability. Summary of the invention

[0004] In view of the deficiencies in the prior art, the object of the present invention is to provide a method for data enhancement using text prompts and image generation technology that can solve one or more of the above-mentioned problems.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for data enhancement using text prompts and image generation technology, characterized in that it includes the following steps:

[0006] Step 1: Design text prompts and mask regions based on the original image to guide the generative model to generate new image content consistent with the existing dataset annotations through text prompts and specify the scope of image restoration through mask regions;

[0007] Step 2: Use the diffusion model to generate high-quality images by gradually adding and removing noise, thereby completing data enhancement for the image.

[0008] As a further improvement of the present invention, the text prompt in step 1 is designed based on the target task and includes descriptions about image style, lighting and other details.

[0009] As a further improvement of the present invention, the mask area design in step 1 is calculated by the following formula:

[0010]

[0011] Wherein, M(x,y) is the mask region matrix, (x,y) is the image coordinate, and targetregion is the target region. As a further improvement of the present invention, the specific steps of using the diffusion model in step 2 to generate a high-quality image by gradually adding and removing noise are as follows:

[0012] Step 21, adding initial noise to the input image;

[0013] Step 22: In the diffusion model, the image is denoised through a multi-step iterative diffusion process, and the image is gradually restored from an initial noisy state to a clear image;

[0014] In the denoising and denoising processes of step 21 and step 22, the image noise is updated only in the mask area to ensure that the generated image content meets the requirements of the target area.

[0015] As a further improvement of the present invention, the specific method of adding the initial noise in step 21 is: adding initial noise to the input image, the initial noise is generated by random sampling, and added to the original image, the formula is as follows:

[0016] χ0=χ original +N(0,σ 2 )

[0017] Among them, χ0 is the initial noisy image, χ original is the original image, N(0,σ 2 ) has a mean of 0 and a variance of σ 2 Gaussian noise.

[0018] As a further improvement of the present invention, the specific steps of gradually restoring the image from the initial noisy state to a clear image in step 22 are as follows:

[0019] Step 221, add a small amount of noise to the current image state, and combine the text prompt to guide the generated image;

[0020] Step 222: After the noise is added in step 221, the noise is removed by learning the inverse process, and the clarity of the image is gradually restored with the help of text prompts.

[0021] As a further improvement of the present invention, the noise adding formula in step 221 is as follows:

[0022]

[0023] Among them, χ t is the current image state, χ t-1 is the image state of the previous step, β t-1 is the noise intensity parameter in step t-1, is random noise,∈(s,χ t-1 ) is the noise signal of the text prompt, and α is the guidance parameter used to adjust the weights of random noise and text prompt noise.

[0024] As a further improvement of the present invention, the denoising process of removing noise in step 222 is implemented by a neural network of a diffusion model, and the denoising formula is as follows:

[0025] χ t = x t+1 -γ·f θ (x t+1 ,t)-(1-γ)·∈(s,χ t+1 )

[0026] Among them, χ t is the denoised image of the current step, χ t+1 is the noisy image of the previous step, f θ is a neural network of a diffusion model, t is the current step, s is the text prompt, and γ is a parameter for controlling the weight of denoising and text prompt. As a further improvement of the present invention, the updating of image noise in the mask area includes mask area noise addition update and mask area noise reduction update, and the specific formula is as follows:

[0027] The mask area noise update formula is as follows:

[0028] ∈′(s,χ t )=M⊙∈(s,χ t )+(1-M)⊙(χ t -x t-1 )

[0029] Where M is the mask region matrix, ⊙ is the element operation, and χ t is the image of the current step, χ t-1 is the image of the previous step;

[0030] The mask area denoising update formula is as follows:

[0031] ∈′(s,χ t )=M⊙∈(s,χ t )+(1-M)⊙(χ t -x t+1 )

[0032] Where M is the mask region matrix, ⊙ is the element operation, and χ t is the image of the current step, χ t+1 is the image from the previous step.

[0033] As a further improvement of the present invention, the step of generating a high-quality image by gradually adding and removing noise using a diffusion model also includes a step of optimizing the diffusion model, which specifically includes the following:

[0034]

[0035] Among them, θ is the parameter of the diffusion model, χ t is the denoised image of the current step, χ t+1 is the noisy image of the previous step, s is the text prompt, and t is the current step.

[0036] The beneficial effect of the present invention is that by using text-to-image generation technology, the present invention automatically generates new images that meet the task requirements, reduces the reliance on manual annotation, and reduces the cost of data annotation. By generating diverse background and target object images, the diversity of training data is increased, the data enhancement process is simplified, and the robustness and generalization ability of the model are significantly improved. This method is not only applicable to target detection and instance segmentation, but can also be extended to various computer vision tasks such as image classification and semantic segmentation, which helps to promote the development of the field of computer vision. Therefore, the method of the present invention:

[0037] 1. No additional annotation is required: New images generated by the generative model do not require additional annotation, which reduces the cost of data enhancement.

[0038] 2. Improve model performance: Through background enhancement, the robustness and generalization ability of object detection and instance segmentation models are significantly improved.

[0039] 3. Strong versatility: This method is applicable to a variety of target detection and instance segmentation tasks and has broad application prospects. DETAILED DESCRIPTION

[0040] The present invention will be further described in detail with reference to the given embodiments below.

[0041] This embodiment uses a method of data enhancement using text prompts and image generation technology, which reasonably designs text prompts and mask areas and uses a diffusion model (such as Stable Diffusion) to generate new image content, thereby achieving efficient data enhancement. The specific implementation steps are as follows:

[0042] Text hint and mask area design

[0043] Text prompt design

[0044] Textual prompts are used to guide generative models to generate new image content that is consistent with existing dataset annotations. Textual prompts are designed based on the target task, such as "generate a clean background" or "generate a new target object", and can contain descriptions about image style, lighting, and other details.

[0045] For example, when enhancing an image containing a car, the text hint could be "car on a city street" to generate an image that fits the scene. The text hint not only describes the content of the image, but also describes specific attributes of the image, such as color, position, size, etc.

[0046] Mask area design

[0047] The mask area is used to specify the range of image restoration. When performing background enhancement, the mask area covers all background parts except the target object; when performing target object enhancement, the mask area only covers the target object part. The mask area can be manually marked or generated by an automated algorithm to ensure the accuracy of the coverage area.

[0048] The formula for generating the mask area is as follows:

[0049]

[0050] Among them, M(x,y) is the mask region matrix, (x,y) is the image coordinate, and targetregion is the target region.

[0051] 2. Diffusion steps

[0052] The diffusion step of the present invention is to generate a high-quality image by gradually adding and removing noise using a diffusion model. This process can be broken down into the following key steps:

[0053] ① Initial noise generation

[0054] First, add initial noise to the input image. The initial noise is generated by random sampling and added to the original image. The formula is as follows:

[0055] χ0=χ original +N(0,σ 2 )

[0056] Among them, χ0 is the initial noisy image, χ original is the original image, N(0,σ 2 ) has a mean of 0 and a variance of σ 2 Gaussian noise.

[0057] ②Multi-step iterative diffusion process

[0058] In the diffusion model, the image is gradually restored from the initial noisy state to a clear image. This process is achieved through multiple iterations, including two stages: noise addition and removal.

[0059] Noise adding stage

[0060] Add a small amount of noise to the current image state and combine it with the text prompt to guide the generated image. The noise addition formula is as follows:

[0061]

[0062] Among them, χ t is the current image state, χ t-1 is the image state of the previous step, β t-1 is the noise intensity parameter in step t-1, is random noise,∈(S,χ t-1 ) is the noise signal of the text prompt, and α is the guidance parameter used to adjust the weights of random noise and text prompt noise.

[0063] Noise removal stage

[0064] After the noise is added, the noise is removed by learning the inverse process, and the clarity of the image is gradually restored with the help of text prompts. The denoising process is implemented by the neural network of the diffusion model, and the denoising formula is as follows:

[0065] χ t = x t+1 -γ·f θ (x t+1 ,t)-(1-γ)·∈(s,χ t+1 )

[0066] Among them, χ t is the denoised image of the current step, χ t+1 is the noisy image of the previous step, f θ is the neural network of the diffusion model, t is the current step, s is the text prompt, and γ is the parameter controlling the weights of denoising and text prompt.

[0067] ③. Noise update of mask area

[0068] In each step of denoising and denoising, the image noise is updated only in the masked area to ensure that the generated image content meets the requirements of the target area.

[0069] The mask area noise update formula is as follows:

[0070] ∈′(s,χ t )=M⊙∈(s,χ t )+(1-M)⊙(χ t -x t-1 )

[0071] Where M is the mask region matrix, ⊙ is the element operation, and χ t is the image of the current step, χ t-1 is the image from the previous step.

[0072] The mask area denoising update formula is as follows:

[0073] ∈′(s,χ t )=M⊙∈(s,χ t )+(1-N)⊙(χ t -x t+1 )

[0074] Where M is the mask region matrix, ⊙ is the element operation, and χ t is the image of the current step, χ t+1 is the image from the previous step.

[0075] ④. Diffusion model optimization

[0076] In order to generate high-quality images, the parameters of the diffusion model need to be optimized, including the noise intensity parameter β t , guidance parameter α, denoising parameter γ, etc. Through experiments and tuning, the optimal parameter combination is selected to generate high-quality images that meet the text prompts. The specific optimization goals are as follows:

[0077]

[0078] Among them, θ is the parameter of the diffusion model, χ t is the denoised image of the current step, χ t+1 is the noisy image of the previous step, s is the text prompt, and t is the current step.

[0079] The present invention uses a diffusion model to generate high-quality images that conform to textual prompts, achieving efficient data augmentation. This process ensures that the generated images are of high quality and naturalness, while increasing the diversity of training data and improving the robustness and generalization ability of the model.

[0080] The following examples are provided in this embodiment:

[0081] Example 1: Background enhancement

[0082] 1. Input image: Select a training image containing the target object.

[0083] 2. Text prompt: Use the text prompt "Generate a clean background".

[0084] 3. Mask area design: Generate a background mask area based on the target object in the image to ensure that only the background part is modified.

[0085] 4. Diffusion step: Using the diffusion model, a new background image is generated through multi-step diffusion.

[0086] 5. Generate results: The new background image is fused with the original image, the target object is retained, and an enhanced training image is generated.

[0087] Example 2: Target object enhancement

[0088] 1. Input image: Select a training image containing the target object.

[0089] 2. Text prompt: Use the target object category as the text prompt.

[0090] 3. Mask area design: Generate the target object mask area according to the target object in the image.

[0091] 4. Diffusion step: Using the diffusion model, a new target object image is generated through multi-step diffusion.

[0092] Generation result: The new target object image is fused with the original image to generate an enhanced training image.

[0093] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A method for data enhancement using text prompts and image generation technology, characterized in that: The steps include: Step 1: Design text prompts and mask regions based on the original image to guide the generative model to generate new image content consistent with the existing dataset annotations through text prompts and specify the scope of image restoration through mask regions; Step 2: Use the diffusion model to generate high-quality images by gradually adding and removing noise, thereby completing data enhancement for the image.

2. The method for data enhancement using text prompts and image generation technology according to claim 1, characterized in that: The text prompts in step 1 are designed based on the target task and contain descriptions about the image style, lighting, and other details.

3. The method for data enhancement using text prompt and image generation technology according to claim 1 or 2, characterized in that: The mask area design in step 1 is calculated by the following formula: Among them, M(x,y) is the mask region matrix, (x,y) is the image coordinate, and targetregion is the target region.

4. The method for data enhancement using text prompt and image generation technology according to claim 1 or 2, characterized in that: The specific steps of using the diffusion model in step 2 to generate a high-quality image by gradually adding and removing noise are as follows: Step 21, adding initial noise to the input image; Step 22: In the diffusion model, the image is denoised through a multi-step iterative diffusion process, and the image is gradually restored from an initial noisy state to a clear image; In the denoising and denoising processes of step 21 and step 22, the image noise is updated only in the mask area to ensure that the generated image content meets the requirements of the target area.

5. The method for data enhancement using text prompts and image generation technology according to claim 4, characterized in that: The specific method of adding the initial noise in step 21 is: Add initial noise to the input image. The initial noise is generated by random sampling and added to the original image. The formula is as follows: x0=x originsl +N(0,σ 2 ) Among them, χ0 is the initial noisy image, χ originsl is the original image, N(0,σ 2 ) has a mean of 0 and a variance of σ 2 Gaussian noise.

6. The method for data enhancement using text prompts and image generation technology according to claim 4, characterized in that: The specific steps of gradually restoring the image from the initial noisy state to a clear image in step 22 are as follows: Step 221, add a small amount of noise to the current image state, and combine the text prompt to guide the generated image; Step 222: After the noise is added in step 221, the noise is removed by learning the inverse process, and the clarity of the image is gradually restored with the help of text prompts.

7. The method for data enhancement using text prompts and image generation technology according to claim 6, characterized in that: The noise adding formula in step 221 is as follows: Among them, χ t is the current image state, χ t-1 is the image state of the previous step, β t-1 is the noise intensity parameter in step t-1, is random noise, ∈(s, χ t-1 ) is the noise signal of the text prompt, and α is the guidance parameter used to adjust the weights of random noise and text prompt noise.

8. The method for data enhancement using text prompts and image generation technology according to claim 6, characterized in that: The denoising process of removing noise in step 222 is implemented by a neural network of a diffusion model, and the denoising formula is as follows: x t =x t+1 -γ·f θ (x t+1 ,t)-(1-γ)·∈(s,X t+1 ) Among them, χ t is the denoised image of the current step, χ t+1 is the noisy image of the previous step, f θ is the neural network of the diffusion model, t is the current step, s is the text prompt, and γ is the parameter controlling the weights of denoising and text prompt.

9. The method for data enhancement using text prompts and image generation technology according to claim 4, characterized in that: The updating of the image noise in the mask area includes the mask area noise addition update and the mask area noise reduction update, and the specific formula is as follows: The mask area noise update formula is as follows: ∈′(s, x t )=M⊙∈(s,χ t )+(1-M)⊙(χ t -x t-1 ) Where M is the mask region matrix, ⊙ is the element operation, and χ t is the image of the current step, χ t-1 is the image of the previous step; The mask area denoising update formula is as follows: ∈′(s,χ t )=M⊙∈(s,χ t )+(1-M)⊙(χ t -x t+1 ) Where M is the mask region matrix, ⊙ is the element operation, and χ t is the image of the current step, χ t+1 is the image from the previous step.

10. The method for data enhancement using text prompts and image generation technology according to claim 4, characterized in that: The step of using the diffusion model to generate a high-quality image by gradually adding and removing noise also includes a diffusion model optimization step, which specifically includes the following: Among them, θ is the parameter of the diffusion model, χ t is the denoised image of the current step, χ t+1 is the noisy image of the previous step, s is the text prompt, and t is the current step.

Images