Data enhancement method and system based on diffusion model

Abstract

The invention relates to a data enhancement method and system based on a diffusion model, and the method comprises the steps: providing a guide source based on an original fine-grained data set, and the guide source comprises a text prompt, a contour map and a fine-grained classifier; and generating an enhanced sample through a diffusion model using the pilot signal from the pilot source; wherein the fine-grained classifier is obtained by training the fine-grained data set, and the contour map is extracted from an original image sample in the fine-grained data set. According to the scheme, multiple guide signals can be coordinated, and high fidelity of fine-grained category features is strictly maintained while diversity of the generated samples is ensured, so that the performance of a finally trained fine-grained visual classification model is effectively improved.

Description

Technical Field

[0001] This application relates to data augmentation techniques in the field of deep learning, and in particular to methods and systems for data augmentation using diffusion models. Background Technology

[0002] The performance of deep learning models is highly dependent on large-scale, diverse, and accurately labeled datasets. However, acquiring such datasets is costly and time-consuming, while data augmentation techniques are a key means to alleviate data scarcity and improve the generalization ability of models. Traditional data methods typically involve performing image processing operations such as geometric transformations and color jittering on original image samples to generate new samples to increase the training data for the model. However, the data diversity provided is extremely limited, making it difficult to effectively improve the generalization ability of the model.

[0003] Generative models, particularly diffusion models, offer a promising alternative for generating novel and realistic data samples. Data augmentation methods based on diffusion models can guide the generation of data samples by setting guiding conditions, thus significantly increasing data diversity and improving model generalization compared to traditional image processing-based data augmentation methods. Summary of the Invention

[0004] However, in their research and practice using deep learning models for fine-grained visual classification and other specialized tasks, the inventors discovered a core problem with existing diffusion model-based data augmentation methods when applied to such tasks: insufficient fidelity in the generated augmented samples. For fine-grained visual classification, the differences between different fine-grained categories often lie in extremely subtle features, such as the shape of a specific bird's beak, the precise texture of an insect's wings, or subtle design variations in a car model. The inventors found that existing diffusion model-based data augmentation methods all employ text- or contour map-based guidance mechanisms to guide the generation of data samples, making it difficult to accurately capture and reproduce these crucial details that determine category classification. For example, when generating an image of a "red-winged blackbird," failure to accurately depict its distinctive red shoulder feathers, or deviations in color or shape, introduces semantic mismatches, thereby reducing the performance of the subsequently trained classifier. This lack of fidelity severely limits the effectiveness of diffusion model-based data augmentation in fine-grained visual classification scenarios.

[0005] In view of this, this application aims to provide a new data augmentation scheme based on a diffusion model, which can intelligently coordinate multiple guiding signals to ensure the diversity of generated samples while strictly maintaining high fidelity of fine-grained category features, thereby helping to improve the performance of the final trained fine-grained visual classification model, and thus can be widely applied to fine-grained visual classification application scenarios.

[0006] According to a first aspect of the embodiments of this application, a data augmentation method based on a diffusion model is proposed, comprising: providing a guidance source based on an original fine-grained dataset, the guidance source including a text prompt, a contour map, and a fine-grained classifier; and generating augmented samples using a guidance signal from the guidance source through a diffusion model; wherein the text prompt includes descriptive information for each category in the fine-grained dataset, the fine-grained classifier is used to predict the confidence level of an image category and is trained based on the fine-grained dataset, and the contour map is extracted from original image samples in the fine-grained dataset.

[0007] In this embodiment, by coordinating the three different levels of signal sources, both coarse and fine control are achieved in the process of augmented sample generation. Text prompts and contour maps are responsible for constructing the "skeleton" and "scene" of the image to ensure the global rationality and diversity of the generated augmented samples; while the fine-grained classifier guide is responsible for refining the "flesh" and "details" to ensure the accuracy of the category attributes of the fine-grained classification of the generated augmented samples.

[0008] In some embodiments, generating enhanced samples includes denoising a noisy image using a denoising process of a diffusion model to generate enhanced samples, wherein the denoising process includes a first stage and a second stage: in the first stage, guidance is provided using guidance signals from the text prompt and the contour map; and in the second stage, guidance is provided using guidance signals from the text prompt, the contour map, and the fine-grained classifier.

[0009] In this embodiment, the process of generating augmented samples is divided into two stages, each using different guiding signals. In the first stage, a combination of guiding signals from text prompts and contour maps is used to guide the generation of the augmented samples, establishing their global structure and style. In the second stage, a combination of guiding signals from text prompts, contour maps, and a fine-grained classifier is used to refine image details and ensure class fidelity. Furthermore, for example, with the same total number of time steps, the number of time steps in each of the first and second stages can be dynamically adjusted. For instance, reducing the number of time steps in the first stage and increasing the number of time steps in the second stage can appropriately reduce the diversity of the augmented samples while further improving their detail fidelity.

[0010] In some embodiments, the guiding signal provided to the denoising network during the denoising process is a weighted combination of guiding signals from various guiding sources and their corresponding weights, thereby changing the influence of each guiding signal on the generated enhanced image by adjusting the weights of the three guiding signal intensities.

[0011] In some embodiments, in the first stage, the weights of the guiding signals from the text prompt and the contour map remain unchanged; and in the second stage, the weights of the guiding signals from the text prompt, the contour map, and the fine-grained classifier are dynamically adjusted based on the prediction confidence of the fine-grained classifier's output of the denoising network. This dynamic adjustment mechanism intelligently balances fidelity and diversity. It avoids generation failures caused by prematurely introducing detail constraints before the image structure is fully formed, while providing a "tailor-made" guiding strategy for each sample through adaptive adjustment based on the results of the fine-grained classifier (i.e., prediction confidence). This not only improves the overall quality and stability of the generated enhanced samples but also effectively prevents mode collapse or artifacts caused by over-guidance.

[0012] In some embodiments, the dynamic adjustment in the second phase includes: for the current time step of the second phase... Get the current noisy image Corresponding estimated clear image ,Will Input this fine-grained classifier to obtain the target category. Prediction confidence Based on prediction confidence Calculate the time step for each time step. The weight of the guiding signal is denoted as , , ,in, In time step The weight of the guiding signal from the text prompt. In time step The weight of the guiding signal from the contour map, In time step The weights of the guiding signal from this fine-grained classifier. The pre-defined weights of the guiding signals from this text prompt. The weights of the guiding signals from this contour map are preset. The weights are the preset guiding signals from the fine-grained classifier.

[0013] In this embodiment, the fine-grained classifier is activated in the second stage of generating enhanced samples, and the relative weights of the three guiding signals are dynamically adjusted based on the prediction confidence of the fine-grained classifier for the currently generated intermediate results. When the prediction confidence is high, the weight of the guiding signal from the classifier is reduced, allowing the guiding signals for text and contours to play a greater role in enhancing the overall diversity of the samples and avoiding overfitting; while when the prediction confidence is low, the weight of the classifier's guiding signal is increased to correct for sample details.

[0014] In some embodiments, at the current time step of the second phase The guiding signal calculated from the fine-grained classifier is based on the current noisy image. Corresponding estimated clear image This is obtained by calculating the gradient, denoted as . ,in, For the target category from the fine-grained classifier The prediction confidence level.

[0015] When a fine-grained classifier is used as the guiding source, the gradient guided by that classifier can be obtained to calculate the guiding signal from it. In a classifier-independent guiding diffusion model, the noise prediction is the noise predicted by the diffusion model. When a classifier is introduced as a guide, the noise prediction becomes the noise predicted by the diffusion model. Gradient guided by classifier The sum of these factors adjusts the noise prediction of the diffusion model, guiding the generation process of enhanced samples in a direction that better reflects the characteristics of the target category.

[0016] In some embodiments, the text prompt may include one or more of a category name, background, scene, and art style.

[0017] In some embodiments, the contour map may be extracted from raw image samples in the original fine-grained dataset, for example, using the Canny operator.

[0018] In some embodiments, the method may further perform a contour transformation on the contour map before guiding the image, to enhance the pose and morphological diversity of the augmented sample. Such a contour transformation may include one or more of the following: applying an affine transformation to the contour map when the original image sample corresponding to the contour map is a rigid object; and applying an affine transformation and / or thin-plate spline interpolation distortion to the contour map when the original image sample corresponding to the contour map is a non-rigid object.

[0019] According to a second aspect of the embodiments of this application, a training method for a fine-grained visual classification model is proposed, comprising: generating one or more augmented samples according to the data augmentation method of the embodiments of this application for each original image sample or a portion of the original image samples in the original fine-grained training set; mixing the generated augmented samples with the original image samples of the original fine-grained training set to form an augmented fine-grained training set; training the fine-grained visual classification model using the augmented fine-grained training set, or fine-tuning a pre-trained fine-grained classifier and using the fine-tuned fine-grained classifier as the final fine-grained visual classification model, wherein the fine-grained classifier is pre-trained on the original fine-grained training set.

[0020] In this embodiment, the generated augmented samples balance diversity and high fidelity, resulting in an enhanced fine-grained training set that possesses sufficient diversity while ensuring the accuracy of category attributes. This augmented training set can improve the performance and generalization of the trained fine-grained visual classification model.

[0021] According to a third aspect of the embodiments of this application, a computer-readable storage medium is also provided, having stored thereon computer instructions that, when executed by a processor, implement the methods described according to the first and second aspects of the embodiments of this application.

[0022] According to a fourth aspect of the embodiments of this application, a data augmentation system based on a diffusion model is also proposed, comprising a guidance source module and a sample generation module, wherein: the guidance source module is configured to provide a guidance source based on an original fine-grained dataset, the guidance source including text prompts, contour maps, and a fine-grained classifier; and the sample generation module is configured to generate augmented samples using guidance signals from the guidance source through a diffusion model; wherein the text prompts include descriptive information for each category in the fine-grained dataset, the fine-grained classifier is used to predict the confidence level of the image category, which is obtained by training on the fine-grained dataset, and the contour map is extracted from original image samples in the fine-grained dataset.

[0023] Compared with existing technologies, the embodiments of this application synergistically integrate three guiding sources: text, transformed contours, and fine-grained classifiers, and combine them with dynamic intensity modulation based on generation timing and classifier confidence. This enables the generation of diverse images while accurately preserving subtle features that are crucial for category recognition, thereby significantly improving the performance of downstream fine-grained visual classification models. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0025] Figure 1 A schematic flowchart of a data augmentation method based on a diffusion model according to an embodiment of this application is shown;

[0026] Figure 2 A schematic flowchart illustrating a data augmentation method based on a diffusion model, representing another embodiment of this application, is shown.

[0027] Figure 3A schematic flowchart of a data augmentation system based on a diffusion model, according to an embodiment of this application, is shown; and

[0028] Figure 4 A schematic flowchart of a diffusion-based data augmentation system according to other embodiments of this application is shown. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description, in conjunction with the accompanying drawings, provides specific embodiments. It should be understood that the described embodiments are only a portion, not all, of the embodiments described herein. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0030] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0031] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0032] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0033] Fine-grained visual classification refers to the task of identifying and classifying subcategories within the same broad category in computer vision. Examples include distinguishing different breeds of dogs, different types of flowers, different models of cars, and different models of airplanes. Typical examples include classifying images from the classic CUB200-2011 dataset (containing 11,788 bird images across 200 subcategories), identifying and classifying images of California gulls and Arctic gulls, and identifying and classifying fabric textures. Because the visual differences between different subcategories are very small, while individuals within the same subcategory can vary significantly due to factors such as pose, lighting, and background—that is, inter-class differences are small but intra-class variations are large—this task is more challenging than coarse-grained visual classification. Therefore, maintaining the diversity and high fidelity of training samples is particularly important when generating training samples.

[0034] Diffusion models are a type of generative model that has achieved significant results in recent years in various fields such as image generation, audio generation, and video generation. They generate data by progressively removing noise, similar to a process from blurry to sharp. Training a diffusion model involves iteratively adding noise to an image and then training a deep neural network (i.e., a denoising network, such as U-Net) to learn how to recover the original image from the noisy data. The application of diffusion models starts with Gaussian noise and gradually denoises to ultimately generate sharp data.

[0035] Data augmentation methods based on diffusion models typically utilize a pre-trained denoising network within the diffusion model to generate images from the input Gaussian noise. This performs the inverse denoising process of the diffusion model. This denoising process generates Gaussian noise from the input randomly. We'll start by gradually reducing the noise. At each time step... , to the current noise data The input is fed into a trained denoising network, which then adjusts the noise based on the input noise data and the current time step. Information, outputting the currently estimated clear image data. Then use the output of the denoising network. As input for the next time step The denoising process continues. This iterative process will continue until the time step... A value of 1 indicates the completion of the entire reverse denoising process. This process is based on the knowledge learned by the denoising network during the training phase; it can identify features in noisy data and gradually recover clear data content. After denoising, a clear image is finally obtained. If it is necessary to generate images that meet specific conditions, conditional signals can be incorporated into the denoising process through conditional guidance. For example, in each denoising step, conditional signals (such as text embedding vectors, contour information, etc.) are provided as additional input to the denoising network to guide the generation process. In conditional guidance, the guidance strength can be controlled by adjusting the weights of each conditional signal.

[0036] In this application, guidance signals from multiple guidance sources are used to generate new images from noisy images through a denoising process involving several time steps using a trained diffusion model. Different guidance sources may have different roles. For example, text prompts can define the global scene, style, and coarse-grained object attributes of the generated image, driving the diversity of generated samples. These can include category names, backgrounds, scenes, and artistic styles (such as "photograph" or "oil painting"). It can be encoded using a pre-trained language model (such as CLIP or BERT) and then combined with the noise predictor of the diffusion model (such as U-Net). Contour maps can extract edge information from the original image samples and serve as auxiliary inputs to control the image structure or shape during image generation. Their core role is to guide the diffusion model to generate images that meet specific geometric or layout requirements. Contour maps generated using the Canny operator can be used to ensure real-time and efficient detection of strong edges in the image. In addition to the multimodal guidance combining text prompts and contour maps, this application also introduces a fine-grained classifier for guidance. This fine-grained classifier is pre-trained using the original fine-grained dataset and accurately identifies the target category of the input image. This fine-grained classifier can provide gradients as guiding signals to adjust the direction of image generation.

[0037] Figure 1 A schematic flowchart illustrating a data augmentation method based on a diffusion model according to an embodiment of this application is shown. Figure 1 As shown, the data augmentation method 100 based on the diffusion model mainly includes: S110, providing a guiding source based on the original fine-grained dataset, the guiding source including text prompts, contour maps and fine-grained classifiers; and S120, generating augmented samples using the guiding signal from the guiding source through the diffusion model.

[0038] Specifically, in step S110, three guidance sources are prepared based on the original fine-grained dataset, including text prompts, contour maps, and a fine-grained classifier. Guidance signals from these three guidance sources are subsequently provided as guidance conditions to the trained diffusion model to guide the generation of augmented samples.

[0039] The original fine-grained dataset typically contains image samples from multiple categories. Text prompts consist of a set of text prompts generated for each category in the original fine-grained dataset. These text prompts include descriptive information for each category in the fine-grained dataset, such as textual indications like category name, background, scene, and art style. Furthermore, text prompts may also include, for example, subject matter, media (e.g., illustration, oil painting, digital painting), art style, similar style artists, similar style websites, resolution, additional details, tone, lighting, etc. Alternatively, text prompts may include negative indications or keyword weights. These text prompts are provided to the diffusion model as embedding vectors. For example, a tokenizer can be used to process the text prompts, converting each word in the text prompt into a token represented as a number, and then each token is converted into a 768-dimensional (or 1024-dimensional) embedding vector for use by the denoising network in the subsequent diffusion model.

[0040] Contour maps are generated by extracting edge information from each image sample in the original fine-grained data. Image edges are fundamental features of an image and contain rich information. Edge detection operators are typically used to detect the edges of targets. These operators mainly include first-derivative edge detection operators (such as the Roberts, Sobel, and Prewitt operators), second-derivative edge detection operators (such as the Laplacian operator), and the Canny operator based on multi-stage optimization. In one embodiment of this application, the Canny operator is used to extract the edges of the image to form a contour map. In one embodiment, the contour map can be a binarized black-and-white image. In yet another embodiment, before providing the contour map to the diffusion model for guidance, various random transformations (such as rotation, flipping, and thin-plate spline interpolation) can be applied to the contour map to further increase the diversity of the pose and morphology of the generated samples.

[0041] In step S110, a fine-grained classifier is pre-trained based on the original fine-grained dataset, which can accurately identify the target category of the input image. Inputting the image into the fine-grained classifier yields the prediction confidence score for the target category. The prediction confidence score is typically a value between 0 and 1. This fine-grained classifier can provide gradients as guiding signals to adjust the direction of image generation by the diffusion model.

[0042] By employing three different guidance sources—coordinated text prompts, contour maps, and fine-grained classifiers—comprehensive control over the sample generation process can be achieved, from coarse to fine. Text and contour prompts are responsible for constructing the "skeleton" and "scene" of the image, ensuring its overall rationality and diversity; while fine-grained classifier prompts are responsible for refining the "flesh" and "details," ensuring the accuracy of its category attributes.

[0043] Continue to refer to Figure 1 In step S120, the guidance signals from the guidance source are input into the diffusion model to generate enhanced samples. Guidance signals from text prompts can be used to define the global scene, style, and coarse-grained object attributes, driving the diversity of generated samples. Guidance signals from contour maps can further enhance the diversity of poses and forms while ensuring the realism of object structure. Guidance signals from fine-grained classifiers can accurately correct subtle visual features related to the category.

[0044] The process of generating enhanced samples is essentially the process of using the denoising process of a trained diffusion model to progressively denoise the noisy image to obtain a clear image. This noisy image can be a pure Gaussian noise image, i.e., an image composed entirely of random pixel values ​​following a normal distribution. Alternatively, it can be another type of noise image, such as gamma-distributed noise or beta-distributed noise, but it should be the same as or correspond to the noise selected in the diffusion model. This noisy image can have the same size as the generated enhanced samples.

[0045] In some embodiments of this application, the denoising process can be divided into a first stage and a second stage. In the first stage, guidance is provided using guidance signals from both the text prompt and the contour map; in the second stage, guidance is provided using guidance signals from the text prompt, the contour map, and the fine-grained classifier, based on the results of the fine-grained classifier. In both stages, the guidance signals provided to the denoising network of the diffusion model are weighted combinations of the guidance signals from each guidance source and their corresponding weights (or guidance signal strengths). Here, the guidance signal strength includes the text guidance strength. Contour Guiding Intensity and classifier guidance strength These correspond to three types of guidance sources: text prompts, contour maps, and fine-grained classifiers, respectively.

[0046] In some embodiments, in the first stage, the text guidance strength and the contour guidance strength remain fixed, while in the second stage, the text guidance strength, contour guidance strength and classifier guidance strength are dynamically adjusted based on the results of the fine-grained classifier.

[0047] Specifically, assuming This is the total noise reduction time step. For the time step of the phase transition, and Therefore, based on the diffusion model's setting of time steps for the denoising process from large to small, the first stage is time step [number missing]. Time to step The second stage is the time step. Time step 0.

[0048] In the first stage, guidance is provided using guidance signals from both the text prompt and the contour map, with the text guidance strength and contour guidance strength remaining fixed. This can be achieved through classifier-independent guidance using the guidance signal from the text prompt, or through a ControlNet structure control model using the guidance signal from the contour map. The text prompt, after being converted into an embedding vector, serves as the cross-attention layer of the conditional input diffusion model. During generation, the model adjusts image features based on this conditional input to ensure the output is semantically aligned with the text prompt. ControlNet achieves precise control over the structure of the generated image by introducing external conditional signals (such as those from edge maps, depth maps, pose maps, etc.) as guidance signals. Its core principle is to copy the U-Net encoder parameters of the diffusion model into a parallel branch, connecting it to the backbone through zero-convolutional layers, allowing for the step-by-step injection of structure control signals while ensuring that the initial stage does not interfere with the feature distribution of the original model. In some embodiments, the text guidance strength and contour guidance strength in the first stage can be preset. In some embodiments, the text guidance strength can be 1-10, preferably 7-10, and the contour guidance strength can be 0.3-2.0, preferably 0.5-1.5.

[0049] In the second stage, based on the guidance from the first stage, guidance is provided using guidance signals from text prompts, contour maps, and fine-grained classifiers. The guidance strength of these three sources is dynamically adjusted based on the prediction confidence of the fine-grained classifier for the currently generated intermediate results.

[0050] For example, the current time step is At that time, acquire the current noisy image. Corresponding estimated clear image ,Will Input into a fine-grained classifier to obtain the target category. Prediction confidence And based on the prediction confidence level The intensities of the three guiding signals are dynamically adjusted, with the text guiding intensity and contour guiding intensity being correlated with the prediction confidence, respectively. Proportional, that is , And classifier guidance strength and prediction confidence A linear relationship, that is ,in, Indicates at time step The intensity of text guidance at that time Indicates at time step Contour guidance intensity at that time Indicates at time step Classifier guidance strength at time The weights of the pre-defined guiding signals from the text prompts, The weights are preset from the guiding signals derived from the contour map. The preset weights are the guiding signals from the fine-grained classifier. In some embodiments, the preset text guiding strength... The preset contour guide strength can be set to 7.5. It can be set to 1.0, the preset classifier guidance strength. It is 10.0. When the time step is... At this time (i.e., in the first step of the second stage), the text guidance intensity is Contour guidance intensity is The classifier guidance strength is This is equal to the preset value. Then, guidance is provided based on the three guidance strengths mentioned above, and the prediction confidence is calculated. When the time step is... At time (i.e., in the second step of the second stage and thereafter), according to the time step The prediction confidence level calculated at the previous time step (i.e., the previous time step) Calculate when the time step is The text guidance strength, contour guidance strength, and classifier guidance strength at time step 1 are used to calculate the text guidance strength, contour guidance strength, and classifier guidance strength when the time step is 1. The system first provides guidance at a specific time step, then applies guidance based on the three guidance strengths mentioned above, calculates the prediction confidence, and so on, until the time step is... At that time, the denoising process of the noisy image is completed to obtain the final enhanced sample.

[0051] In the above embodiments, dynamic guidance intensity based on different generation stages and confidence levels achieves comprehensive control over the production process from coarse to fine. In the early stages of generation, text prompts and contour map guidance with fixed guidance intensity are mainly relied upon to quickly establish the overall structure of the image. In the later stages of generation, fine-grained classifier guidance is activated, and the relative guidance intensity of all three guidance signals is dynamically adjusted according to the prediction confidence of the fine-grained classifier for the current intermediate generation result. When the prediction confidence is low, the weight of fine-grained classifier guidance is increased to correct the details of the generated image; when the prediction confidence is high, the weight of fine-grained classifier guidance is decreased, allowing text prompts and contour map guidance to play a greater role in enhancing diversity and avoiding overfitting. Therefore, this dynamic mechanism intelligently balances fidelity and diversity when generating fine-grained data samples. It avoids generation failure caused by prematurely introducing detail constraints before the image structure is formed, while providing a "tailor-made" guidance strategy for each sample through adaptive adjustment based on confidence level. This not only improves the overall quality and stability of the generated images, but also effectively prevents mode collapse or artifacts caused by over-guiding.

[0052] In some embodiments, the guiding signal from the fine-grained classifier is obtained by calculating the gradient of the fine-grained classifier on the currently estimated sharp image, denoted as . This is used to guide the generation process in a direction that better reflects the characteristics of the target category. That is, at the current time step of the second stage... The guiding signal calculated from the fine-grained classifier is based on the current noisy image. Corresponding estimated clear image This is obtained by calculating the gradient. For the target category, The prediction confidence score for the target class. Based on Bayes' theorem, the conditionally generated logarithmic gradient can be decomposed into the unconditional gradient. Classifier gradient With constant term The sum of Among them, the constant term This is usually negligible. Let the noise predicted by the diffusion model be denoted as... If only the guiding signal of the fine-grained classifier is introduced, its noise prediction is the same as the noise predicted by the diffusion model. Gradient guided by classifier The sum of Thus, the direction of sample generation in the diffusion model can be adjusted using such guiding signals.

[0053] Figure 2 The illustration shows a schematic flow of a data augmentation method based on a diffusion model, representing other embodiments of this application. For example... Figure 2 As shown, in some embodiments, in addition to steps S110 and S120 described above, the data augmentation method 100 based on the diffusion model may further include: S115, performing a contour transformation on the contour map. Before using the contour map for guidance, one or more contour transformations may be performed on part or all of the contour map. For example, when the original image sample corresponding to the contour map is a rigid object, an affine transformation is applied to the contour map; when the original image sample corresponding to the contour map is a non-rigid object, an affine transformation and / or thin-plate spline interpolation distortion is applied to the contour map. In some embodiments, the affine transformation may include flipping and rotation. The contour map may be horizontally flipped to swap the left and right sides. The contour map may be flipped at a small angle. The rotation angle may be random, less than ±30°, preferably less than ±15°. One or more transformations may be performed on the same contour map to generate one or more contour maps that have undergone contour transformation. Such contour transformations can increase the diversity of the pose and shape of the generated samples.

[0054] Figure 3 A schematic flowchart of a data augmentation system based on a diffusion model according to an embodiment of this application is shown. Figure 3 As shown, the data augmentation system 200 based on the diffusion model includes a guidance source module 210 and a sample generation module 220. The guidance source module provides three guidance sources: text prompts 211, contour maps 212, and fine-grained classifiers 213, as detailed above in conjunction with step S110. The sample generation module 220 includes a diffusion model 221, used to generate augmented samples using the guidance signals provided by the guidance source module through the diffusion model, as detailed above in conjunction with step S120.

[0055] Figure 4 A schematic flowchart of a data augmentation system based on a diffusion model, according to yet another embodiment of this application, is shown. Figure 4 As shown, in addition to the guide source module 210 and the sample generation module 220, the data augmentation system 200 based on the diffusion model may also include a contour transformation module 230, which is used to perform contour transformation on the contour map. For details, please refer to the content introduced above in conjunction with step S1115.

[0056] In another embodiment of this application, a training method for a fine-grained visual classification model is proposed. This method mainly includes: for each original image sample or a portion of the original image samples in the original fine-grained training set, generating one or more augmented samples according to the data augmentation method described in the above embodiment; mixing the generated augmented samples with the original image samples of the original fine-grained training set to form an augmented fine-grained training set; and using the augmented fine-grained training set to train a fine-grained visual classification model. In this embodiment, because the generated augmented samples balance diversity and high fidelity, the data samples in the formed augmented fine-grained training set possess both sufficient diversity and ensure the accuracy of category attributes. Based on such an augmented training set, the performance and generalization of the trained fine-grained visual classification model can be improved, making it applicable to fine-grained visual classification tasks.

[0057] In another embodiment, the enhanced fine-grained training set can be used to fine-tune the fine-grained classifier pre-trained on the original fine-grained training set in step S110, and the fine-tuned fine-grained classifier can be used as the final fine-grained visual classification model, thereby further improving training efficiency.

[0058] In yet another embodiment of this application, a computer-readable storage medium is provided, on which computer instructions are stored, which, when executed by a processor, implement the data augmentation method and training method as described in the embodiments of this application.

[0059] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium. As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media generally include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0060] It should be noted that although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowchart can be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0061] It should be understood that when the terms "first," "second," "third," and "fourth," etc., are used in the claims, specification, and drawings of this application, they are used only to distinguish different objects and not to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this application indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0062] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this specification and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this specification and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0063] Although the embodiments of this application are described above, the content is merely an example adopted for the purpose of facilitating understanding of this application and is not intended to limit the scope and application scenarios of this application. Any person skilled in the art described in this application may make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in this application, but the scope of patent protection of this application shall still be determined by the scope defined in the appended claims.

Claims

1. A data augmentation method based on a diffusion model, comprising: The guidance source is provided based on the original fine-grained dataset, which includes text prompts, contour maps, and fine-grained classifiers; as well as Enhanced samples are generated using a diffusion model with a guiding signal from the guiding source; The text prompts include descriptive information for each category in the fine-grained dataset. The fine-grained classifier, which is used to predict the confidence level of the image category, is trained on the fine-grained dataset. The contour map is extracted from the original image samples in the fine-grained dataset.

2. The method according to claim 1, wherein, The generation of enhanced samples includes denoising the noisy image using a diffusion model to generate enhanced samples, wherein the denoising process includes a first stage and a second stage: In the first stage, guidance is provided using guiding signals from the text prompts and the outline map; and In the second stage, guidance is provided using guidance signals from the text prompts, the contour map, and the fine-grained classifier.

3. The method according to claim 2, wherein, The guiding signal provided to the denoising network during the denoising process is a weighted combination of guiding signals from various guiding sources and their corresponding weights.

4. The method according to claim 3, wherein, In the first stage, the weights of the guiding signals from the text prompt and the contour map remain unchanged; and in the second stage, the weights of the guiding signals from the text prompt, the contour map, and the fine-grained classifier are dynamically adjusted based on the prediction confidence of the fine-grained classifier for the output of the denoising network.

5. The method of claim 4, wherein the dynamic adjustment in the second stage comprises: For the current time step of the second stage Get the current noisy image Corresponding estimated clear image ,Will Input the fine-grained classifier to obtain the target category. Prediction confidence ; Based on prediction confidence Calculate the time step for each time step. The weight of the guiding signal is denoted as , , , in, In time step The weight of the guiding signal from the text prompt, In time step The weights of the guiding signals from the contour map, In time step The weights of the guiding signal from the fine-grained classifier. The weights of the pre-defined guiding signals from the text prompts are: The weights of the guiding signals from the contour map are preset. The weights are preset from the guiding signals of the fine-grained classifier.

6. The method according to claim 5, wherein, It also includes the current time step in the second phase. The guiding signal calculated from the fine-grained classifier is based on the current noisy image. Corresponding estimated clear image This is obtained by calculating the gradient, denoted as . ,in, For the target category from the fine-grained classifier The prediction confidence level.

7. The method according to any one of claims 1-6, wherein, The method also includes performing a contour transformation on the contour map, the contour transformation including one or more of the following: When the original image sample corresponding to the contour map is a rigid object, an affine transformation is applied to the contour map; and When the original image sample corresponding to the contour map is a non-rigid object, affine transformation and / or thin plate spline interpolation distortion are applied to the contour map.

8. A training method for a fine-grained visual classification model, comprising: For each original image sample or a subset of original image samples in the original fine-grained training set The data augmentation method according to any one of claims 1-7 generates one or more augmented samples; The generated enhanced samples are mixed with the original image samples of the original fine-grained training set to form an enhanced fine-grained training set. The fine-grained visual classification model can be trained using the enhanced fine-grained training set, or a pre-trained fine-grained classifier can be fine-tuned and the fine-tuned fine-grained classifier can be used as the final fine-grained visual classification model, wherein the fine-grained classifier is obtained by pre-training on the original fine-grained training set.

9. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-8.

10. A data augmentation system based on a diffusion model, comprising a guiding source module and a sample generation module, wherein: The guidance source module is configured to provide guidance sources based on the original fine-grained dataset, the guidance sources including text prompts, contour maps, and fine-grained classifiers; as well as The sample generation module is configured to generate enhanced samples using a diffusion model via a guiding signal from the guiding source; The text prompts include descriptive information for each category in the fine-grained dataset. The fine-grained classifier is used to predict the confidence level of the image category and is trained based on the fine-grained dataset. The contour map is extracted from the original image samples in the fine-grained dataset.

Images