Corn leaf disease image generation method and device based on ControlNet and LoRA enhanced Stable Diffusion model

Abstract

The invention discloses a corn leaf disease image generation method and device based on a ControlNet and LoRA enhanced Stable Diffusion model, and relates to a corn leaf disease image generation method and a corn leaf disease image generation device based on a ControlNet and LoRA enhanced Stable Diffusion model. Belongs to the technical field of image processing. According to the method, a data set covering seven types of corn leaf diseases is constructed, and a Stable Diffusion model integrating ControlNet and LoRA is trained on the basis of the data set so as to generate an image of a corresponding disease type. Afterwards, the generated image is combined with a real image to be used for training a Swin Transform model and generating a vivid corn leaf disease image, the problems of scarcity of field data and insufficient diversity are solved, and the effectiveness of the image in a disease recognition task is verified through quantitative evaluation.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a method and device for generating maize leaf disease images based on a ControlNet and LoRA-enhanced Stable Diffusion model. Background Technology

[0002] Leaf diseases in maize severely threaten the healthy growth of the crop, thus affecting grain yield. Computer vision-based disease identification methods largely rely on diverse field image datasets. Existing publicly available maize leaf disease image datasets provide valuable resources for disease identification research; however, most images were collected in laboratory environments with monotonous backgrounds and a lack of variation in natural field conditions, affecting the accuracy and robustness of models in real-world environments. Furthermore, some field images were collected relatively early (mostly before 2022), resulting in significant deficiencies in data diversity. The acquisition of high-quality disease images is often limited by actual collection conditions. Image generation methods can, to some extent, compensate for the shortcomings in actual collection and management, especially in enhancing disease identification capabilities, and have shown great promise.

[0003] However, existing methods using GAN models still suffer from problems such as pattern collapse and training difficulties, limiting their application in high-quality image generation tasks. Meanwhile, although current research shows that DDPM performs well in plant disease image synthesis, learning the appearance of specific diseases in different contexts remains a challenge. Furthermore, DDPM operates in pixel space, resulting in high computational resource consumption and time-consuming generation.

[0004] Therefore, how to provide a method and device for generating maize leaf disease images based on the ControlNet and LoRA-enhanced Stable Diffusion model is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a method and device for generating maize leaf disease images based on a Stable Diffusion model enhanced by ControlNet and LoRA, which is used to generate realistic maize leaf disease images, alleviate the problem of scarce and insufficient field data, and verify its effectiveness in disease identification tasks through quantitative evaluation.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A method for generating maize leaf disease images based on a ControlNet and LoRA-enhanced Stable Diffusion model includes the following steps:

[0008] S100: Construct the original dataset containing images of various types of maize leaf diseases;

[0009] S200: Divide the original dataset into a ControlTrain subset and a LoRATrain subset. The ControlTrain subset contains disease types with a sufficient number of images and corresponding leaf structure masks, while the LoRATrain subset contains disease types with a scarce number of images.

[0010] S300: The ControlNet module is trained using a subset of ControlTrain to learn the mapping relationship between the leaf structure mask and the image structure;

[0011] S400: Multiple LoRA modules are trained using a subset of LoRAATrain, each LoRA module focusing on learning lesion features for one disease type;

[0012] S500: Integrates the trained ControlNet and LoRA modules into the pre-trained Stable Diffusion model;

[0013] S600: Generates images of maize leaf diseases using an integrated Stable Diffusion model.

[0014] Furthermore, it also includes:

[0015] S700: Train the SwinTransformer classification model using generated maize leaf disease images;

[0016] S800: Uses a trained SwinTransformer classification model to identify corn leaf disease images.

[0017] Furthermore, the S600 includes:

[0018] The ControlNet module inputs a leaf structure mask to control the overall structure and layout of the generated image; the LoRA module injects the style features of the target disease and combines them with text prompts as semantic conditions to guide the Stable Diffusion model in generating maize leaf disease images.

[0019] Furthermore, during the training process of the ControlNet module, manually labeled leaf structure masks are used as conditional inputs and injected into the Stable Diffusion backbone network via residuals.

[0020] Furthermore, the LoRA module achieves fine-tuning of lesion style features by inserting a low-rank weight matrix into the attention layer of the Stable Diffusion model.

[0021] Furthermore, the S600 also includes image post-processing and filtering steps:

[0022] The CLIP model was used to calculate the semantic similarity between the generated maize leaf disease images and the text prompts, and images with similarity below a preset threshold were removed.

[0023] Calculate the structural similarity index between the generated maize leaf disease image and the input structural mask, and remove images with similarity below a preset threshold;

[0024] Images are filtered out to remove those whose lesion shape, color, or location does not conform to prior knowledge.

[0025] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a method for generating maize leaf disease images based on a ControlNet and LoRA-enhanced StableDiffusion model.

[0026] As can be seen from the above technical solutions, compared with the prior art, this invention discloses a method and device for generating maize leaf disease images based on a Stable Diffusion model enhanced by ControlNet and LoRA. It integrates field images and experimental environment images from multiple public datasets to construct a synthetic dataset SLB containing seven types of maize leaf diseases to support high-quality image generation. It proposes for the first time a SD maize leaf disease image generation framework that integrates ControlNet and LoRA, and uses precise leaf structure masking and lesion feature learning to achieve controllable synthesis of leaf morphology, lesion distribution location, and natural field background. The perceptual quality of the generated images is evaluated, and the SwinTransformer model is trained by combining real and generated images to verify the role of generated image data in improving model performance in disease identification tasks. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of the method flow provided by the present invention;

[0029] Figure 2 A flowchart illustrating the overall modeling process for constructing the image generation framework provided by this invention;

[0030] Figure 3 This is a map showing the distribution of maize diseases generated based on Fr'echet's initial distance (FID) and initial score (IS).

[0031] Figure 4 Comparison of real images and generated examples of corn leaves for seven types of diseases;

[0032] Figure 5 This is a schematic diagram of the Grad-CAM visualization results for identifying corn diseases in test images based on the Swin Transformer model. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Example 1:

[0035] See Figure 1 This invention discloses a method for generating maize leaf disease images based on the ControlNet and LoRA-enhanced StableDiffusion model, comprising the following steps:

[0036] S100: Construct the original dataset containing images of various types of maize leaf diseases;

[0037] S200: Divide the dataset into a ControlTrain subset and a LoRATrain subset. The ControlTrain subset contains disease types with a sufficient number of images and corresponding leaf structure masks, while the LoRATrain subset contains disease types with a scarce number of images.

[0038] S300: The ControlNet module is trained using a subset of ControlTrain to learn the mapping relationship between the leaf structure mask and the image structure;

[0039] S400: Multiple LoRA modules are trained using a subset of LoRAATrain, each LoRA module focusing on learning lesion features for one disease type;

[0040] S500: Integrates the trained ControlNet and LoRA modules into the pre-trained Stable Diffusion model;

[0041] S600: Generates images of maize leaf diseases using an integrated Stable Diffusion model.

[0042] Furthermore, it also includes:

[0043] The SwinTransformer classification model was trained using generated images of corn leaf diseases.

[0044] The trained Swing Transformer classification model was used to identify maize leaf disease images.

[0045] In one specific embodiment, the constructed original dataset SLB contains images of seven typical maize leaf diseases: GLS (Gray LeafSpot), NLS (Northern Leaf Spot), NLB (Northern LeafBlight), Rust, ZD (Zinc Deficiency), FW (Fallarmy Worm), and BS (Brown Spot), providing good coverage of disease types. The images come from diverse sources, covering multiple authoritative public datasets and related research findings. Specifically, GLS (518 images) and NLS (467 images) are from the CD&S dataset; NLB (1500 images) are from the CD&S and OSF datasets; BS (107 images) were obtained through online collection and field sampling; Rust (1381 images) are from the PlantVillage dataset; and ZD (179 images) and FW (145 images) are from research data from the WG-MARNet project.

[0046] Overall, the SLB dataset exhibits significant differences in imaging backgrounds (such as field soil and leaf occlusion), lighting conditions (natural and artificial light), plant growth stages, and data acquisition equipment configurations, demonstrating good diversity and real-world adaptability. This provides a solid data foundation for building disease identification and image generation models with strong generalization capabilities. Specifically, SLB includes images with complex field backgrounds (such as GLS, NLS, and NLB), standardized images from experimental environments (Rust), and images with simple backgrounds and prominent features (ZD and FW). Rust, ZD, and FW images have fewer interfering factors, facilitating the extraction of clear lesion features. More importantly, the introduction of these images lays the data foundation for the subsequent innovative generation of Rust, ZD, and FW disease images with realistic field backgrounds, aiming to improve the model's adaptability in real-world application scenarios and the visual realism of the generated images.

[0047] Specifically, the overall scale of currently available maize disease images remains relatively limited, especially for the ZD, FW, and BS categories, which presents challenges for training diffusion-based models. Furthermore, the SLB dataset exhibits an uneven distribution of images across categories, with NLB images being the most numerous. This imbalance may lead to bias in disease identification models towards high-frequency categories (such as NLB, GLS, and NLS). It is worth noting, however, that NLB, GLS, and NLS samples are not only abundant but also of high quality with rich background scenes, providing strong support for the synthesis and enhancement of low-frequency category (Rust, ZD, FW, BS) images. This helps alleviate the data imbalance problem and improve model generalization performance.

[0048] In one specific embodiment, considering the differences in original resolution and size caused by different image acquisition devices, all images were uniformly adjusted to 512×512 pixels during preprocessing to meet the model training requirements. Furthermore, to adapt to different generation tasks, the SLB dataset was divided into two training subsets: ControlTrain and LoRATrain. ControlTrain contains GLS, NLS, and NLB images with sufficient samples. Leaf regions are labeled using the Labelme tool, and corresponding structural masks are generated. It is mainly used to train the ControlNet and conditional diffusion models to achieve fine-grained control over structure and background. LoRATrain contains images of Rust, ZD, FW, and BS with fewer samples. It is used to train the LoRA model, focusing on learning the potential feature distribution of different types of lesions, providing prior guidance for the high-quality generation of disease images with fewer samples.

[0049] In one specific embodiment, the conditional diffusion image generation method based on ControlNet and LoRA includes:

[0050] The proposed image generation framework is built on the Stable Diffusion (SD) model and further introduces structure control and style regulation mechanisms to achieve dual fine guidance on the structure and lesion patterns in maize leaf images. The structure control part is implemented using the ControlNet module.

[0051] ControlNet, as a structure-aware conditional diffusion control module, enhances the ability to model image structures on the basis of the original SD architecture.

[0052] Specifically, the present invention through The system receives structural maps (such as edge maps, pose maps, and mask maps) as conditional inputs. During training, this structural branch is used to perceive and extract structure-guided features; during the inference phase, its output is injected into the backbone network as residuals, thereby achieving precise control over the structural space while ensuring semantic consistency of the image.

[0053] Specifically, to adapt to the task of synthesizing maize disease images, this invention uses the ControlTrain dataset to train ControlNet. The mask images in this dataset clearly separate leaf regions from lesion regions, significantly enhancing the model's ability to perceive leaf margin structures and lesion distribution, thereby promoting the synthesis of images with reasonable structure and rich detail. The structure-guided diffusion prediction form of this process is as follows:

[0054]

[0055] Where, x t Let c represent the noisy image at step t, where t represents the time step and c represents the conditional information (such as the text prompt). θ This represents a conditional denoising network. It is the residual output of the structure-aware branch. This represents the final predicted clean image. The control information of the final structure diagram is injected into the backbone network through residuals, and can be expressed by formula (2):

[0056]

[0057] Where λ is the control weight, which determines the structural guidance strength.

[0058] Building upon the structural control modeling, the style adjustment mechanism is implemented using LoRA (Low-Rank Adaptation). LoRA is an efficient parameter fine-tuning method. Its core idea is to insert a small number of trainable low-rank weight matrices into the key attention layers while freezing the pre-trained model parameters, thereby enabling rapid model adaptation to downstream tasks. In this invention, LoRA is primarily used to learn the style features of different lesion types (Rust, ZD, FW, BS) in terms of shape, color, texture, and spatial distribution.

[0059] Specifically, to achieve the aforementioned style modeling, LoRA inserts a weight matrix in low-rank form into the attention layer. Let the weights of a certain linear transformation in the pre-trained attention layer be... The interpolation formula for LoRA is:

[0060] W = W0 + ΔW ΔW = A cls B cls cls∈{Rust,ZD,FW,BS} (3)

[0061] in, Let r represent the LoRA parameters for the i-th type of lesion, and d represent the rank. During training, only the weights of groups A and B are updated, while W0 remains frozen. After the model is trained, various LoRA modules can be loaded into the SD main model as needed to achieve controllable synthesis of lesion styles.

[0062] Specifically, with the introduction of the structure and style control modules, the overall image generation objective is also uniformly expanded within the original latent space framework of SD. After incorporating the structural information introduced by ControlNet and the style features modeled by LoRA, the latent diffusion loss function under joint modeling can be expressed as:

[0063]

[0064] in, This indicates that the LoRA backbone network has been loaded, and ∈ represents the actual noise added. To further improve the semantic expressiveness and visual style consistency of the generated images, perceptual loss and style loss are introduced as auxiliary terms, and the final objective function is extended to:

[0065]

[0066] in Indicates perceived loss. This represents the style loss, where α and β are the corresponding weighting coefficients.

[0067] Specifically, to more clearly present the structural composition and operational logic of the method proposed in this embodiment, Figure 2This paper demonstrates the overall modeling process of the constructed image generation framework. Based on a pre-trained Stable Diffusion 2.1 (SD 2.1) model, the framework jointly introduces the ControlNet structure control module and the LoRA style modulation mechanism to achieve hierarchical modeling and joint control of background generation, structure restoration, and lesion style expression in maize leaf images. In the modeling process, disease images with real field backgrounds from the ControlTrain dataset are first used, along with corresponding structure masks, to train the ControlNet module. This enhances the model's ability to perceive the structure of leaf edge contours and lesion distribution patterns. The mask image, obtained through manual annotation, clearly separates leaf and lesion regions, allowing structural information to act as a guiding signal injected into the diffusion process, achieving precise control over the spatial layout of the image. Simultaneously, leveraging the strong modeling capabilities of the SD model for natural scenes, background features and overall visual style under field conditions are effectively preserved. Building upon this, the style modeling path further incorporates the LoRATrain dataset. For four typical disease types—Rust, ZD, FW, and BS—corresponding LoRA fine-tuning modules are trained and embedded into the key attention layer of the SD model. While keeping the core parameters frozen, LoRA achieves controllable modeling capabilities for lesions in style dimensions such as color, texture, and shape by inserting low-rank trainable weights, significantly enhancing the diversity of image generation results and the realism of lesions.

[0068] During the image inference stage, ControlNet and LoRA work synergistically with the denoising UNet in the diffusion process from both structural and stylistic dimensions, ensuring that the synthesized image retains a natural background while possessing clear structural boundaries and realistically diverse lesion features. Finally, a new training set is constructed by mixing generated images with real images, which is then used to train the downstream lesion recognition model, empirically verifying the effectiveness of this framework in enhancing data diversity and improving model recognition performance.

[0069] In terms of data partitioning, the ControlTrain dataset is divided into training set, validation set and test set according to the ratio of 75%:20%:5%. The training set contains a total of 1864 images, which are used for training the ControlNet module and also serve as the image source for the SD model.

[0070] In the inference settings, the text prompt "Field background, high detail of natural diseased leaves" was used consistently to ensure semantic consistency. The classifier-free guidance scale was set to 9.0, which is in the middle of the commonly used range [0,20], to balance the generation weights between textual conditions and structural controls. The strength parameter was set to 1.0 to encourage diversity in structure and detail in the image; the inference steps were set to 50 to ensure generation quality, but in the actual image generation process, the step size was reduced to 20 to improve synthesis efficiency.

[0071] In terms of training configuration, the SD model fused with ControlNet was trained for 75 epochs, with a batch size of 2 per epoch. Experiments have verified that this setting can achieve effective convergence. The LoRA fine-tuning module was trained for 20 epochs for each of the four lesion types, with a learning rate of 1e-5. The remaining hyperparameters were implemented according to publicly available technologies.

[0072] Specifically, all model training and image generation were performed using the PyTorch v2.4.1 framework on a high-performance computing platform running Linux. The experimental platform was equipped with an Intel i9-10900X (10-core) processor, 256GB of RAM, and two NVIDIA RTX 4090 graphics cards (24GB of VRAM per card), enabling efficient operation of large-scale image synthesis and multi-model training tasks.

[0073] In image generation processing, due to inherent imperfections in the generation modeling process, using unrealistic or low-quality generated images may negatively impact the performance of downstream recognition models. To ensure that the generated images have a positive effect on the disease recognition task, this embodiment introduces a screening process after image generation. First, the CLIP model is used to map the image and the text prompt to the same semantic space, and cosine similarity is calculated to assess the semantic consistency between the image and the text description, thereby eliminating abnormal images with significant semantic deviations. Second, combined with the input conditions of ControlNet, the generated image and the conditional image are structurally compared, for example, by calculating the SSIM (structural similarity) of the edge image. Finally, image samples that do not visually match the characteristics of real lesions are further manually screened out, such as whether the lesion morphology, color features, and location conform to prior disease knowledge. High-quality generated images selected through the above process are added to the real image dataset.

[0074] like Figure 2 As shown, the new maize disease image dataset is divided into training, validation, and test sets in a 75%:5%:20% ratio. Although the newly generated images change the sample size ratio, the validation and test sets are always composed solely of real images, thus ensuring the objectivity and consistency of the evaluation process. Meanwhile, by introducing training data with diversity and field context, the model's generalization ability is effectively enhanced, laying the foundation for subsequent improvements in recognition performance.

[0075] In one specific embodiment, this invention proposes using the state-of-the-art and lightweight Swin Transformer model for maize leaf disease identification. This model, through a hierarchical window self-attention mechanism, effectively controls computational overhead while ensuring high identification accuracy, achieving a good balance between detection efficiency and accuracy, and possessing strong adaptability for engineering deployment.

[0076] Specifically, the SwinTransformer model was trained to evaluate the effectiveness of generated images at different scales in the corn disease identification task. The generated images used were added to the original real image training set at five scales, from 20% to 100%, with 20% intervals, representing 100%, 200%, 300%, 400%, and 500% of the number of images for each disease category in the original real image training set (e.g., ...). Figure 2 (As shown). All input images (including real and generated images) were uniformly adjusted to a resolution of 224×224 to fit the input size requirements of the Swin Transformer, while providing suitable spatial information representation while preserving key lesion features. Compared to the original high-resolution images, this downsampling process has a smaller impact on lesion detection accuracy. The model was trained for 100 epochs with a batch size of 8, and other hyperparameter configurations adopted the default settings from the official implementations published in existing technologies.

[0077] Example 2:

[0078] I. Performance Indicators:

[0079] The quality of generated images is primarily evaluated based on two metrics: Fréchet Inception Distance (FID) and Inception Score (IS). FID measures the difference between the generated and real image distributions in the feature space to assess image fidelity; while IS assesses the diversity of generated images without relying on the involvement of real images. Both metrics rely on the Inception-v3 model pre-trained on the ImageNet dataset.

[0080] In the calculation of FID, feature vectors of the real image and the generated image are first extracted from the last pooling layer of the Inception-v3 model, and these vectors are treated as two multivariate Gaussian distributions with estimated mean and covariance, respectively. Then, the difference between the two Gaussian distributions is measured by the Fréchet distance (also known as the Wasserstein-2 distance), and its calculation formula is as follows (6):

[0081]

[0082] Where X is the set of real images, and g is the set of generated images; μ r and μ g Σ represents the mean of the feature representations of the real and generated images, respectively. r and Σ g These are the corresponding covariance matrices of the real and generated images, respectively. Tr represents the trace of the matrix. ||·|| 2 This represents the squared Euclidean distance.

[0083] The IS index measures the quality and diversity of an image by calculating the Kullback-Leibler divergence (KL divergence) between the predicted class distribution p(y|x) and the marginal class distribution p(y) of the generated image. Its formula is (7):

[0084]

[0085] Among them, D KL Denotes KL divergence, This represents the expectation for the generated image.

[0086] Furthermore, to evaluate the model's classification performance in the maize leaf disease identification task, this embodiment uses four commonly used classification metrics: accuracy (Accuracy, Acc), precision (Precision, P), recall (Recall, R), and F1 score (F1), the calculation formulas of which are as follows:

[0087]

[0088] Wherein, TP (True Positive) represents the number of samples correctly identified as positive, FP (False Positive) represents the number of negative samples incorrectly identified as positive, TN (True Negative) represents the number of samples correctly identified as negative, and FN (False Negative) represents the number of positive samples that were not identified.

[0089] II. Quality assessment of generated images:

[0090] Figure 3The distribution of generated disease images is shown based on FID and IS metrics (average for each disease category). The average FID and IS scores for all disease categories are 49.19 and 7.72, respectively. Generally, a lower FID value indicates that the generated image is closer to the real image in the feature space, reflecting higher image fidelity; while a higher IS value indicates better diversity in the generated image. In the FID–IS scatter plot, disease categories located in the lower right region (e.g., GLS and NLS) perform better on both metrics, indicating that their generated images have better overall visual perception quality. Of the seven disease categories evaluated, four categories have an average FID score below 50, showing high generation quality. ZD has a slightly higher FID value of 62.75, although this category used the most iterations during training, there is still a certain gap between the generated image and the real image. It is worth noting that NLS achieved the lowest FID score (36.46) and a high IS score, indicating that this category produces images with the best fidelity and diversity, making it the disease category with the highest quality output.

[0091] In the process of generating disease images, this invention uses ControlNet and LoRA to guide and control the image structure and lesion features, respectively. Figure 4This section presents a comparison of disease images generated by different models under the same text prompt (each model's generated image is not based on a modified original image). The original reference images for each disease type are displayed at the top. Test1 represents the image generated using the original SD model, Test2 represents the result after adding LoRA, and Test3 represents the result after combining ControlNet and LoRA. In Test1, the SD model can generate images with certain disease characteristics based on the text prompt, but its control over leaf structure and background is weak, often resulting in false backgrounds, leaf deformation, and color distortion. Although it exhibits moderate feature similarity in most cases, these images deviate significantly from real images in structure and texture. Test2 incorporates LoRA for learning disease features, and compared to Test1, the generated image is closer to the real situation in terms of lesion distribution. However, problems such as distorted leaf structure, blurred edges, and monotonous backgrounds still exist, and the overall realism and diversity of the image remain insufficient. Test3 further introduces ControlNet to constrain the leaf structure, resulting in a significant improvement in both structural accuracy and background naturalness of the generated image. The model can generate disease images with realistic morphology, clear texture, and backgrounds closely resembling the field environment. Overall, Test3, combined with a suitable image selection mechanism, can generate disease images with significant semantic features and visual consistency under different field conditions, providing potential support for enhancing the performance of maize leaf disease identification models and alleviating the problem of scarcity of real data.

[0092] III. Performance Evaluation of Disease Identification:

[0093] The first experiment aimed to establish baseline performance by using only the SLB dataset for disease identification. Subsequently, to evaluate the role of generated images in disease identification, a series of independent experiments were designed. These experiments trained the model using individual generated images, real images, and combinations of generated images at different proportions, systematically evaluating the impact of generated images on model performance. The second experiment focused on comparing the impact of real images combined with combinations of generated images at different proportions and the use of traditional image augmentation techniques to augment real images on model accuracy. This embodiment uses traditional data augmentation (DA) methods implemented with the Augmentor module, including random horizontal flipping, random rotation, brightness adjustment, contrast changes, and color perturbation. While preserving the real images, this approach expands the number of disease type images to further evaluate the supplementary value of generated images.

[0094] Table 1 shows the results of maize leaf disease identification based on real and generated images.

[0095]

[0096] Table 1 summarizes the results of the Swin Transformer model in identifying maize leaf diseases using the same set of real images across all experimental scenarios. Here, R represents real SLB dataset images, G represents manually filtered generated images, DA represents data augmentation using traditional image enhancement techniques, and 1G, 2G, 3G, 4G, and 5G are the mixture ratios of disease categories between generated and real images in the training set. The model trained using only R achieved mean accuracy (ACC), precision (Precision), recall (Recall), and F1 score (F1) of 0.8158, 0.8114, 0.8158, and 0.8065, respectively. In contrast, the model trained using only generated images showed a significant performance decline, with a mean accuracy approximately 10% lower than the model trained using real images. This indicates a significant inter-domain distribution difference between the generated image training set and the real image test set. Nevertheless, compared to the baseline model, combining real and generated images for training showed an overall improvement in the accuracy of maize leaf disease identification, with various evaluation metrics (ACC, Precision, Recall, F1) fluctuating slightly with the increase in the proportion of generated images. In most disease categories, the R+4S experiment (i.e., the number of generated images is approximately 400% of the number of real images) achieved the best performance with an average accuracy improvement of about 11 percentage points, with the BS category showing the largest improvement in accuracy, reaching 32%. This further verifies that introducing high-quality generated images in data-scarce categories helps improve the model's recognition ability. However, in this embodiment, further increasing the number of generated images has limited marginal improvement on model performance. Although the number of generated images varied in the mixed-proportion experiments, the obtained model accuracy was similar. This phenomenon may be attributed to information saturation (i.e., the model failed to learn more useful details from newly added images) and distribution bias (i.e., the model weights are more biased towards synthetic features).

[0097] Furthermore, the last column of Table 1 presents performance metrics data for models trained using real images and traditional image augmentation techniques. Comparison with the R+4S and R+5S experiments reveals that using DA techniques to augment the dataset has limited effectiveness, and the augmented data relies solely on real images, resulting in minimal changes in diversity. Consequently, the model's generalization performance is inferior to that of the hybrid generated image dataset.

[0098] Figure 5This paper presents a comparison of Grad-CAM heatmaps generated by the best-performing models trained on the SLB dataset (Ronly) and the mixed real and generated image dataset (R+4S) in a maize leaf disease identification task. While both models showed some focus on lesion areas in most categories, the introduction of generated images significantly enhanced the model's focus on key disease areas, especially in samples with complex leaf structures or strong background interference, such as the gls, nlb, and zd categories. The R+4S model more accurately located lesion areas, demonstrating stronger discriminative ability and a more stable spatial focus pattern, indicating that high-quality generated images play a positive role in enhancing the model's feature perception capabilities.

[0099] Specifically, this invention introduces advanced ControlNet and LoRA technologies into the Stable Diffusion model for the first time to generate images of various maize leaf diseases and assist in disease identification tasks. This invention constructs a dataset covering seven types of maize leaf diseases and trains a Stable Diffusion model combining ControlNet and LoRA on this dataset to generate images of the corresponding disease types. Subsequently, the generated images are combined with real images to train the SwinTransformer model, and its performance in disease identification tasks is evaluated. Experimental results show that this image generation model, customized for maize leaf diseases, can generate high-fidelity and highly diverse images, with average Fréchet Inception Distance (FID) and Inception Score (IS) of 50.98 and 7.63, respectively. Compared with the baseline model trained only using real images, the mixed training scheme using manually selected generated images and real images improves the average recognition accuracy by up to approximately 11%, significantly enhancing the model's recognition ability.

[0100] The above results demonstrate that the Stable Diffusion model, which integrates ControlNet and LoRA, has broad application prospects in agricultural image generation and data augmentation. It is expected to effectively expand and enrich real-world disease image datasets and improve the performance of agricultural visual recognition systems.

[0101] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0102] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating maize leaf disease images based on a ControlNet and LoRA-enhanced Stable Diffusion model, characterized in that, Includes the following steps: S100: Construct the original dataset containing images of various types of maize leaf diseases; S200: Divide the original dataset into a ControlTrain subset and a LoRATrain subset. The ControlTrain subset contains disease types with a sufficient number of images and corresponding leaf structure masks, while the LoRATrain subset contains disease types with a scarce number of images. S300: The ControlNet module is trained using a subset of ControlTrain to learn the mapping relationship between the leaf structure mask and the image structure; S400: Multiple LoRA modules are trained using a subset of LoRAATrain, each LoRA module focusing on learning lesion features for one disease type; S500: Integrates the trained ControlNet and LoRA modules into the pre-trained Stable Diffusion model; S600: Generates images of maize leaf diseases using an integrated Stable Diffusion model.

2. The method for generating maize leaf disease images based on a ControlNet and LoRA-enhanced Stable Diffusion model according to claim 1, characterized in that, Also includes: S700: Train the Swin Transformer classification model using generated maize leaf disease images; S800: Uses a trained Swin Transformer classification model to identify corn leaf disease images.

3. The method for generating maize leaf disease images based on a ControlNet and LoRA-enhanced Stable Diffusion model according to claim 1, characterized in that, The S600 includes: The ControlNet module inputs a leaf structure mask to control the overall structure and layout of the generated image; the LoRA module injects the style features of the target disease and combines them with text prompts as semantic conditions to guide the Stable Diffusion model in generating maize leaf disease images.

4. The method for generating maize leaf disease images based on a ControlNet and LoRA-enhanced Stable Diffusion model according to claim 1, characterized in that, During the training of the ControlNet module, manually labeled leaf structure masks are used as conditional inputs and injected into the Stable Diffusion backbone network via residuals.

5. The method for generating maize leaf disease images based on a ControlNet and LoRA-enhanced Stable Diffusion model according to claim 1, characterized in that, The LoRA module achieves fine-tuning of lesion style features by inserting a low-rank weight matrix into the attention layer of the StableDiffusion model.

6. The method for generating maize leaf disease images based on a ControlNet and LoRA-enhanced Stable Diffusion model according to claim 1, characterized in that, The S600 also includes image post-processing and filtering steps: The CLIP model was used to calculate the semantic similarity between the generated maize leaf disease images and the text prompts, and images with similarity below a preset threshold were removed. Calculate the structural similarity index between the generated maize leaf disease image and the input structural mask, and remove images with similarity below a preset threshold; Images are filtered out to remove those whose lesion shape, color, or location does not conform to prior knowledge.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for generating maize leaf disease images based on the ControlNet and LoRA-enhanced Stable Diffusion model as described in any one of claims 1 to 6.

Images