Litchi disease and pest detection method based on large-scale visual model and diffusion model

Abstract

The invention discloses a litchi disease and insect pest detection method based on a large-scale visual model and a diffusion model. According to the method, the InterImage model utilizes a 3 * 3DCNv3 operator to realize adaptive space aggregation and a dynamic receptive field, global image information is effectively captured, and feature extraction is enhanced, especially in a complex orchard environment. Compared with other backbone networks (such as ResNet50, ConvNeXt and SwinTransform), the detection precision is improved by the aid of the Internet. In addition, the DyHead module integrates attention mechanisms of scale, space and task awareness, and the ability of the model to detect objects of various sizes and distinguish targets close to or overlapped in position is enhanced. In addition, compared with the NMS, the CP-Cluster is a novel clustering framework which can be completely parallel. In the post-processing stage, CP-Cluster is adopted, a real square frame is enhanced by generating positive information, and redundant square frames are punished by synthesizing negative information, so that the detection result is enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of pest detection, and in particular relates to a litchi pest and disease detection method based on a large-scale visual model and a diffusion model. Background Art

[0002] Litchi is a tropical fruit that is loved by consumers all over the world for its delicious taste and rich nutritional value, and has considerable economic value. However, litchi trees are susceptible to a series of diseases and pests all year round, posing a huge threat to litchi production. Timely and accurate detection of these threats is essential to maintaining yield and fruit quality. Traditional detection methods mainly rely on manual field reconnaissance and visual inspection, which are inefficient, costly and highly subjective. These limitations make them unable to meet the needs of large-scale modern agriculture. Therefore, the development of intelligent and accurate litchi disease and pest detection methods is crucial to the sustainable development of the litchi industry.

[0003] In recent years, driven by deep learning technology, the field of plant disease and pest detection has developed rapidly. Researchers have extensively explored the application of convolutional neural networks (CNNs) and Transformers in disease and pest detection, and have achieved significant breakthroughs in accuracy and efficiency. Among various object detection frameworks, the YOLO (You Only Look Once) series has gained popularity due to its impressive speed and accuracy. The YOLO model surpasses traditional two-stage detectors in real-time detection while maintaining extremely high accuracy. This advantage has prompted the development of several improved YOLO variants specifically for accurate and efficient disease and pest detection. For example, Li et al. (2022) proposed an improved vegetable disease detection method based on YOLOv5, which combines a cross-stage partial transformer, an initial module, and a confluence module to improve the accuracy and speed of vegetable disease detection. Similarly, Tian et al. (2023) proposed the MD-YOLO model, which combines DenseNet and an adaptive attention module to improve the utilization of feature maps and improve the detection efficiency of small pests.

[0004] However, the small size of pest and disease targets poses a persistent challenge to effective feature extraction. The complexity of the background in the orchard environment, including the visual similarity of pests and diseases to plants and soil, and the variability of lighting conditions, can cause serious interference, resulting in reduced detection accuracy. Summary of the invention

[0005] The purpose of the present invention is to provide a litchi pest and disease detection method based on a large-scale visual model and a diffusion model in order to solve the above-mentioned problems.

[0006] The technical solution adopted by the present invention is as follows: a litchi pest and disease detection method based on a large-scale visual model and a diffusion model, the method comprising the following steps:

[0007] S1: First, collect images. In an orchard environment, use a smartphone or a high-pixel camera to collect images of litchi pests and diseases at different angles, distances, and lighting conditions;

[0008] Use the labeling tool LabelImg to mark the pests and disease spots in the above 4312 images with rectangular boxes

[0009] S2: Select the Shifted Diffusion model as the image generation tool and perform fine-tuning. Use the pre-processed litchi pest and disease images to train the Shifted Diffusion model so that it can generate high-quality litchi pest and disease images;

[0010] A large number of litchi pest and disease images were generated using the trained Shifted Diffusion model and combined with the original dataset to form a hybrid dataset.

[0011] S3: Select Intern Image as the backbone network of the detection model to take advantage of its powerful feature extraction capabilities;

[0012] Combine PAFPN and Channel Mapper for feature fusion and alignment so that feature maps of different scales can effectively transmit information;

[0013] The DyHead module is introduced to further optimize feature extraction and object detection by using its scale-aware, space-aware, and task-aware attention mechanisms.

[0014] Use CP-Cluster as a post-processing algorithm to replace the traditional NMS to more accurately distinguish true positives and redundant boxes;

[0015] S4: Train the Diff-InternImage-YOLO model using the mixed dataset and optimize it using classification loss, bounding box loss, and confidence loss;

[0016] Update model parameters through the back-propagation algorithm to continuously improve the detection accuracy and robustness of the model;

[0017] S5: Use the test set to evaluate the performance of the model, including mAP, accuracy, recall, F1 score and other indicators. Compare with other existing litchi pest and disease detection models to verify the superiority of Diff-InternImage-YOLO;

[0018] S6: Deploy the trained Diff-InternImage-YOLO model to a mobile device or server for real-time litchi pest and disease detection. Develop a user interface to facilitate users to upload litchi images and perform pest and disease detection;

[0019] S7: Regularly collect new litchi pest and disease images and update the training data set to maintain the timeliness and accuracy of the model. Monitor model performance and adjust and optimize according to actual conditions;

[0020] S8: Apply the Diff-InternImage-YOLO model to litchi pest and disease monitoring and management, helping fruit farmers to find problems in time and take corresponding prevention and control measures. Promote the Diff-InternImage-YOLO model so that it can be more widely used in the agricultural field.

[0021] In a preferred embodiment, in step S1, the acquisition equipment is a smart phone and three Hikvision DS2DC3A20IW-D variable-focus cameras, which collect images of 6 types of litchi pests and diseases at different angles and distances, and the image resolution includes 3024 pixels × 4032 pixels and 3000 pixels × 4000 pixels. In order to collect sample images of lesions at different scales and different light intensities in natural environments, and collect sample images in different weather conditions such as sunny days, cloudy days and after rain, and include angles such as front light and back light, so as to increase the sample diversity of pest and disease images and restore the detection in actual scenes. A total of six types of litchi pests and diseases were collected, including four types of pests such as stink bugs, borer pupae, loopers and leaf gall midges, as well as two diseases of anthrax and algal spot. Finally, 4312 images of litchi pests and diseases were collected;

[0022] First, process the original image, use the SAM model to remove the pest background on the original image, and use black as the background, such as Figure 1 As shown in (b), the original image is cropped to 512×512 pixels. Figure 1 (c) is shown. Subsequently, the training set images are amplified, and the segmented and cropped images are mirror-flipped, rotated 90° counterclockwise, and the brightness is adjusted to form the diffusion model training set.

[0023] In a preferred embodiment, in step S2, the Shifted Diffusion model uses a pre-trained CLIP image encoder to guide the noise generation process. Shifted Diffusion also adds a shift term at each time step in the diffusion process, providing additional control and flexibility. The CLIP-based Shifted Diffusion model consists of three key components: (1) an input image encoder, which is responsible for mapping the image to its embedded representation; (2) a decoder, which generates an image based on the corresponding embedding; and (3) a prior model that can generate image embeddings. Given a pure image dataset, the generative model can be trained or fine-tuned by freezing the encoder and training the decoder to generate images based on the image embedding. When fine-tuning using Stable Diffusion 2 as the base model, a projection layer is introduced to map the CLIP image embedding into four vectors, which are then input into the U-Net architecture. In the image-to-image generation process, the fine-tuned model encodes the input image using CLIP, and the U-Net iteratively denoises by predicting and subtracting noise from the noise map, ultimately generating a refined output image.

[0024] In a preferred embodiment, in step S3, InternImage replaces the CSPDarknet backbone network in YOLOv5 for feature extraction, and its structure is as follows: Figure 3 As shown in the figure, it consists of stem, downsampling layer and basic blocks. The stem consists of 3×3 convolution, LN and GELU, which performs preliminary processing and feature extraction on the input image. The downsampling layer downsamples the input feature map by 2 times. The basic block includes LN, feedforward network (FFN) and GELU, and its core operator is DCNv3, G i is the number of DCNv3 groups in the i-th stage, L i is the number of basic blocks in the i-th stage. In this study, InternImage-T is used, and the number of basic blocks in the four stages is 4, 4, 18, and 4, respectively. The input features are passed through separable convolution in the basic blocks to predict the sampling offsets Δp and modulation scales m. For the input x and the current pixel p0, DCNv3 can be expressed as formula (1), where K represents the total number of sampling points, w g represents the projection weight of group g, Δp gk is the sampling point p in group g k The corresponding offset.

[0025]

[0026] In a preferred embodiment, in step S3, PAFPN is used as the Neck layer to connect the backbone network and the Head layer, including two paths from top to bottom and bottom to top, and performs path aggregation at different levels of the feature pyramid to achieve information transmission and fusion between features at different levels, and output feature maps of three different scales. The Channel Mapper module performs convolution operations on the input feature map, and combines normalization and activation functions to align the number of channels.

[0027] In a preferred embodiment, in step S3, a dynamic detection head DyHead is used. In the natural environment, the scale and spatial distribution of litchi pests and diseases are not the same, and they are easy to be similar to the background, which increases the difficulty of detection. Therefore, after the PAFPN of Diff-InternImage-YOLO, the Channel Mapper module is inserted to align the number of channels, and a dynamic detection head (Dynamic Head, DyHead) is added to further process the high-resolution multi-scale semantic information generated by PAFPN and enhance the extraction of effective feature information. DyHead integrates the scale-aware, space-aware, and task-aware attention mechanisms into one framework, deploys the scale-aware attention module in the horizontal dimension of the feature tensor, and the space-aware attention module is deployed in its spatial dimension, and the task-aware attention module is deployed in its channel dimension. For a given feature vector DyHead uses three types of attention in superposition, as shown in formula (2).

[0028]

[0029] Among them, π L is the scale-aware attention function deployed in the horizontal dimension, π S is the spatially aware attention function deployed on the spatial dimension and, π C It is a task-aware attention function deployed on the channel dimension. DyHead is composed of a stack of DyHeadBlocks. The structure of DyHead Block is as follows Figure 4 As shown in Figure 2, the attention module is stacked twice in Diff-InternImage-YOLO.

[0030] In a preferred embodiment, in step S3, the CP-Cluster post-processing is based on the graph model and the confidence message propagation, and does not rely on the ordering of the prediction boxes according to the confidence values, so full parallelism can be achieved. CP-Cluster does not assume that the prediction box with the highest confidence is necessarily the best, but makes full use of the relationship between the prediction boxes, draws on the iterative information transmission in the belief propagation, designs positive information to improve the confidence of positive samples, and negative information to suppress redundant boxes. The framework of CP-Cluster is as follows Figure 5 As shown in the figure, first, connections between the prediction boxes are created according to their IOUs to form several undirected graphs; then, positive messages (blue arrows) and negative messages (orange arrows) are iteratively propagated in each graph to improve the confidence of positive samples and suppress redundant boxes; finally, the prediction boxes with high confidence are left.

[0031] In a preferred embodiment, in step S5, the generated image is evaluated using Fréchet Inception Distance (FID) and CLIP (Contrastive Language-Image Pretraining) score. FID and CLIP score are commonly used to evaluate the performance of generative models. FID uses the pre-trained InceptionV3 model to extract image features and measures the quality of the generated image by comparing the distribution differences between the generated image and the real image in the feature space. The lower the FID value, the closer the distribution of the generated image is to the real image, that is, the higher the quality of the generated image. The calculation formula of FID is:

[0032] FID=‖μ r -μ g || 2 +Tr(∑ r +∑ g -2(∑ r ∑ g ) 1 / 2 ) (3).

[0033] Among them, μ r , μ g are the feature means of the real image and the generated image, ∑ r ,∑ g are the covariance matrices of the real image and the generated image respectively, and Tr represents the trace of the matrix.

[0034] In a preferred embodiment, in step S5, the original image and the generated image are mapped to the same embedding space by CLIP, a multimodal learning method proposed by OpenAI, and their cosine similarity in the embedding space is calculated to obtain the CLIP score between the generated image and the original image, where the higher the value, the higher the similarity. CLIP score focuses on the correlation between the generated image and the original image, while FID focuses more on the visual quality and authenticity of the generated image.

[0035] In a preferred embodiment, in step S5, in order to quantitatively evaluate the performance of the detection model, the mean average precision (mAP), accuracy (P), recall (R), harmonic score F1-score, model size, and inference speed of six categories of pests and diseases are used as evaluation indicators. Precision P represents the ratio of the number of correctly detected pests and diseases to the number of all detected targets; recall R represents the ratio of the number of correctly detected pests and diseases to the number of all real pests and diseases; F1-score is the harmonic mean of precision and recall, which comprehensively considers the precision and recall of the model. The area enclosed by the PR curve and the coordinate axis is the average precision (AP). mAP@0.5 refers to the average value of AP of all categories when the intersection over union (IoU) threshold is set to 0.5, and mAP@0.5:0.95 refers to the average value of AP calculated with an increment of 0.05 within the range of IoU thresholds from 0.5 to 0.95.

[0036] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0037] 1. In the present invention, the InternImage model uses the 3x3DCNv3 operator to achieve adaptive spatial aggregation and dynamic receptive field, effectively captures global image information and strengthens feature extraction, especially in complex orchard environments. Compared with other backbone networks (such as ResNet50, ConvNeXt and SwinTransformer), Internation improves detection accuracy. In addition, the DyHead module integrates scale, space and task-aware attention mechanisms, which enhances the model's ability to detect objects of various sizes and distinguish between targets with close or overlapping positions. In addition, compared with NMS, CP-Cluster is a new clustering framework that is fully parallelizable. CP-Cluster is used in the post-processing stage to enhance the real positive boxes by generating positive information and synthesizing negative information to punish redundant boxes, thereby enhancing the detection results.

[0038] 2. In the present invention, the diffusion model has more advantages in training stability, image quality control and output diversity. After incorporating the images generated by ShiftedDiffusion into the training process, the detection mAP of Diff-InternImage-YOLO is further improved by 2.6%, which shows that image generation technology has the potential to reduce the burden of data collection and improve detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 For the present invention Figure 1(a) Original image (b) Segmented image (c) Cropped image;

[0040] Figure 2 Schematic diagram of the workflow of the litchi pest and disease detection method of the present invention;

[0041] Figure 3 This is the Diff-InternImage-YOLO network structure in the present invention.

[0042] Figure 4 It is the structure of DyHeadBlock in the present invention;

[0043] Figure 5 The Overall pipeline of CP-Cluster in the present invention

[0044] Figure 6 The original image and generated image of litchi pests and diseases in the present invention

[0045] Figure 7 This is an example of the detection results of DiffInternImage-YOLO, YOLOv5-s and YOLOv8-s in the present invention, where the ground truth bounding box has been marked on the left image. The detection results obtained using Diff-InternImage-YOLO, YOLOv5-s and YOLOv8-s are shown in the three columns of images on the right. Different categories are represented by borders of different colors, and the numbers on the boxes are confidence scores. Yellow rounded rectangular boxes represent missed detections of all categories, and gray rounded rectangular boxes represent false detections. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0047] Reference Figure 1-7 ,

[0048] A litchi pest and disease detection method based on a large-scale visual model and a diffusion model, the method comprising the following steps:

[0049] S1: Image acquisition

[0050] The images of litchi pests and diseases in this study were collected in the litchi orchard (23°1′58.29″N, 113°3′59.93″E) of the teaching and research base of the South China Agricultural University in Wushan, Guangzhou, China. The images were collected from May to July 2023 under the guidance of agricultural experts. The collection equipment was a smartphone and three Hikvision DS2DC3A20IW-D (4 million pixels) variable focal length cameras to collect images of 6 types of litchi pests and diseases at different angles and distances. The image resolutions included 3024 pixels × 4032 pixels and 3000 pixels × 4000 pixels. In order to collect sample images of lesions at different scales and different light intensities in natural environments, sample images were collected in different weather conditions, such as sunny days, cloudy days and after rain, and angles such as front light and back light were included to increase the sample diversity of pest and disease images and restore the detection in actual scenes. A total of six types of litchi pests and diseases were collected, including four types of pests, namely stink bugs, peduncle borer pupae, geometrids and leaf gall midges, as well as two diseases, anthracnose and algal spot. In the end, 4,312 images of litchi pests and diseases were collected.

[0051] S2: Dataset construction

[0052] The labeling tool LabelImg is used to label the pests and disease spots in the above 4312 images with rectangular boxes. The number of targets in each category is shown in Table 1.

[0053] 20% of the dataset is randomly selected as the validation set for the detection network, and 80% is used to train the diffusion model and the detection network.

[0054] Table 1 Litchi pest and disease dataset

[0055]

[0056] Since the images in the dataset were taken in an orchard, a single image usually contains multiple pests and diseases, and the background is complex. In order to improve the diffusion model's ability to learn various litchi pests and diseases, it is necessary to preprocess the images. First, process the original image, use the SAM model to remove the pest and disease background on the original image, and use black as the background, such as Figure 1 As shown in (b), the original image is cropped to 512×512 pixels. Figure 1 (c) is shown. Subsequently, the training set images are amplified, and the segmented and cropped images are mirror-flipped, rotated 90° counterclockwise, and the brightness is adjusted to form the diffusion model training set.

[0057] S3: Generate pest and disease images using the Shifted Diffusion model

[0058] Current research on diffusion models has focused on the Denoising Diffusion Probabilistic Model (DDPM), which typically utilizes a standard Gaussian distribution before generating noise and is the basis for popular models such as Stable Diffusion. ShiftedDiffusion proposes a new approach that leverages a pre-trained CLIP image encoder to guide the noise generation process. Unlike DDPM, the noise distribution used in Shifted Diffusion is not a standard Gaussian distribution, but a parametric distribution derived from the training data. In addition, Shifted Diffusion incorporates a shift term at each time step in the diffusion process, providing additional control and flexibility. The CLIP-based Shifted Diffusion model consists of three key components: (1) an input image encoder that maps an image to its embedding representation; (2) a decoder that generates an image based on the corresponding embedding; and (3) a prior model that generates an image embedding. Given a pure image dataset, the generative model can be trained or fine-tuned by freezing the encoder and training the decoder to generate images based on the image embedding. When fine-tuning using Stable Diffusion 2 as the base model, a projection layer is introduced to map the CLIP image embedding into four vectors, which are then fed into the U-Net architecture. In the image-to-image generation process, the fine-tuned model encodes the input image using CLIP, and U-Net iteratively denoises it by predicting and subtracting noise from the noise map, ultimately generating a refined output image.

[0059] S3-1: Large-scale base model InternImage

[0060] InternImage is a large-scale visual infrastructure model based on CNN. Expanding the model and increasing parameters are important strategies to improve model performance. InternImage is the first convolutional model to be expanded to over one billion parameters, achieving performance comparable to or even better than the state-of-the-art ViT. The basic blocks of InternImage draw on the network structure of ViT, while the backbone and downsampling layers follow the design of traditional CNN.

[0061] InternImage uses DeformableConvolutionv3 (DCNv3) as its core operator. DCNv3 inherits the inductive bias of conventional convolution and also makes up for the shortcomings of conventional convolution in long-range dependency and adaptive spatial aggregation. Compared with multi-head self-attention, it has higher computational and memory efficiency and can achieve higher efficiency with shorter training time and less training data.

[0062] S3-2: Building a litchi pest and disease detection model based on InternImage

[0063] In order to improve the feature extraction capability and adapt to the task of pest and disease detection in orchard scenarios, this paper will use YOLOv5 as the detection framework, introduce the large-scale basic model InternImage as the backbone network, and combine the dynamic detection head (Dynamic Head, DyHead) based on the attention mechanism to effectively realize multi-scale feature fusion. In post-processing, Confidence Propagation Cluster (CP-Cluster) is used to replace the NMS-based method to improve the model detection accuracy. The litchi pest and disease detection model Diff-InternImage-YOLO is proposed. The network structure is as follows: Figure 2 As shown. The input image is extracted with InternImage, and the extracted features are input into PAFPN (Path Aggregation Feature Pyramid Network) and ChannelMapper for feature fusion and alignment of the number of channels. The DyHead module introduces the attention mechanism to further process the fused features to generate multiple output features. The subsequent Head module predicts the output features to obtain the bounding box, category and confidence of the target. During the training process, the loss of Diff-InternImage-YOLO is obtained by the weighted sum of classification loss, bounding box loss and confidence loss. The classification loss and bounding box loss are both calculated using BCEloss, while the confidence loss is calculated using CIoUloss. According to the loss, the model updates the parameters through the back propagation algorithm to continuously optimize the performance of the model. Finally, in the prediction stage, the CP-Cluster algorithm screens the prediction results, removes boxes with more overlap, and outputs the detection results.

[0064] Feature extraction and feature fusion. InternImage replaces the CSP Darknet backbone network in YOLOv5 to extract features. Its structure is as follows Figure 2 As shown in the figure, it consists of stem, downsampling layer and basic blocks. The stem consists of 3×3 convolution, LN and GELU, which performs preliminary processing and feature extraction on the input image. The downsampling layer downsamples the input feature map by 2 times. The basic block includes LN, feedforward network (FFN) and GELU, and its core operator is DCNv3, G i is the number of DCNv3 groups in the i-th stage, L iis the number of basic blocks in the i-th stage. In this study, InternImage-T is used, and the number of basic blocks in the four stages is 4, 4, 18, and 4, respectively. The input features are passed through separable convolution in the basic blocks to predict the sampling offsets Δp and modulation scales m. For the input x and the current pixel p0, DCNv3 can be expressed as formula (1), where K represents the total number of sampling points, w g represents the projection weight of group g, Δp gk is the sampling point p in group g k The corresponding offset.

[0065]

[0066] PAFPN is used as the Neck layer to connect the backbone network and the Head layer. It includes two paths, top-down and bottom-up. It aggregates paths at different levels of the feature pyramid to achieve information transmission and fusion between features at different levels, and outputs feature maps of three different scales. The Channel Mapper module performs convolution operations on the input feature maps, and combines normalization and activation functions to align the number of channels.

[0067] Dynamic detection head DyHead. In the natural environment, the size and spatial distribution of litchi pests and diseases vary, and they are often similar to the background, which increases the difficulty of detection.

[0068] After Diff-InternImage-YOLO’s PAFPN, we insert the Channel Mapper module to align the number of channels and add a dynamic detection head (DyHead) to further process the high-resolution multi-scale semantic information generated by PAFPN and enhance the extraction of effective feature information. DyHead integrates the scale-aware, spatial-aware, and task-aware attention mechanisms into one framework, deploying the scale-aware attention module in the horizontal dimension of the feature tensor, the spatial-aware attention module in its spatial dimension, and the task-aware attention module in its channel dimension. For a given feature vector DyHead uses three types of attention in superposition, as shown in formula (2).

[0069]

[0070] Among them, π L is the scale-aware attention function deployed in the horizontal dimension, π S is the spatially aware attention function deployed on the spatial dimension and, π CIt is a task-aware attention function deployed on the channel dimension. DyHead is composed of a stack of DyHeadBlocks. The structure of DyHead Block is as follows Figure 4 As shown in Figure 2, the attention module is stacked twice in Diff-InternImage-YOLO.

[0071] CP-Cluster post-processing. In post-processing, Diff-InternImage-YOLO uses bounding box clustering CP-Cluster based on a graph model to replace NMS to achieve higher accuracy. Unlike NMS, CP-Cluster is based on a graph model and confidence message propagation. It does not rely on the sorting of prediction boxes according to confidence values, so it can achieve complete parallelism. CP-Cluster does not believe that the prediction box with the highest confidence is necessarily the best, but makes full use of the relationship between prediction boxes, draws on iterative information transmission in belief propagation, designs positive information to improve the confidence of positive samples, and negative information to suppress redundant boxes. The framework of CP-Cluster is as follows Figure 5 As shown in the figure, first, connections between the prediction boxes are created according to their IOUs to form several undirected graphs; then, positive messages (blue arrows) and negative messages (orange arrows) are iteratively propagated in each graph to improve the confidence of positive samples and suppress redundant boxes; finally, the prediction boxes with high confidence are left.

[0072] Evaluation indicators include:

[0073] In order to analyze the results of image generation, this paper uses Fréchet Inception Distance (FID) and CLIP (Contrastive Language-Image Pretraining) score to evaluate the generated images. FID and CLIP score are often used to evaluate the performance of generative models. FID uses the pre-trained InceptionV3 model to extract image features and measures the quality of generated images by comparing the distribution differences between generated images and real images in the feature space. The lower the FID value, the closer the distribution of the generated image is to the real image, that is, the higher the quality of the generated image. The calculation formula of FID is:

[0074] FID=‖μ r -μ g || 2 +Tr(∑ r +∑ g -2(∑ r ∑ g ) 1 / 2 ) (3)

[0075] Among them, μ r , μ gare the feature means of the real image and the generated image, ∑ r ,∑ g are the covariance matrices of the real image and the generated image respectively, and Tr represents the trace of the matrix.

[0076] Through CLIP, a multimodal learning method proposed by OpenAI, the original image and the generated image are mapped to the same embedding space, and their cosine similarity in the embedding space is calculated to obtain the CLIP score between the generated image and the original image. The higher the value, the higher the similarity. CLIP score focuses on the correlation between the generated image and the original image, while FID focuses more on the visual quality and authenticity of the generated image.

[0077] In order to quantitatively evaluate the performance of the detection model, this paper uses the mean Average Precision (mAP), precision (P), recall (R), harmonic score F1-score, model size, and inference speed of six categories of pests and diseases as evaluation indicators. Precision P represents the ratio of the number of correctly detected pests and diseases to the number of all detected targets; recall R represents the ratio of the number of correctly detected pests and diseases to the number of all real pests and diseases; F1-score is the harmonic mean of precision and recall, which comprehensively considers the precision and recall of the model. The area enclosed by the PR curve and the coordinate axis is the average precision (AP). mAP@0.5 refers to the average AP of all categories when the intersection over union (IoU) threshold is set to 0.5, and mAP@0.5:0.95 refers to the average AP calculated with an increment of 0.05 and within the range of IoU thresholds from 0.5 to 0.95.

[0078] S4: Train the Diff-InternImage-YOLO model using the mixed dataset and optimize it using classification loss, bounding box loss, and confidence loss.

[0079] Update model parameters through back-propagation algorithm to continuously improve the detection accuracy and robustness of the model

[0080] S5: Use the test set to evaluate the performance of the model, including mAP, accuracy, recall, F1 score and other indicators. Compare with other existing litchi pest and disease detection models to verify the superiority of Diff-InternImage-YOLO

[0081] S6: Deploy the trained Diff-InternImage-YOLO model to a mobile device or server for real-time litchi pest and disease detection. Develop a user interface to facilitate users to upload litchi images and perform pest and disease detection

[0082] S7: Regularly collect new litchi pest and disease images and update the training data set to maintain the timeliness and accuracy of the model. Monitor model performance and adjust and optimize according to actual conditions.

[0083] S8: Apply the Diff-InternImage-YOLO model to litchi pest and disease monitoring and management, helping fruit farmers to find problems in time and take corresponding prevention and control measures. Promote the Diff-InternImage-YOLO model so that it can be more widely used in the agricultural field.

[0084] 4. Experimental Results

[0085] 4.1 Evaluation of Generated Images

[0086] Shifted Diffusion generates 300 images for each of the six categories of pests and diseases, for a total of 1,800 images. Figure 6 The generated images are illustrated with examples. Table 2 shows the FID and CLIP scores of Shifted Diffusion on six types of pests and diseases. The images generated by Shifted Diffusion have clear pest and disease contours and realistic tones, demonstrating the ability of Shifted Diffusion to generate high-quality litchi pest and disease images in an orchard environment.

[0087] Table 2 Results of the generation model for six types of pests and diseases

[0088]

[0089] 4.2 Backbone Network

[0090] In order to verify the effectiveness and superiority of InternImage in litchi pest and disease detection, a comparative analysis was conducted with other backbone networks. These backbone networks were integrated into the mature Faster-RCNN and YOLOv5 object detection algorithms, representing single-stage and two-stage algorithms, respectively. Each model was trained for 200 rounds on the mixed training dataset. Compared with the two-stage algorithm, the single-stage algorithm directly predicts on the image without region proposals and has a faster processing speed. Therefore, in the task of plant pest and disease detection, the performance of YOLOv5 is often competitive, even better than two-stage detectors such as Faster-RCNN, especially in terms of speed and real-time performance. As shown in Table 3, YOLOv5 with InternImage as the backbone network achieved the highest mAP@0.5 compared with other backbone architectures. Faster-RCNN with InternImage as the backbone network achieved the highest mAP@0.5:0.95 compared with other backbone architectures. This shows that InternImage can well extract effective features from litchi pest and disease images in orchard environments, thereby improving detection accuracy and robustness.

[0091] Table 3. Detection performance evaluation using Faster-RCNN and YOLOv5 with various backbone architectures on the litchi pest and disease dataset.

[0092]

[0093] 4.3 Ablation Experiment

[0094] In order to verify the effectiveness of the image generation data augmentation method for litchi pest and disease detection, different data augmentation methods were used to train the Diff-InternImage-YOLO model. The basic data augmentation methods include mirroring, increasing / decreasing brightness, adding Gaussian noise, random horizontal flipping, Mosaic, RandomAffine random affine transformation, and HSV color space enhancement. The experimental results are shown in Table 4. As can be seen from the table, both types of data augmentation methods have a positive effect on pest and disease detection, improve the mAP of the Diff-InternImage-YOLO model, and have a significant impact on the model accuracy. Among them, the combination of basic data augmentation method and image generation achieved the highest mAP@0.5 value of 88.2%, which is 2.6% higher than that using the basic data augmentation method.

[0095] Table 4 Performance comparison of different data augmentation methods

[0096]

[0097] To further evaluate the effectiveness of the proposed Diff-InternImage-YOLO model for litchi pest and disease detection, another ablation study was conducted with the same parameter settings, combining traditional data augmentation techniques and ShiftedDiffusion-based image generation techniques. The impact of InternImage, DyHead, and CPcluster on the overall performance of Diff-InternImage-YOLO was evaluated by replacing InternImage with CSPDarknet, removing DyHead, and replacing CP-cluster with NMS in the ablation experiments, where “×” indicates the removal of a component. The results shown in Table 5 demonstrate the effectiveness of the component. Compared with Diff-InternImage-YOLO, both mAP@0.5 and mAP@[0.5:0.95] are reduced without InternImage, DyHead, and CP cluster.

[0098] Table 5. Ablation experiment of Diff-InternImage-YOLO. The symbol “√” indicates that a component is included, and the symbol “×” indicates that a component is excluded.

[0099]

[0100]

[0101] 4.4 Comparison and Verification

[0102] To evaluate the performance of Diff-InternImage-YOLO with existing methods, a comparative analysis was performed. Each model was trained for 200 times and evaluated on the training and validation datasets using the same parameter settings. The comparison results of each model are shown in Table 6. As shown in Table 6, Diff-InternImage-YOLO significantly outperforms all compared methods in terms of mAP@0.5 and mAP@[0.5:0.95]. In addition, it also achieved the highest P, R, and F1 scores compared to all evaluated methods. Specifically, it significantly improves the accuracy by 1.0%, 5.7%, 3.9%, 9.2%, and 4.0% compared to Faster R-CNN, Deformable DETR, Swin Transformer, YOLOv5-s, and YOLOv8-s. Similarly, compared with Faster R-CNN, Deformable DETR, Swin Transformer, YOLOv5-s, and YOLOv8-s, the F1 score is improved by 5.9%, 6%, 2.2%, 6.9%, and 7.6%, respectively. These results show that Diff-InternImage-YOLO has excellent accuracy in identifying litchi pests and diseases. Figure 7 The detection examples of Diff-InternImage-YOLO, YOLOv5-s and YOLOv8-s are shown. It can be seen that compared with Diff-InternImage-YOLO, other methods have higher missed detection and false detection rates, and the confidence of correctly detected targets is also lower.

[0103] Table 6. Detection performance comparison results.

[0104]

[0105] 5. Discussion

[0106] 5.1 Advantages of the proposed method

[0107] This study demonstrates the effectiveness of InternImage based on a novel large-scale CNN backbone network for identifying litchi pests and diseases. InternImage uses 3x3DCNv3 operators to achieve adaptive spatial aggregation and dynamic receptive field, effectively capturing global image information and strengthening feature extraction, especially in complex orchard environments. Compared with other backbone networks such as ResNet50, ConvNeXt, and SwinTransformer, Internation improves detection accuracy. In addition, the DyHead module integrates scale-, spatial-, and task-aware attention mechanisms, which enhances the model's ability to detect objects of various sizes and distinguish between objects with close or overlapping locations. In addition, CP-Cluster is a new clustering framework that is fully parallelizable compared to NMS. CP-Cluster is adopted in the post-processing stage to enhance the detection results by generating positive information to enhance the true positive boxes and synthesizing negative information to penalize the redundant boxes.

[0108] The diffusion model has advantages in training stability, image quality control, and output diversity. After incorporating the images generated by ShiftedDiffusion into the training process, the detection mAP of Diff-InternImage-YOLO is further improved by 2.6%, which shows that image generation technology has the potential to reduce the burden of data collection and improve detection accuracy.

[0109] 5.2 Limitations

[0110] Although Diff-InternImage-YOLO has strong detection performance, its inference speed is slow due to its complex structure and large number of parameters. Compared with SwinTransformer, InternImage shows good robustness to data scale, but it requires more training data to achieve optimal performance compared with CNN-based models such as ResNet and ConvNeXt. In this study, the computational requirements of diffusion model training limited the generated image resolution to 512×512 pixels. Higher resolution images can further improve detection performance. Future research will focus on optimizing the detection model architecture for efficiency and optimizing the generation model for cost-effectiveness, while ensuring that the generated images remain realistic and diverse.

[0111] 6. Conclusion

[0112] This study proposes a novel litchi pest and disease detection method Diff-InternImage-YOLO based on a large-scale visual base model InternImage and a diffusion model. The evaluation results show that the mAP@0.5 is 88.2% and the mAP@0.5:0.95 is 61.3%, indicating that the model can accurately detect litchi pests and diseases in a complex orchard environment. The model has a precision of 91.1%, and the recall and F1 scores are 83.6% and 87.2%, respectively. By using the InternImage model as the backbone network, Diff-InternImage-YOLO effectively extracts features from challenging images. In addition, the addition of the attention-based DyHead and the parallel post-processing algorithm CP-Cluster also improves its performance. After removing InternImage, DyHead and CP-Cluster, the mAP@0.5 is reduced by 12.9%, highlighting their important contributions. It turns out that data augmentation using images generated by shift diffusion is crucial to improving the generality and robustness of the model, which improves the mAP@0.5 performance by 2.6%. Comparison results show that Diff-InternImage-YOLO outperforms FasterR-CNN, DeformableDETR, SwinTransformer, YOLOv5-s, and YOLOv8-s. These findings highlight the potential of Diff-InternImage-YOLO in providing effective and reliable technical assistance for litchi pest and disease management, ultimately contributing to sustainable agricultural practices.

[0113] From the above, it can be known that: in the present invention, the InternImage model uses the 3x3DCNv3 operator to realize adaptive spatial aggregation and dynamic receptive field, effectively captures global image information and strengthens feature extraction, especially in complex orchard environments. Compared with other backbone networks (such as ResNet50, ConvNeXt and SwinTransformer), Internation improves detection accuracy. In addition, the DyHead module integrates scale, space and task-aware attention mechanisms, which enhances the model's ability to detect objects of various sizes and distinguish between targets with close or overlapping positions. In addition, compared with NMS, CP-Cluster is a new clustering framework that is fully parallelizable. CP-Cluster is used in the post-processing stage to enhance the real positive boxes by generating positive information and synthesize negative information to punish redundant boxes, thereby enhancing the detection results.

[0114] In this invention, the diffusion model has more advantages in training stability, image quality control and output diversity. After incorporating the images generated by ShiftedDiffusion into the training process, the detection mAP of Diff-InternImage-YOLO is further improved by 2.6%, which shows that image generation technology has the potential to reduce the burden of data collection and improve detection accuracy.

[0115] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0116] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A litchi pest and disease detection method based on a large-scale visual model and a diffusion model, characterized in that: The method comprises the following steps: S1: First, collect images. In an orchard environment, use a smartphone or a high-pixel camera to collect images of litchi pests and diseases at different angles, distances, and lighting conditions; Use the labeling tool LabelImg to mark the pests and disease spots in the above 4312 images with rectangular boxes S2: Select the Shifted Diffusion model as the image generation tool and perform fine-tuning; use the preprocessed litchi pest and disease images to train the Shifted Diffusion model so that it can generate high-quality litchi pest and disease images; A large number of litchi pest and disease images were generated using the trained Shifted Diffusion model and combined with the original dataset to form a hybrid dataset. S3: Select Intern Image as the backbone network of the detection model to take advantage of its powerful feature extraction capabilities; Combine PAFPN and Channel Mapper for feature fusion and alignment so that feature maps of different scales can effectively transmit information; The DyHead module is introduced to further optimize feature extraction and object detection by using its scale-aware, space-aware, and task-aware attention mechanisms. Use CP-Cluster as a post-processing algorithm to replace the traditional NMS to more accurately distinguish true positives and redundant boxes; S4: Train the Diff-InternImage-YOLO model using the mixed dataset and optimize it using classification loss, bounding box loss, and confidence loss; Update model parameters through the back-propagation algorithm to continuously improve the detection accuracy and robustness of the model; S5: Use the test set to evaluate the performance of the model, including mAP, accuracy, recall, and F1 score indicators; compare with other existing litchi pest and disease detection models to verify the superiority of Diff-InternImage-YOLO; S6: Deploy the trained Diff-InternImage-YOLO model to a mobile device or server for real-time litchi pest and disease detection; develop a user interface to facilitate users to upload litchi images and perform pest and disease detection; S7: Regularly collect new litchi pest and disease images and update the training data set to maintain the timeliness and accuracy of the model; monitor the model performance and adjust and optimize it according to the actual situation; S8: Apply the Diff-InternImage-YOLO model to litchi pest and disease monitoring and management to help fruit farmers identify problems in a timely manner and take corresponding prevention and control measures; promote the Diff-InternImage-YOLO model so that it can be more widely used in the agricultural field.

2. The litchi pest and disease detection method based on large-scale visual model and diffusion model as claimed in claim 1, characterized in that: In the step S1, the acquisition equipment is a smart phone and three Hikvision DS2DC3A20IW-D variable-focus cameras, which collect images of 6 types of litchi pests and diseases at different angles and distances, and the image resolution includes 3024 pixels × 4032 pixels and 3000 pixels × 4000 pixels; to collect sample images of disease spots at different scales and different light intensities in the natural environment, and to collect sample images in different weather conditions such as sunny days, cloudy days and after rain, and including front light and back light angles, so as to increase the sample diversity of pest and disease images and restore the detection in the actual scene; a total of six types of litchi pests and diseases were collected, namely, four types of pests, namely, stink bugs, borer pupae, loopers and leaf gall midges, and two types of diseases, namely, anthrax and algal spot disease; finally, 4312 images of litchi pests and diseases were collected; First, the original image is processed, and the background of pests and diseases on the original image is removed using the SAM model, and black is used as the background. Then, the training set images are amplified, and the segmented and cropped images are mirrored, rotated 90° counterclockwise, and the brightness is adjusted to form the diffusion model training set.

3. The litchi pest and disease detection method based on large-scale visual model and diffusion model as claimed in claim 1, characterized in that: In step S2, the Shifted Diffusion model uses a pre-trained CLIP image encoder to guide the noise generation process; Shifted Diffusion also adds a shift term at each time step in the diffusion process, thereby providing additional control and flexibility; the CLIP-based Shifted Diffusion model consists of three key parts: (1) an input image encoder, which is responsible for mapping the image to its embedded representation; (2) a decoder, which generates an image based on the corresponding embedding; (3) a prior model, which can generate image embeddings; given a pure image dataset, the generative model is trained or fine-tuned by freezing the encoder and training the decoder to generate images based on the image embedding; when fine-tuning using Stable Diffusion 2 as the base model, a projection layer is introduced to map the CLIP image embedding into four vectors, which are then input into the U-Net architecture; in the process of generating image to image, the fine-tuned model uses CLIP to encode the input image, and the U-Net iteratively denoises by predicting and subtracting noise from the noise map, and finally generates a refined output image.

4. The litchi pest and disease detection method based on large-scale visual model and diffusion model as claimed in claim 1, characterized in that: In step S3, InternImage replaces the CSP Darknet backbone network in YOLOv5 for feature extraction, and is composed of stem, downsampling layer and basic blocks; the stem is composed of 3×3 convolution, LN and GELU, and performs preliminary processing and feature extraction on the input image; the downsampling layer downsamples the input feature map by 2 times; the basic block includes LN, feedforward network (FFN) and GELU, and its core operator is DCNv3, G i is the number of DCNv3 groups in the i-th stage, L i is the number of basic blocks in the i-th stage. In this study, InternImage-T is used, and the number of basic blocks in the four stages is 4, 4, 18, and 4 respectively. In the basic blocks, the input features are transferred through separable convolution to predict the sampling offsets Δp and modulation scales m. For the input x and the current pixel p0, DCNv3 is expressed as follows, where K represents the total number of sampling points and w g represents the projection weight of group g, Δp gk is the sampling point p in group g k The corresponding offset; 5. The litchi pest and disease detection method based on large-scale visual model and diffusion model as claimed in claim 1, characterized in that: In step S3, PAFPN is used as the Neck layer to connect the backbone network and the Head layer, including two paths from top to bottom and bottom to top, and performs path aggregation at different levels of the feature pyramid to achieve information transmission and fusion between features at different levels, and outputs feature maps of three different scales; the Channel Mapper module performs convolution operations on the input feature maps, and combines normalization and activation functions to align the number of channels.

6. The litchi pest and disease detection method based on large-scale visual model and diffusion model as claimed in claim 1, characterized in that: In the step S3, a dynamic detection head DyHead is performed; In the natural environment, the scale and spatial distribution of litchi pests and diseases are different, and they are easy to be similar to the background, which increases the difficulty of detection. Therefore, after the PAFPN of Diff-InternImage-YOLO, the Channel Mapper module is inserted to align the number of channels, and a dynamic detection head is added to further process the high-resolution multi-scale semantic information generated by PAFPN to enhance the extraction of effective feature information. DyHead integrates the scale-aware, space-aware and task-aware attention mechanisms into one framework, deploys the scale-aware attention module in the horizontal dimension of the feature tensor, the space-aware attention module in its spatial dimension, and the task-aware attention module in its channel dimension. For a given feature vector DyHead uses three types of attention in superposition, as shown in the formula; Among them, π L is the scale-aware attention function deployed in the horizontal dimension, π S is the spatially aware attention function deployed on the spatial dimension and, π C It is a task-aware attention function deployed in the channel dimension; DyHead consists of a stack of DyHead Blocks.

7. The litchi pest and disease detection method based on large-scale visual model and diffusion model as claimed in claim 1, characterized in that: In step S3, CP-Cluster post-processing is based on the graph model and confidence message propagation, and does not rely on the sorting of prediction boxes according to confidence values, thereby achieving complete parallelism; CP-Cluster does not believe that the prediction box with the highest confidence is necessarily the best, but makes full use of the relationship between prediction boxes, draws on iterative information transmission in belief propagation, designs positive information to improve the confidence of positive samples, and negative information to suppress redundant boxes; first, connections between prediction boxes are created based on their IOUs to form several undirected graphs; then, positive messages and negative messages are iteratively propagated in each graph to improve the confidence of positive samples and suppress redundant boxes; finally, prediction boxes with high confidence are left.

8. The litchi pest and disease detection method based on large-scale visual model and diffusion model as claimed in claim 1, characterized in that: In step S5, the generated image is evaluated using Fréchet Inception Distance (FID) and CLIP score; FID and CLIP score are often used to evaluate the performance of the generated model; FID uses the pre-trained InceptionV3 model to extract image features, and measures the quality of the generated image by comparing the distribution difference between the generated image and the real image in the feature space; the lower the FID value, the closer the distribution of the generated image is to the real image, that is, the higher the quality of the generated image; the calculation formula of FID is: FID=||μ r -μ g || 2 +Tr(∑ r +∑ g -2(∑ r ∑ g ) 1 / 2 ); Among them, μ r , μ g are the feature means of the real image and the generated image, ∑ r ,∑ g are the covariance matrices of the real image and the generated image respectively, and Tr represents the trace of the matrix.

9. The litchi pest and disease detection method based on large-scale visual model and diffusion model as claimed in claim 1, characterized in that: In step S5, the original image and the generated image are mapped to the same embedding space through CLIP, a multimodal learning method proposed by OpenAI, and their cosine similarity in the embedding space is calculated to obtain the CLIP score between the generated image and the original image. The higher the value, the higher the similarity. The CLIP score focuses on the correlation between the generated image and the original image, while the FID focuses more on the visual quality and authenticity of the generated image.

10. The litchi pest and disease detection method based on large-scale visual model and diffusion model as claimed in claim 1, characterized in that: In step S5, in order to quantitatively evaluate the performance of the detection model, the average precision, accuracy, recall rate, harmonic score F1-score of the six types of pests and diseases, model size, and inference speed are used as evaluation indicators; the precision P represents the ratio of the number of correctly detected pests and diseases to the number of all detection targets; the recall rate R represents the ratio of the number of correctly detected pests and diseases to the number of all real pests and diseases; the F1-score is the harmonic mean of the precision rate and the recall rate, which comprehensively considers the precision rate and the recall rate of the model; the area enclosed by the PR curve and the coordinate axis is the average precision; mAP@0.5 refers to the average AP of all categories when the intersection-over-union threshold is set to 0.5, and mAP@0.5:0.95 refers to the average AP calculated in increments of 0.05 over the range of IoU thresholds from 0.5 to 0.95.

Images