A rapid detection method for strawberry powdery mildew in greenhouse production environment

Abstract

The invention discloses a method for rapid detection of powdery mildew of strawberry under a greenhouse production environment. The method comprises: collecting back images of powdery mildew diseased leaves under different illumination conditions under a greenhouse strawberry production environment, and marking the back images according to diseases, diseased leaves and other leaves; constructing a DW-YOLOv4 network, wherein the network comprises a backbone network module and a fusion feature extraction module: the backbone network module is obtained by serially connecting five groups of cross-stage local network modules based on deep separable convolution; using three output feature maps of the third, fourth and fifth cross-stage local networks based on a deep separable convolution structure in the backbone network module as inputs to form output feature maps of three different scales; inputting the collected images into the DW-YOLOv4 network to realize detection of powdery mildew diseased areas and diseased leaves of strawberry leaves, thereby solving the problem of difficulty in detecting strawberry powdery mildew in the early stage of onset and realizing rapid detection and disease grading of strawberry powdery mildew.

Description

Technical Field

[0001] The technical solution of the present invention relates to the technical field of strawberry powdery mildew detection, and specifically to a rapid detection method for strawberry powdery mildew in a greenhouse production environment based on DW-YOLOv4. Background Art

[0002] Strawberry is one of the high-value fruits that has been deeply loved by users in recent years. In the north, it is planted in August-September every year and ends in March-April of the following year. As it is a winter crop, strawberries are usually cultivated in greenhouses. Affected by the greenhouse structure, problems such as high humidity, insufficient light, and poor ventilation are prone to occur during the strawberry planting period, providing a breeding ground for the occurrence of strawberry fungal diseases. Powdery mildew is a serious disease that has been prevalent in strawberry production in recent years. The disease has a high frequency of occurrence and high economic losses. After powdery mildew occurs, strawberry production generally decreases by 20%-30%, and in severe cases it can reach more than 50%, or even a total crop failure, seriously affecting the yield and quality of strawberries. In the early stage of strawberry powdery mildew, the affected parts will be covered with a layer of white powder, which is manifested as white hyphae on the leaf surface. As the disease worsens, dark spots and white powder appear on the leaves, and the leaf edges roll up. In the later stage, the spots turn reddish-brown, the leaves become brittle, and the leaf edges shrink and scorch. At present, the discovery of powdery mildew mainly relies on manual inspection, which has a long cycle and delayed discovery. In addition, due to the lack of expert participation, farmers are prone to inaccurate types and dosages of drugs used in diagnosis and application of drugs after discovery, which further causes losses in yield and quality and environmental pollution. Even if experts assist and guide on site, due to the limited number of observations, there is still a situation where the estimated disease index is based on experience and the data is inaccurate. With the development of artificial intelligence, it has become possible to use computer vision technology to identify diseases and diagnose diseases. The use of visual technology can realize disease monitoring at more points, diagnose the disease, and provide reasonable treatment plans. Therefore, rapid detection and identification of powdery mildew on strawberry leaves in a real production environment has become one of the main research directions of smart agriculture.

[0003] Jaemyung Shin et al. compared the effects of using AlexNet, SqueezeNet, GoogLeNet, ResNet-50, SqueezeNet-MOD1 and SqueezeNet-MOD2 to identify powdery mildew on strawberry leaves in "Adeep learning approach for RGB image-based powdery mildew disease detection on strawberry leaves". They determined that ResNet-50 had a higher recognition rate, while SqueezeNet-MOD2 had a slightly lower recognition rate than ResNet-50. However, the dataset they used was images in a simple laboratory environment, and they could only identify whether the leaves were diseased. They did not detect the location of the disease or estimate the severity of the disease. Patrick Wspanialy et al. proposed a method for evaluating the severity of greenhouse tomato diseases in "A detection and severity estimation system for generic diseases of tomato greenhouse plants". However, they also used leaf disease samples collected in a simple laboratory environment. The instance segmentation method was used to identify the disease. The identified leaves and disease edges were irregular areas. The algorithm ran slowly and was difficult to promote in practical applications. A Chinese patent with publication number CN 112150481 A discloses a powdery mildew image segmentation method, which uses a powdery mildew grayscale image to perform instance segmentation of disease points, and then calculates the degree of disease by the ratio of the pixels of the powdery mildew disease area to the pixels of the collected image. The denominator is the pixels of the collected image, not the pixels of the complete leaf, which is prone to deviations in the calculation of the degree of disease. In addition, the algorithm efficiency of instance segmentation is often lower than the recognition efficiency of the target detection algorithm, and cannot meet the real-time requirements in practical applications. Summary of the invention

[0004] In view of the shortcomings of the prior art, the technical problem to be solved by the present invention is: a rapid detection method for strawberry powdery mildew based on DW-YOLOv4 (Depthwise separable-You Only Look Once v4, deep separable YOLOv4 network) in a greenhouse production environment is proposed. The method embeds a deep separable convolution structure into a YOLOv4 framework, realizes the detection of powdery mildew diseased areas and diseased leaves of strawberry leaves through multi-scale target detection, and estimates the disease index based on the area of ​​the disease detection area, solves the problem of difficult detection of strawberry powdery mildew in the early stage of onset, and realizes rapid detection and disease grading of strawberry powdery mildew.

[0005] The technical solution adopted by the present invention to solve the technical problem is: a rapid detection method for strawberry powdery mildew in a greenhouse production environment, the method comprising the following contents:

[0006] Collect images of the back of leaves infected with powdery mildew under different lighting conditions in a greenhouse strawberry production environment. The image background is the strawberry growth environment. Label the back images according to disease, infected leaves, and other leaves to create a data set.

[0007] Build a DW-YOLOv4 network, which consists of two parts: the backbone network module and the fusion feature extraction module:

[0008] Backbone network module: five groups of cross-stage local network modules DW-CSPNet based on deep separable convolution are connected in series, where DW-CSPNet includes downsampling through convolutional layers with a convolution kernel size of 3×3, compression or expansion of channels through convolutional layers with a convolution kernel size of 1×1, deep separable convolution with a convolution kernel of 3×3, channel splicing, and feature map summation operations;

[0009] Fusion feature extraction module: Using the three output feature maps of the 3rd, 4th, and 5th cross-stage local network DW-CSPNet based on the deep separable convolution structure in the backbone network module as input, the fusion feature extraction module includes maximum pooling, channel splicing, convolution layer with a convolution kernel size of 1×1, upsampling, and downsampling, and finally forms three output feature maps of different scales;

[0010] The collected images are input into the DW-YOLOv4 network to detect the powdery mildew diseased areas and diseased leaves of strawberry leaves.

[0011] The disease index was obtained based on the ratio of the powdery mildew disease area to the diseased leaf area.

[0012] The specific calculation process of the disease index is:

[0013] The incidence rate of the fusion adjustment coefficient is calculated using formula (24),

[0014]

[0015] Among them, a i represents the incidence of the i-th diseased leaf, A represents the area of ​​the disease, j represents the j-th disease point on the i-th diseased leaf, n represents the number of disease points on the i-th diseased leaf, B i represents the area of ​​the i-th diseased leaf, σ represents the leaf area adjustment coefficient;

[0016] The leaf area adjustment coefficient was calculated by using binarization and OTSU to extract the number of green pixels from the marked diseased leaf part and dividing it by the number of pixels in the marked box of the diseased leaf;

[0017] Calculate disease index for individual leaves The disease is divided into levels 1-9. When the incidence rate is less than 5%, the disease level is 1; when the incidence rate is between 5%-25%, the disease level is 3; when the incidence rate is between 25%-50%, the disease level is 5; when the incidence rate is between 50%-75%, the disease level is 7; when the incidence rate is greater than 75%, the disease level is 9. The specific formula is as follows:

[0018]

[0019] Where i represents the i-th diseased leaf;

[0020] The overall disease index is calculated according to formula (26):

[0021]

[0022] Among them, N is the number of diseased leaves, and i is the i-th diseased leaf;

[0023] According to the different severity of the disease, corresponding expert treatment opinions will be given based on empirical data, and different disease indexes correspond to corresponding expert treatment opinions.

[0024] The specific steps of this detection method are:

[0025] The first step is to collect image data of strawberry leaf diseases in a greenhouse production environment and create a dataset

[0026] Step 1.1: In the strawberry production environment of the solar greenhouse, multiple sampling points are set to collect images of the back of diseased strawberry leaves under different lighting conditions in the real production environment. The sampling points should cover the entire greenhouse as much as possible. The back of the diseased leaves needs to be turned over manually, but the leaves do not separate from the plants. When sampling, the background includes other leaves other than the observed diseased leaves.

[0027] Step 1.2: Based on the sampling point selection, the diseased leaves are collected. Once the disease is found, all leaves of each plant are identified and photographed. All images are obtained by manual observation by professionals.

[0028] Step 1.3: Use the labelimg tool to label the image according to the disease (powdery mildew), diseased leaves (leaf with disease), and other leaves (leaf without disease). That is, the location of the disease is selected by box, the entire leaf where the disease is located is selected by box, and the leaf contours visible to the naked eye other than the diseased leaf are selected by box. For leaves with severe blur in the image, if the edge area cannot be identified by the naked eye, it will not be marked;

[0029] Step 1.4: Generate a dataset by collecting all images with disease and annotating them under different lighting backgrounds. Each image will contain one or more of the three annotations, with at least one annotation for diseased and infected leaves, and other leaves may have none or multiple annotations.

[0030] The second step is image preprocessing

[0031] Step 2.1: Take the image in the data set obtained in step 1.4 as input, process it by data enhancement, and splice it using the mosaic method to form an enhanced image;

[0032] In step 2.2, the enhanced images in step 2.1 are randomly scaled to 10-19 times of 32 for training;

[0033] The third step is to build the backbone network module of DW-YOLOv4 using the deep separable convolution based cross-stage local network (DW-CSPNet)

[0034] Step 3.1: Construct a cross-stage local network module based on deep separable convolution (DW-CSPNet)

[0035] In step 3.1.1, the output image D of the previous step is used as input, and the convolution kernel size is 3×3, the step length is 2, and the number of output channels is C. out The downsampling layer performs image processing to form a feature map D1, as shown in formula (1).

[0036]

[0037] in, Indicates a convolution operation with a kernel size of 3×3, a stride of 2, and normalization and a LeakyReLU activation function;

[0038] In step 3.1.2, the processed results are divided into two branches. One branch uses a convolution layer with a convolution kernel size of 1×1 to compress the channel to form a feature map D2, as shown in formula (2).

[0039] D2=γ1(D1) (2),

[0040] Among them, γ1 represents the convolution operation with a convolution kernel size of 1×1, normalization and LeakyReLU activation function;

[0041] In step 3.1.3, another branch passes through the convolution layer compression channel with a convolution kernel size of 1×1 to form a feature map D3. The feature of D3 is extracted using a depthwise separable convolution with a convolution kernel size of 3×3 and a convolution layer with a convolution kernel size of 1×1, and the feature map D4 is obtained by summing it with D3, as shown in formula (3).

[0042] D3=γ1(D1), D4=γ1(δ3(D3))+D3 (3),

[0043] Among them, δ3 represents a 3×3, depth-wise separable convolution operation with normalization and LeakyReLU activation function; + represents the feature map summation operation;

[0044] In step 3.1.4, D2 and D4 are concatenated and then compressed using a convolutional layer with a convolution kernel size of 1×1 to form the output feature map DW_CSP of the cross-stage local network module based on depthwise separable convolution. Specifically, as shown in formula (4), the cross-stage local network module based on depthwise separable convolution is denoted as f DW_CSP , as shown in formula (5);

[0045]

[0046] DW_CSP=f DW_CSP (D) (5),

[0047] in, Represents the feature map concatenation operation;

[0048] Assume that the width, height, and number of channels of the input image D are w0, h0, and c0 respectively. The dimension of the feature map processed by DW-CSPNet changes, and the width, height, and number of channels of the output feature map DW_CSP are w0 / 2, h0 / 2, and 2c0 respectively, which achieves the purpose of downsampling and increasing the depth at the same time.

[0049] Step 3.2: Build the backbone network of DW-YOLOv4

[0050] Five cross-stage local networks based on depthwise separable convolution are connected in series in sequence. The output of the third cross-stage local network based on depthwise separable convolution is recorded as feature map DW_CSP1, as shown in formula (6); the output of the fourth cross-stage local network based on depthwise separable convolution is recorded as feature map DW_CSP2, as shown in formula (7); the output of the fifth cross-stage local network based on depthwise separable convolution is recorded as feature map DW_CSP3, as shown in formula (8); feature maps DW_CSP1, DW_CSP2 and DW_CSP3 will be used as input feature maps and participate in the calculation of the fourth step fusion feature extraction module;

[0051] DW_CSP1=f DW_CSP (f DW_CSP (f DW_CSP (D))) (6),

[0052] DW-CSP2=f DW_CSP (DW_CSP1) (7),

[0053] DW_CSP3=f DW_CSP (DW_CSP2) (8),

[0054] Among them, the number of output channels of the first cross-stage local network based on depthwise separable convolution is 32, and the others are processed according to the output channel number rule in step 3.1;

[0055] The fourth step is to build the fusion feature extraction module of DW-YOLOv4

[0056] Step 4.1: Take DW_CSP3 as input and use 3×3 convolution to perform channel compression to obtain the feature map D 4.1 , as shown in formula (9);

[0057] D 4.1 =γ3(DW_CSP3) (9),

[0058] Among them, γ3 represents the convolution operation with a convolution kernel size of 3×3, normalization and LeakyReLU activation function;

[0059] Step 4.2, for D 4.1 Perform maximum pooling with step sizes of 5, 9, and 13, and compare the obtained results with D 4.1 Perform channel splicing to obtain feature map D 4.2 , as shown in formula (10);

[0060]

[0061] Among them, τ5, τ9, τ 13Respectively represent the maximum pooling operations with step sizes of 5, 9, and 13;

[0062] Step 4.3: Use a 1×1 convolution kernel to adjust D 4.2 Perform channel compression to obtain feature map D 4.3 , as shown in formula (11),

[0063] D 4.3 =γ1(D 4.2 ) (11),

[0064] Step 4.4, use 1×1 convolution on D 4.3 After channel compression, upsampling is performed, channel splicing is performed with DW_CSP2, and then channel compression is performed using a 1×1 convolution kernel to obtain the feature map D 4.4 , as shown in formula (12),

[0065]

[0066] Among them, σ is the upsampling operation;

[0067] Step 4.5: Use 1×1 convolution on D 4.4 After channel compression, upsampling is performed, channel splicing is performed with DW_CSP1, and then channel compression is performed using a 3×3 convolution kernel to obtain the feature map D 4.5 , as shown in formula (13),

[0068]

[0069] Step 4.6: Use a convolution kernel of size 3×3 and stride 2 to calibrate D 4.5 Downsampling and channel expansion, with D 4.4 After splicing, a 3×3 convolution kernel is used to perform channel compression to obtain the feature map D 4.6 , as shown in formula (14),

[0070]

[0071] Step 4.7: Use a convolution kernel of size 3×3 and stride 2 to adjust D 4.6 Downsampling and channel expansion, with D 4.3 After splicing, a 3×3 convolution kernel is used to perform channel compression to obtain the feature map D 4.7 , as shown in formula (15),

[0072]

[0073] Step 4.8: Use three l×1 convolutions to perform 4.5 , D 4.6 , D 4.7Adjust the output dimension to form the final feature map D 4.8.1 , D 4.8.2 , D 4.8.3 , as shown in formula (16),

[0074] D 4.8.1 =ω1(D 4.5 ), D 4.8.2 =ω1(D 4.6 ), D 4.8.2 =ω1(D 4.7 ) (16),

[0075] Among them, convolution ω1 is a convolution operation with a convolution kernel size of 1×1 without normalization and LeakyReLU activation function, thus completing the construction of the fusion feature extraction module of DW-YOLOv4;

[0076] The fifth step is to use the loss function that combines CIOU and exponentially enhanced binary cross entropy to calculate the loss between the predicted value output in step 4.8 and the actual value. The predicted value output in step 4.8 includes the coordinates of the center point of the target box, width, height, category, and confidence:

[0077] Total loss The formula is (23):

[0078]

[0079] in, is the loss function of the target detection task, To use exponentially enhanced binary cross entropy to represent the loss of category prediction value and actual value, is the confidence loss calculated using binary cross entropy;

[0080] The sixth step is to perform non-maximum suppression (NMS) on the model prediction results after loss calculation and adjust the results.

[0081] In step 6.1, NMS is used to process the prediction results of the model after loss calculation, and the prediction results of the three types of targets, namely, powdery mildew, leaf with disease, and leaf without disease, which do not exceed the NMS threshold, are obtained, including the coordinates of the center point of the target box, width, height, category, and confidence level;

[0082] Step 6.2: Adjust the prediction results after non-maximum suppression

[0083] Since the disease must occur on the leaves, the prediction results obtained in step 6.1 may not necessarily identify the diseased leaves. When the image identifies the disease but not the diseased leaves, the other leaf categories with the largest IoU with the disease are adjusted to diseased leaves. If there is no leaf with an IoU greater than 0 with the identified disease, the disease point is removed to ensure that all detected diseases have corresponding diseased leaves.

[0084] The backbone network module and the fusion feature extraction module, loss calculation and non-maximum suppression and adjustment together constitute the DW-YOLOv4 network;

[0085] Step 7: Calculate the disease index

[0086] The specific calculation process of the disease index is:

[0087] The incidence rate of the fusion adjustment coefficient is calculated using formula (24),

[0088]

[0089] Among them, a i represents the incidence of the i-th diseased leaf, A represents the area of ​​the disease, j represents the j-th disease point on the i-th diseased leaf, n represents the number of disease points on the i-th diseased leaf, B i represents the area of ​​the i-th diseased leaf, σ represents the leaf area adjustment coefficient;

[0090] The leaf area adjustment coefficient was calculated by using binarization and OTSU to extract the number of green pixels from the marked diseased leaf part and dividing it by the number of pixels in the marked box of the diseased leaf;

[0091] Calculate disease index for individual leaves The disease is divided into levels 1-9. When the incidence rate is less than 5%, the disease level is 1; when the incidence rate is between 5%-25%, the disease level is 3; when the incidence rate is between 25%-50%, the disease level is 5; when the incidence rate is between 50%-75%, the disease level is 7; when the incidence rate is greater than 75%, the disease level is 9. The specific formula is as follows:

[0092]

[0093] Where i represents the i-th diseased leaf;

[0094] The overall disease index is calculated according to formula (26):

[0095]

[0096] Among them, N is the number of diseased leaves, and i is the i-th diseased leaf;

[0097] According to the different degrees of disease, corresponding expert treatment opinions will be given through empirical data, and different disease indexes correspond to corresponding expert treatment opinions;

[0098] So far, the rapid detection and quantitative disease assessment of strawberry powdery mildew based on DW-YOLOv4 in the greenhouse production environment have been completed.

[0099] The present invention also protects a strawberry powdery mildew rapid detection system, including a memory and a processor, the memory stores the DW-YOLOv4 network and disease index calculation, and the program on the memory is used to execute the above-mentioned detection method when loaded by the processor.

[0100] The beneficial effects of the present invention are:

[0101] (1) The method of the present invention uses deep separable convolution to extract features, which greatly reduces the number of parameters required for the model. It does not reduce the accuracy while reducing the model size, and has a faster recognition speed. The DW-YOLOv4 model size is 20.7MB, and the YOLOv4 model size is 256.1MB. Compared with the YOLOv4 model, the DW-YOLOv4 model is more suitable for deployment on mobile devices. At the same time, the detection speed is also significantly improved. The detection speed of DW-YOLOv4 is 264fps, which is much higher than the 68fps detection speed of YOLOv4.

[0102] (2) The DW-YOLOv4 network constructed by the present invention includes a backbone module based on a deep separable convolutional structure and a fusion feature extraction module. The feature maps of different layers are fused in the fusion feature extraction module, which can simultaneously retain the features extracted by the shallow and deep networks, improve the network's ability to recognize targets of different sizes, and overcome the problem of difficulty in detection due to the small disease area in the early stage of powdery mildew and the large interference from light.

[0103] (3) The detection method of the present invention uses exponentially enhanced binary cross entropy to optimize the loss function to solve the problem of imbalance between the number of diseases and leaves, which can help the DW-YOLOv4 network pay more attention to category information.

[0104] (4) Based on the detection of strawberry powdery mildew disease and diseased leaves, the present invention further proposes a disease index calculation method for the leaf fusion adjustment coefficient, which overcomes the problem of low disease index caused by the area difference between the detection frame and the actual diseased leaf area, and can more accurately obtain the disease situation, providing a basis for the identification and prevention of strawberry powdery mildew.

[0105] (5) The data set of the detection method of the present invention is an image collected under the actual production environment of greenhouse strawberries, which can meet various lighting conditions and achieve good disease area detection under different lighting conditions, providing a basis for the detection of powdery mildew on strawberry leaves in actual production. BRIEF DESCRIPTION OF THE DRAWINGS

[0106] Figure 1 It is an overall flow chart of the strawberry powdery mildew rapid detection method based on DW-YOLOv4 of the present invention;

[0107] Figure 2 Schematic diagram of a deep separable convolution-based cross-stage local network (DW-CSPNet) of a strawberry powdery mildew rapid detection method based on DW-YOLOv4 of the present invention;

[0108] Figure 3 It is a backbone network module diagram of the strawberry powdery mildew rapid detection method based on DW-YOLOv4 of the present invention;

[0109] Figure 4 It is a fusion feature extraction module diagram of the strawberry powdery mildew rapid detection method based on DW-YOLOv4 of the present invention;

[0110] Figure 5 This is the effect of using the strawberry leaf disease image in the production environment to detect the strawberry powdery mildew rapid detection method based on DW-YOLOv4 of the present invention. Figure 1 ;

[0111] Figure 6 This is the effect of using the strawberry leaf disease image in the production environment to detect the strawberry powdery mildew rapid detection method based on DW-YOLOv4 of the present invention. Figure 2 . DETAILED DESCRIPTION

[0112] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0113] Figure 1 The illustrated embodiment shows that the overall process of a rapid detection method for strawberry powdery mildew based on DW-YOLOv4 in a greenhouse production environment of the present invention is as follows:

[0114] Collect strawberry image data in a greenhouse production environment and create a dataset → preprocess the image, including data enhancement, random scaling, etc. → use the backbone network module built based on a cross-stage local network with a deep separable convolutional structure to extract features → use the fusion feature extraction module to further extract features → calculate the loss → NMS non-maximum suppression → adjust the test results → obtain the test results → calculate the disease index.

[0115] Figure 2 The illustrated embodiment shows that the process of the strawberry powdery mildew rapid detection method based on DW-YOLOv4 of the present invention is as follows:

[0116] Use a convolution layer with a convolution kernel size of 3×3, a stride of 2, and an output channel number that is twice the number of input channels to process the input feature map → divide into two branches → the first branch uses a convolution layer with a convolution kernel size of 1×1 to compress the channel → the second branch uses a convolution layer with a convolution kernel size of 1×1 to compress the channel, uses a depthwise separable convolution with a convolution kernel size of 3×3 to extract features, and uses a convolution layer with a convolution kernel size of 1×1 to expand the number of channels. The output after the expansion of the number of channels is added to the result of the second branch compressing the channel → the result of the second branch is concatenated with the result of the first branch → use a convolution layer with a convolution kernel size of 1×1 to compress the channel.

[0117] Figure 3 The illustrated embodiment shows that the backbone network module process of the strawberry powdery mildew rapid detection method based on DW-YOLOv4 of the present invention is as follows:

[0118] Input feature map → Use 3 cross-stage local networks based on depthwise separable convolution to downsample and expand the number of channels to obtain the output feature map DW_CSP1 → The output feature map DW_CSP1 is directly output on the one hand, and on the other hand, it is output to the fourth cross-stage local network based on depthwise separable convolution, downsample and expand the number of channels to obtain the output feature map DW_CSP2 → The output feature map DW_CSP2 is directly output on the one hand, and on the other hand, it is output to the fifth cross-stage local network based on depthwise separable convolution, downsample and expand the number of channels to obtain the output feature map DW_CSP3. Therefore, the backbone network module has three outputs, namely DW_CSP1, DW_CSP2 and DW_CSP3.

[0119] Figure 4 The illustrated embodiment shows that the fusion feature extraction module of the strawberry powdery mildew rapid detection method based on DW-YOLOv4 of the present invention is used to fuse the three outputs of the backbone network module, and the specific process is as follows:

[0120] The last output feature map DW_CSP3 of the backbone network module is compressed using a convolution layer with a convolution kernel size of 3×3, and then three maximum pooling processes with step sizes of 5, 9, and 13 are performed respectively. The three maximum pooling layers are in parallel, that is, the inputs of the three are the same. The results of the three maximum pooling processes are then concatenated with the results of the 3×3 convolution layer compression channel in the previous step, and the convolution layer compression channel with a convolution kernel size of 1×1 is used again to obtain the feature map D 4.3 , feature map D 4.3 On the one hand, after upsampling, the second output feature map DW_CSP2 of the backbone network module is channel-joined, and then the feature map D is obtained by a convolution layer with a convolution kernel size of 1×1. 4.4 , feature map D 4.4On the one hand, the convolution layer with a convolution kernel size of 1×1 compresses the channel and then performs upsampling, and then performs channel splicing with the first output feature map DW_CSP1 of the backbone network module, and then uses the convolution layer with a convolution kernel size of 3×3 to compress the channel to obtain the feature map D 4.5 ; Feature map D 4.5 On the one hand, we use a convolutional layer with a convolution kernel size of 3×3 and a stride of 2 to downsample and expand the channel before adding it to the feature map D 4.4 The channel splicing is performed, and the result of the splicing of the two channels is compressed using a convolution layer with a convolution kernel size of 3×3 to obtain the feature map D 4.6 , feature map D 4.6 On the one hand, we use a convolutional layer with a convolution kernel size of 3×3 and a stride of 2 to downsample and expand the channel before adding it to the feature map D 4.3 Perform channel splicing. The result of channel splicing uses a convolution layer with a convolution kernel size of 3×3 to compress the channel to obtain the feature map D 4.7 , feature map D 4.7 The output feature map D is obtained by adjusting the channel through a convolution layer with a convolution kernel size of 1×1. 4.8.3 ;

[0121] Feature map D 4.6 On the other hand, the convolution layer with a convolution kernel size of 1×1 is also used to adjust the channel to obtain the output feature map D 4.8.2 , feature map D 4.5 On the other hand, the convolution layer with a convolution kernel size of 1×1 is also used to adjust the channel to obtain the output feature map D 4.8.1 , feature map D 4.8.3 , feature map D 4.8.2 , feature map D 4.8.1 are the three outputs of the fusion feature extraction module.

[0122] Figure 5 , Figure 6 It is a detection effect diagram of a strawberry leaf disease image in a greenhouse production environment after being detected using the strawberry powdery mildew rapid detection method based on DW-YOLOv4 of the present invention.

[0123] The specific steps of the method for rapid detection of strawberry powdery mildew under a greenhouse production environment of the present invention are as follows:

[0124] The first step is to collect image data of strawberry leaf diseases in a greenhouse production environment and create a dataset:

[0125] In step 1.1, multiple sampling points are set up in the strawberry production environment of the sunlight greenhouse to collect images of the back of diseased strawberry leaves under real production conditions under different lighting conditions. The sampling points should cover the entire greenhouse as much as possible. The back of the diseased leaves needs to be turned over manually, but the leaves are not separated from the plants. When sampling, the background includes other leaves (i.e., not the diseased leaves being observed). This sampling method actually has no effect on plant growth. The data set produced by the images collected in this way is significantly different from the data set collected by picking the leaves and taking pictures in the laboratory under a single background.

[0126] In this embodiment, 30 sampling points can be set, and the sampling points will be marked with labels in the greenhouse. A greenhouse is about 667 square meters, each greenhouse is about 60 furrows, and 1 furrow is about 7-8 meters long. About 80 strawberries are planted, so there are about 4,800 strawberries in a greenhouse, and each strawberry has about 20-40 leaves, so there are about 150,000 leaves in a greenhouse. Five sampling points will be selected at equidistant intervals in the 5th, 15th, 25th, 35th, 45th, and 55th furrows in the greenhouse, and a total of 6 furrows * 5 sampling points / furrow = 30 sampling points, so the selected sampling points cover the entire greenhouse as much as possible. Three plants are selected from each sampling point every week, and three leaves are selected from each plant to observe the incidence of powdery mildew on strawberry leaves.

[0127] Step 1.2: Based on the sampling point selection, the diseased leaves are collected. Once the disease is found, all leaves of each plant are identified and photographed. All images are obtained by manual observation by professionals.

[0128] Step 1.3: Use the labelimg tool to label the image according to the disease (powdery mildew), diseased leaves (leaf with disease), and other leaves (leaf without disease). That is, select the location of the disease, select the entire leaf where the disease is located, and select the leaf contours visible to the naked eye except the diseased leaf. For leaves with severe blur in the image, if the edge area cannot be identified by the naked eye, it will not be marked;

[0129] Step 1.4, generate a data set; create a data set by collecting and annotating all diseased images under different lighting background environments. In this embodiment, the data set contains 1023 images, and each image will contain one or more of the three types of annotations, among which diseased and diseased leaves will have at least one, and other leaves may have none or multiple.

[0130] The second step is image preprocessing:

[0131] In step 2.1, the images in the dataset obtained in step 1.4 are used as input. They are first processed by data enhancement methods such as brightness, contrast, hue, saturation, noise, random scaling, cropping, flipping, and rotation, and then spliced ​​using the mosaic method to form an enhanced image of 416×416×3.

[0132] In step 2.2, the enhanced image in step 2.1 is randomly scaled to 10-19 times of 32 for multi-scale training. The randomly scaled size is between 320×320 and 608×608. Different input scales have different output scales. For example, when the input of 320×320 is output to the fusion feature extraction module, it is 40×40, 20×20, and 10×10, while when the input of 608×608 is output to the fusion feature extraction module, it is 76×76, 38×38, and 19×19. The following text uses the image scale of 608×608×3 as an input example for explanation.

[0133] In the third step, the backbone network module of DW-YOLOv4 is constructed using the Deep Separable Convolution-based Cross-Stage Local Network (DW-CSPNet).

[0134] Step 3.1: Construct a cross-stage local network module based on deep separable convolution (DW-CSPNet)

[0135] In step 3.1.1, the output image D of the previous step is used as input, and the convolution kernel size is 3×3, the step length is 2, and the number of output channels is C. out The downsampling layer performs image processing to form a feature map D1, as shown in formula (1).

[0136]

[0137] in, It indicates a convolution operation with a kernel size of 3×3, a stride of 2, normalization, and a LeakyReLU activation function.

[0138] In step 3.1.2, the processed results are divided into two branches. One branch uses a convolution layer with a convolution kernel size of 1×1 to compress the channel to form a feature map D2, as shown in formula (2).

[0139] D2=γ1(D1) (2),

[0140] Among them, γ1 represents the convolution operation with a convolution kernel size of 1×1, normalization and LeakyReLU activation function.

[0141] In step 3.1.3, the other branch passes through a convolution layer with a convolution kernel size of 1×1 to compress the channel and form feature map D3. The feature of D3 is extracted using a depthwise separable convolution with a convolution kernel size of 3×3 and a convolution layer with a convolution kernel size of 1×1, and the feature map D4 is obtained by summing it with D3, as shown in formula (3).

[0142] D3=γ1(D1), D4=γ1(δ3(D3))+D3 (3),

[0143] Among them, δ3 represents a 3×3 depth-wise separable convolution operation with normalization and LeakyReLU activation function; + represents the feature map summation operation.

[0144] In step 3.1.4, D2 and D4 are concatenated and then compressed using a convolutional layer with a convolution kernel size of 1×1 to form the output feature map DW_CSP of the cross-stage local network module based on depthwise separable convolution, as shown in formula (4).

[0145] The cross-stage local network module based on depthwise separable convolution is denoted as f DW_CSP , as shown in formula (5).

[0146]

[0147] DW_CSP=f DW_CSP (D) (5),

[0148] in, Represents the feature map concatenation operation.

[0149] Assume that the width, height, and number of channels of the input image D are w0, h0, and c0 respectively. The dimension of the feature map processed by DW-CSPNet changes, and the width, height, and number of channels of the output feature map DW_CSP are w0 / 2, h0 / 2, and 2c0 respectively, which achieves the purpose of downsampling and increasing the depth at the same time.

[0150] Step 3.2: Build the backbone network of DW-YOLOv4

[0151] Five cross-stage local networks based on depthwise separable convolutions are connected in series. The number of channels of the first cross-stage local network based on depthwise separable convolutions is C. out =32; the number of channels C of the second cross-stage local network based on depthwise separable convolution out =64; the number of channels C of the third cross-stage local network based on depthwise separable convolution out =128, the output is recorded as the feature map DW_CSP1, as shown in formula (6); the number of channels C of the fourth cross-stage local network based on depthwise separable convolution out=258, the output is recorded as the feature map DW_CSP2, as shown in formula (7); the number of channels C of the fifth cross-stage local network based on depthwise separable convolution out =512, and the output is recorded as feature map DW_CSP3, as shown in formula (8). Feature maps DW_CSP1, DW_CSP2, and DW_CSP3 will be used as input feature maps and participate in the calculation of the fourth step fusion feature extraction module.

[0152] DW_CSP1=f DW_CSP (f DW_CSP (f DW_CSP (D))) (6),

[0153] DW-CSP2=f DW_CSP (DW_CSP1) (7),

[0154] DW_CSP3=f DW_CSP (DW_CSP2) (8),

[0155] Taking the input image D with a dimension of 608×608×3 as an example, the feature maps DW_CSP1, DW_CSP2, and DW_CSP3 output by the backbone network of the DW-YOLOv4 framework have dimensions of 76×76×128, 38×38×256, and 19×19×512, respectively.

[0156] The fourth step is to build the fusion feature extraction module of DW-YOLOv4

[0157] Step 4.1: Take DW_CSP3 as input and use 3×3 convolution to perform channel compression to obtain the feature map D 4.1 , the dimension is 19×19×256, as shown in formula (9).

[0158] D 4.1 =γ3(DW_CSP3) (9),

[0159] Among them, γ3 indicates that the convolution kernel size is 3×3, including the convolution operation of normalization and LeakyReLU activation function.

[0160] Step 4.2, for D 4.1 Perform maximum pooling with step sizes of 5, 9, and 13, and compare the obtained results with D 4.1 Perform channel splicing to obtain feature map D 4.2 , the dimension is 19×19×1024, as shown in formula (10).

[0161]

[0162] Among them, τ5, τ9, τ13 They represent maximum pooling operations with step sizes of 5, 9, and 13 respectively.

[0163] Step 4.3: Use a 1×1 convolution kernel to adjust D 4.2 Perform channel compression to obtain feature map D 4.3 , the dimension is 19×19×512, as shown in formula (11).

[0164] D 4.3 =γ1(D 4.2 ) (11),

[0165] Step 4.4, use 1×1 convolution on D 4.3 After channel compression, upsampling is performed, channel splicing is performed with DW_CSP2, and then channel compression is performed using a 1×1 convolution kernel to obtain the feature map D 4.4 , the dimension is 38×38×256, as shown in formula (12).

[0166]

[0167] Among them, σ is the upsampling operation.

[0168] Step 4.5: Use 1×1 convolution on D 4.4 After channel compression, upsampling is performed, channel splicing is performed with DW_CSP1, and then channel compression is performed using a 3×3 convolution kernel to obtain the feature map D 4.5 , the dimension is 76×76×128, as shown in formula (13).

[0169]

[0170] Step 4.6: Use a convolution kernel of size 3×3 and stride 2 to calibrate D 4.5 Downsampling and channel expansion, with D 4.4 After splicing, a 3×3 convolution kernel is used to perform channel compression to obtain the feature map D 4.6 , the dimension is 38×38×256, as shown in formula (14).

[0171]

[0172] Step 4.7: Use a convolution kernel of size 3×3 and stride 2 to adjust D 4.6 Downsampling and channel expansion, with D 4.3 After splicing, a 3×3 convolution kernel is used to perform channel compression to obtain the feature map D 4.7 , the dimension is 19×19×512, as shown in formula (15).

[0173]

[0174] Step 4.8: Use three 1×1 convolutions to perform 4.5 , D 4.6 , D 4.7 Adjust the output dimension to form the final feature map D 4.8.1 , D 4.8.2 , D 4.8.3 , the dimensions are 76×76×Out, 38×38×Out, and 19×19×Out, as shown in formula (16).

[0175] D 4.8.1 =ω1(D 4.5 ), D 4.8.2 =ω1(D 4.6 ), D 4.8.2 =ω1(D 4.7 ) (16),

[0176] Among them, convolution ω1 is a convolution operation with a convolution kernel size of 1×1 without normalization and LeakyReLU activation function.

[0177] Here Out = 24;

[0178] This completes the construction of the DW-YOLOv4 fusion feature extraction module.

[0179] In the fifth step, the loss function combining CIOU and exponentially enhanced binary cross entropy is used to calculate the loss between the predicted value output in step 4.8 and the actual value. The prediction results of three types of targets in the current image, namely, powdery mildew, leaf with disease, and leaf without disease, are predicted, including location information (target box center coordinates, width, height), category, and confidence:

[0180] Step 5.1: Calculate CIOU. The center coordinates, width, and height of the target box are represented by (x, y), w, and h, respectively. The real box and predicted box are represented by subscripts gt and pr, respectively. That is, the real box can be represented by B gt =(x gt ,y gt , w gt ,h gt ) indicates that the prediction box can be expressed as B pr =(x pr ,y pr , w pr ,h pr ) is expressed as follows:

[0181]

[0182]

[0183]

[0184]

[0185] Among them, B gt represents the real box, B pr represents the prediction box, B gt ∩B pr Represents the intersection of the real box and the predicted box, B gt ∪B pr represents the union of the real box and the predicted box, and ρ represents the calculation (x gt ,y gt ) and (x pr ,y pr ), c represents the Euclidean distance between B gt and B pr The diagonal distance of the closure, IoU is the intersection-over-Union ratio, which is the ratio of the intersection area to the union area between two boxes, αv is the influencing factor, v is the parameter used to measure the consistency of the aspect ratio, and α is the parameter used for trade-off. is the loss function for the target detection task.

[0186] Step 5.2: Calculate the classification loss and use exponentially enhanced binary cross entropy to represent the loss between the predicted value and the actual value of the category. The formula is as follows:

[0187]

[0188] Among them, cls kj is the actual value of the category, is the category prediction value, S is the number of cells divided by the feature map output in step 4.8, k is the cell number, B is the prediction box of each category, j is the prediction box number, and η is the enhancement index.

[0189] Step 5.3: Calculate confidence loss using binary cross entropy The formula is as follows:

[0190]

[0191] Among them, conf kj is the actual value of the category, is the category prediction value, S is the number of divided cells, k is the cell number, B is the prediction box of each category, and j is the prediction box number.

[0192] Step 5.4: Calculate the total loss The formula is as follows:

[0193]

[0194] In the sixth step, non-maximum suppression (NMS) is performed on the model prediction results after loss calculation, and the results are adjusted.

[0195] In step 6.1, NMS is used to process the model prediction results after loss calculation to obtain the prediction results that do not exceed the NMS threshold for the three types of targets: powdery mildew, leaf with disease, and leaf without disease, including location information (target box center point coordinates, width, height), category, and confidence.

[0196] In step 6.2, the prediction results after non-maximum suppression are adjusted.

[0197] Since the disease must occur on the leaves, the prediction results obtained in step 6.1 do not necessarily identify the diseased leaves. Therefore, when the disease is identified in the image but the diseased leaves are not identified, the other leaf categories with the largest IoU with the disease are adjusted to diseased leaves. If there is no leaf with an IoU greater than 0 with the identified disease, the disease point is removed to ensure that all detected diseases have corresponding diseased leaves.

[0198] The backbone network module and the fusion feature extraction module, loss calculation and non-maximum suppression and adjustment together constitute the DW-YOLOv4 network.

[0199] Step 7: Calculate the disease index

[0200] Step 7.1: Calculate the incidence rate of the fusion adjustment coefficient, that is, the proportion of the diseased area of ​​the i-th diseased leaf to the leaf area a i , as the leaf incidence rate. Since multiple diseases may be detected on the same leaf and the diseases may intersect, it is necessary to calculate the union of the disease detection areas of all diseases on the diseased leaf as the diseased area. However, since the diseased leaf identified by the DW-YOLOv4 network of the present invention is a rectangular frame, but there is a gap between the actual leaf and the rectangular frame, the superposition coefficient is required to adjust the leaf area. The specific incidence calculation formula is as follows.

[0201]

[0202] Among them, a i represents the incidence of the i-th diseased leaf, A represents the area of ​​the disease, j represents the j-th disease point on the i-th diseased leaf, n represents the number of disease points on the i-th diseased leaf, Bi represents the area of ​​the i-th diseased leaf, and σ represents the leaf area adjustment coefficient (preferably 0.6-0.7).

[0203] The leaf area adjustment coefficient is calculated by using binarization and OTSU to extract the number of green pixels from the marked diseased leaf part and dividing it by the number of pixels in the diseased leaf marking box, which is 0.6439 in this case.

[0204] Step 7.2, calculate the disease index of a single leaf, and divide the disease into 1-9 levels. When the disease incidence is less than 5%, the disease level is 1; when the disease incidence is between 5%-25%, the disease level is 3; when the disease incidence is between 25%-50%, the disease level is 5; when the disease incidence is between 50%-75%, the disease level is 7; when the disease incidence is greater than 75%, the disease level is 9. The specific formula is as follows.

[0205]

[0206] Here, i represents the i-th diseased leaf.

[0207] Step 7.3: Calculate the overall disease index

[0208]

[0209] Where N is the number of diseased leaves, and i is the i-th diseased leaf.

[0210] For the index given for each inspection, if a disease is found, a certain number of plants will be sampled near the sampling point to calculate the disease index, and the result of the calculation represents the current disease severity of the greenhouse. Each leaf can only calculate 1 disease index based on the diseased area, with a maximum of 9 and a minimum of 1. The numerator is the sum of the disease indexes of all leaves, and the denominator is the number of diseased leaves multiplied by 9. The closer the disease index of this inspection in the greenhouse is to 100%, the more serious the disease. According to the different degrees of disease, corresponding expert treatment opinions will be given through empirical data, and different disease indexes correspond to corresponding expert treatment opinions. For example, different disease indexes appear at different stages of strawberry growth, and a certain drug is sprayed at what concentration, etc.

[0211] So far, the rapid detection and quantitative disease assessment of strawberry powdery mildew based on DW-YOLOv4 in the greenhouse production environment have been completed.

[0212] The NMS in the above-mentioned strawberry powdery mildew rapid detection method based on DW-YOLOv4 is called Non-Maximum Suppression in Chinese, which is a well-known algorithm in the technical field. YOLO v4, which is called YouLook Only Once v4 in full, is a well-known algorithm in the technical field.

[0213] A strawberry powdery mildew rapid detection system includes a memory and a processor, wherein the memory stores the DW-YOLOv4 network and disease index calculation, and the program on the memory is used to execute the above detection method and disease index calculation when loaded by the processor. The strawberry powdery mildew rapid detection system can be integrated into smart terminals such as mobile phones, iPads, notebooks, etc., and then farmers take photos of leaves, the system identifies the disease level, and gives treatment advice.

[0214] Example 1

[0215] This embodiment is based on the strawberry powdery mildew rapid detection method of DW-YOLOv4, and the specific steps are as follows:

[0216] The first step is to collect image data of strawberry leaf diseases in a greenhouse production environment and create a dataset:

[0217] Step 1.1: In the greenhouse strawberry production environment (including different weather conditions, light intensity, etc.), 30 sampling points were set up, 3 plants were sampled from each sampling point every week, and three leaves were sampled from each plant to observe the incidence of powdery mildew on strawberry leaves;

[0218] Step 1.2: Once the disease is found, all leaves of each plant under observation are identified and photographed. All images are obtained by manual observation by professionals.

[0219] Step 1.3: Use labelimg tool to label the image according to the disease (powdery mildew), diseased leaves (leaf with disease), and other leaves (leaf without disease). For leaves with severe blur in the image, if the edge area cannot be identified by naked eye, it will not be marked;

[0220] Step 1.4, generate data set;

[0221] The second step is image preprocessing:

[0222] In step 2.1, the images in the dataset obtained in step 1.4 above are used as input. They are first processed by data enhancement methods such as brightness, contrast, hue, saturation, noise, random scaling, cropping, flipping, and rotation, and then spliced ​​using the mosaic method to form an enhanced image of 416×416×3.

[0223] In step 2.2, the enhanced image in step 2.1 is randomly scaled to 10-19 times of 32 for multi-scale training. The following text uses the image scale of 608×608×3 as an input example for explanation.

[0224] In the third step, the backbone network module of DW-YOLOv4 is constructed using the Deep Separable Convolution-based Cross-Stage Local Network (DW-CSPNet).

[0225] Step 3.1: Construct a cross-stage local network module based on deep separable convolution (DW-CSPNet)

[0226] In step 3.1.1, the output image D of the previous step is used as input, and the convolution kernel size is 3×3, the step length is 2, and the number of output channels is C. out The downsampling layer performs image processing to form a feature map D1, as shown in formula (1).

[0227]

[0228] in, It indicates a convolution operation with a kernel size of 3×3, a stride of 2, normalization, and a LeakyReLU activation function.

[0229] In step 3.1.2, the processed results are divided into two branches. One branch uses a convolution layer with a convolution kernel size of 1×1 to compress the channel to form a feature map D2, as shown in formula (2).

[0230] D2=γ1(D1) (2),

[0231] Among them, γ1 represents the convolution operation with a convolution kernel size of 1×1, normalization and LeakyReLU activation function.

[0232] In step 3.1.3, the other branch passes through a convolution layer with a convolution kernel size of 1×1 to compress the channel and form feature map D3. The feature of D3 is extracted using a depthwise separable convolution with a convolution kernel size of 3×3 and a convolution layer with a convolution kernel size of 1×1, and the feature map D4 is obtained by summing it with D3, as shown in formula (3).

[0233] D3=γ1(D1), D4=γ1(δ3(D3))+D3 (3),

[0234] Among them, δ3 represents a 3×3 depth-wise separable convolution operation with normalization and LeakyReLU activation function; + represents the feature map summation operation.

[0235] In step 3.1.4, D2 and D4 are concatenated and then compressed using a convolutional layer with a convolution kernel size of 1×1 to form the output feature map DW_CSP of the cross-stage local network module based on depthwise separable convolution, as shown in formula (4).

[0236] The cross-stage local network module based on depthwise separable convolution is denoted as f DW_CSP, as shown in formula (5).

[0237]

[0238] DW_CSP=f DW_CSP (D) (5),

[0239] in, Represents the feature map concatenation operation.

[0240] Assume that the width, height, and number of channels of the input image D are w0, h0, and c0 respectively. The dimension of the feature map processed by DW-CSPNet changes, and the width, height, and number of channels of the output feature map DW_CSP are w0 / 2, h0 / 2, and 2c0 respectively, which achieves the purpose of downsampling and increasing the depth at the same time.

[0241] Step 3.2: Build the backbone network of DW-YOLOv4

[0242] Five cross-stage local networks based on depthwise separable convolutions are connected in series. The first cross-stage local network based on depthwise separable convolutions is C out =32; C of the second cross-stage local network based on depthwise separable convolution out =64; The third C of the cross-stage local network based on depthwise separable convolution out =128, the output is recorded as feature map DW_CSP1, as shown in formula (6); the fourth C of the cross-stage local network based on deep separable convolution out =258, the output is recorded as the feature map DW_CSP2, as shown in formula (7); the fifth C of the cross-stage local network based on deep separable convolution out =512, and the output is recorded as feature map DW_CSP3, as shown in formula (8). Feature maps DW_CSP1, DW_CSP2, and DW_CSP3 will be used as input feature maps and participate in the calculation of the fourth step fusion feature extraction module.

[0243] DW_CSP1=f DW_CSP (f DW_CSP (f DW_CSP (D))) (6),

[0244] DW-CSP2=f DW_CSP (DW_CSP1) (7),

[0245] DW_CSP3=f DW_CSP (DW_CSP2) (8),

[0246] Taking the input image D with a dimension of 608×608×3 as an example, the feature maps DW_CSP1, DW_CSP2, and DW_CSP3 output by the backbone network of the DW-YOLOv4 framework have dimensions of 76×76×128, 38×38×256, and 19×19×512, respectively.

[0247] The fourth step is to build the fusion feature extraction module of DW-YOLOv4

[0248] Step 4.1: Take DW_CSP3 as input and use 3×3 convolution to perform channel compression to obtain the feature map D 4.1 , the dimension is 19×19×256, as shown in formula (9).

[0249] D 4.1 =γ3(DW_CSP3) (9),

[0250] Among them, γ3 represents the convolution operation with a convolution kernel size of 3×3.

[0251] Step 4.2, for D 4.1 Perform maximum pooling with step sizes of 5, 9, and 13, and compare the obtained results with D 4.1 Splice and get the feature map D 4.2 , the dimension is 19×19×1024, as shown in formula (10).

[0252]

[0253] Among them, τ5, τ9, τ 13 They represent maximum pooling operations with step sizes of 5, 9, and 13 respectively.

[0254] Step 4.3: Use a 1×1 convolution kernel to adjust D 4.2 Perform channel compression to obtain feature map D 4.3 , the dimension is 19×19×512, as shown in formula (11).

[0255] D 4.3 =γ1(D 4.2 ) (11),

[0256] Step 4.4, use 1×1 convolution on D 4.3 After channel compression, upsampling is performed, channel splicing is performed with DW_CSP2, and then channel compression is performed using a 1×1 convolution kernel to obtain the feature map D 4.4 , the dimension is 38×38×256, as shown in formula (12).

[0257]

[0258] Among them, σ is the upsampling operation.

[0259] Step 4.5: Use 1×1 convolution on D 4.4 After channel compression, upsampling is performed, channel splicing is performed with DW_CSP1, and then channel compression is performed using a 3×3 convolution kernel to obtain the feature map D 4.5 , the dimension is 76×76×128, as shown in formula (13).

[0260]

[0261] Step 4.6: Use a convolution kernel of size 3×3 and stride 2 to calibrate D 4.5 Downsampling and channel expansion, with D 4.4 After splicing, a 3×3 convolution kernel is used to perform channel compression to obtain the feature map D 4.6 , the dimension is 38×38×256, as shown in formula (14).

[0262]

[0263] Step 4.7: Use a convolution kernel of size 3×3 and stride 2 to adjust D 4.6 Downsampling and channel expansion, with D 4.3 After splicing, a 3×3 convolution kernel is used to perform channel compression to obtain the feature map D 4.7 , the dimension is 19×19×512, as shown in formula (15).

[0264]

[0265] Step 4.8, use 1×1 convolution on D 4.5 , D 4.6 , D 4.7 Adjust the output dimension to form the final feature map D 4.8.1 , D 4.8.2 , D 4.8.3 , the dimensions are 76×76×Out, 38×38×Out, and 19×19×Out, as shown in formula (16).

[0266] D 4.8.1 =ω1(D 4.5 ), D 4.8.2 =ω1(D 4.6 ), D 4.8.2 =ω1(D 4.7 ) (16),

[0267] Here, the convolution ω1 is a convolution operation with a convolution kernel size of 1×1 without normalization and LeakyReLU activation function. Here, Out=24.

[0268] The fifth step is to calculate the loss between the predicted value output in step 4.8 and the actual value using the loss function that combines CIOU and exponentially enhanced binary cross entropy:

[0269] Step 5.1: Calculate CIOU. The center coordinates, width, and height of the target box are represented by (x, y), w, and h, respectively. The real box and predicted box are represented by subscripts gt and pr, respectively. That is, the real box can be represented by B gt =(x gt ,y gt , w gt ,h gt ) indicates that the prediction box can be expressed as B pr =(x pr ,y pr , w pr ,h pr ) is expressed as follows:

[0270]

[0271]

[0272]

[0273]

[0274] Among them, B gt represents the real box, B pr represents the prediction box, B gt ∩B pr Represents the intersection of the real box and the predicted box, B gt ∪B pr represents the union of the real box and the predicted box, and ρ represents the calculation (x gt ,y gt ) and (x pr ,y pr ), c represents the Euclidean distance between B gt and B pr The diagonal distance of the closure, IoU is the intersection-over-Union, which is the ratio of the intersection area to the union area between two boxes.

[0275] Step 5.2, calculate the classification loss, and use the exponentially enhanced binary cross entropy to represent the loss between the category prediction value and the actual value. The formula is as follows:

[0276]

[0277] Among them, cls kj is the actual value of the category, is the category prediction value, S is the number of divided cells, k is the cell number, B is the prediction box of each category, j is the prediction box number, and η is the enhancement index.

[0278] In step 5.3, the confidence loss is calculated using binary cross entropy, and the formula is as follows:

[0279]

[0280] Among them, conf kj is the actual value of the category, is the category prediction value, S is the number of divided cells, k is the cell number, B is the prediction box of each category, and j is the prediction box number.

[0281] Step 5.4, calculate the total loss, the formula is as follows:

[0282]

[0283] In the sixth step, non-maximum suppression (NMS) is performed on the model prediction results after loss calculation, and the results are adjusted.

[0284] In step 6.1, NMS is used to process the model prediction results after loss calculation to obtain the prediction results that do not exceed the NMS threshold for the three types of targets: powdery mildew, leaf with disease, and leaf without disease, including location information (target box center point coordinates, width, height), category, and confidence.

[0285] Step 6.2: Adjust the prediction results after non-maximum suppression

[0286] Since the disease must occur on the leaves, but the detection results do not necessarily identify the diseased leaves, when the image identifies the disease but not the diseased leaves, the other leaf categories with the largest IoU (intersection over union) with the disease are adjusted to diseased leaves. If there is no leaf with an diseased point (disease area) whose IoU>0, the diseased point is removed to ensure that all detected diseased points have corresponding diseased leaves.

[0287] Step 7: Calculate the disease index

[0288] Step 7.1: Calculate the incidence rate of the fusion adjustment coefficient, that is, the proportion of the diseased area of ​​the i-th diseased leaf to the leaf area a i, as the leaf incidence rate. Since multiple diseases may be detected on the same leaf and the disease points may intersect, it is necessary to calculate the union of the disease detection areas of all disease points on the diseased leaf as the diseased area. Since the diseased leaf identified by this model is a rectangular frame, but there is a gap between the actual leaf and the rectangular frame, the superposition coefficient is required to adjust the leaf area. The specific incidence calculation formula is as follows.

[0289]

[0290] Among them, a i represents the incidence of the i-th diseased leaf, A represents the area of ​​the disease, j represents the j-th disease point on the i-th diseased leaf, n represents the number of disease points on the i-th diseased leaf, B i represents the area of ​​the i-th diseased leaf, and σ represents the leaf area adjustment coefficient.

[0291] Step 7.2, calculate the disease index of a single leaf, and divide the disease into 1-9 levels. When the disease incidence is less than 5%, the disease level is 1; when the disease incidence is between 5%-25%, the disease level is 3; when the disease incidence is between 25%-50%, the disease level is 5; when the disease incidence is between 50%-75%, the disease level is 5; when the disease incidence is greater than 75%, the disease level is 9. The specific formula is as follows.

[0292]

[0293] Here, i represents the i-th diseased leaf.

[0294] Step 7.3: Calculate the overall disease index

[0295]

[0296] Where N is the number of diseased leaves, and i is the i-th diseased leaf.

[0297] So far, the rapid detection of strawberry powdery mildew based on DW-YOLOv4 in the greenhouse production environment has been completed.

[0298] Figure 5 , Figure 6 It is a detection effect diagram of a strawberry leaf disease image in a production environment after being detected using the strawberry powdery mildew rapid detection method based on DW-YOLOv4 of the present invention.

[0299] The invention detects strawberry powdery mildew in a greenhouse production environment, designs a cross-stage local network based on a deep separable structure to construct a YOLOv4 backbone network, designs a fusion feature extraction module, obtains DW-YOLOv4, and can realize strawberry powdery mildew detection; then, an exponentially enhanced binary cross entropy is introduced to improve the loss function of the algorithm, and the powdery mildew disease index is evaluated, which is helpful for the discovery and management of strawberry powdery mildew.

[0300] Anything not described in the present invention is applicable to the prior art.

Claims

1. A method for rapid detection of strawberry powdery mildew in a greenhouse production environment, the method comprising the following contents: Collect images of the back of leaves infected with powdery mildew under different lighting conditions in a greenhouse strawberry production environment. The image background is the strawberry growth environment. Label the back images according to disease, infected leaves, and other leaves to create a data set. Build a DW-YOLOv4 network, which includes a backbone network module, a fusion feature extraction module, and a loss calculation and non-maximum suppression and adjustment module: Backbone network module: five groups of cross-stage local network modules DW-CSPNet based on deep separable convolution are connected in series. The process of DW-CSPNet is: use a convolution kernel size of 3×3, a step size of 2, and an output channel number of C out The convolution layer processes the input feature map, and then it is divided into two branches. The first branch uses a convolution layer with a convolution kernel size of 1×1 to compress the channel; the second branch uses a convolution layer with a convolution kernel size of 1×1 to compress the channel, uses a depth-separable convolution with a convolution kernel size of 3×3 to extract features, and uses a convolution layer with a convolution kernel size of 1×1 to expand the number of channels. The output after the number of channels is expanded is added to the result of the second branch compressing the channel; then the result of the second branch is channel-joined with the result of the first branch, and a convolution layer with a convolution kernel size of 1×1 is used to compress the channel to obtain the output feature map DW_CSP; Fusion feature extraction module: Using the three output feature maps of the 3rd, 4th, and 5th cross-stage local network DW-CSPNet based on the deep separable convolution structure in the backbone network module as input, the fusion feature extraction module includes maximum pooling, channel splicing, convolution layer with a convolution kernel size of 1×1, upsampling, and downsampling, and finally forms three output feature maps of different scales; The loss function combining CIOU and exponentially enhanced binary cross entropy is used to calculate the loss between the predicted value and the actual value, and non-maximum suppression (NMS) is performed on the model prediction results after the loss calculation, and the results are adjusted; The collected images are input into the DW-YOLOv4 network to detect the powdery mildew diseased areas and diseased leaves of strawberry leaves.

2. The method for rapid detection of strawberry powdery mildew under greenhouse production environment according to claim 1, characterized in that: The disease index was obtained based on the ratio of the powdery mildew disease area to the diseased leaf area.

3. The method for rapid detection of strawberry powdery mildew under greenhouse production environment according to claim 2, characterized in that: The specific calculation process of the disease index is: The incidence rate of the fusion adjustment coefficient is calculated using formula (24), Among them, a i represents the incidence of the i-th diseased leaf, A represents the area of ​​the disease, j represents the j-th disease point on the i-th diseased leaf, n represents the number of disease points on the i-th diseased leaf, B i represents the area of ​​the i-th diseased leaf, σ represents the leaf area adjustment coefficient; The leaf area adjustment coefficient was calculated by using binarization and OTSU to extract the number of green pixels from the marked diseased leaf part and dividing it by the number of pixels in the marked box of the diseased leaf; Calculate disease index for individual leaves The disease is divided into levels 1-9. When the incidence rate is less than 5%, the disease level is 1; when the incidence rate is between 5%-25%, the disease level is 3; when the incidence rate is between 25%-50%, the disease level is 5; when the incidence rate is between 50%-75%, the disease level is 7; when the incidence rate is greater than 75%, the disease level is 9. The specific formula is as follows: Where i represents the i-th diseased leaf; The overall disease index is calculated according to formula (26): Among them, N is the number of diseased leaves, and i is the i-th diseased leaf; According to the different severity of the disease, corresponding expert treatment opinions will be given based on empirical data, and different disease indexes correspond to corresponding expert treatment opinions.

4. The method for rapid detection of strawberry powdery mildew under greenhouse production environment according to claim 2, characterized in that: The blade area adjustment coefficient is 0.6-0.

7.

5. The method for rapid detection of strawberry powdery mildew under greenhouse production environment according to claim 1, characterized in that: The process of the backbone network module is: The input feature map is used, and three cross-stage local networks based on depthwise separable convolution are used to downsample and expand the number of channels to obtain the output feature map DW_CSP1. The output feature map DW_CSP1 is directly output on the one hand, and on the other hand, it is output to the fourth cross-stage local network based on depthwise separable convolution. The number of channels is downsampled and expanded to obtain the output feature map DW_CSP2; the output feature map DW_CSP2 is directly output on the one hand, and on the other hand, it is output to the fifth cross-stage local network based on depthwise separable convolution. The number of channels is downsampled and expanded to obtain the output feature map DW_CSP3. Therefore, the backbone network module has three outputs, namely DW_CSP1, DW_CSP2 and DW_CSP3.

6. The method for rapid detection of strawberry powdery mildew under greenhouse production environment according to claim 1, characterized in that: The fusion feature extraction module is used to fuse the three outputs of the backbone network module. The specific process is: The last output feature map of the backbone network module is compressed using a convolution layer with a convolution kernel size of 3×3, and then three maximum pooling processes with step sizes of 5, 9, and 13 are performed respectively. The three maximum pooling layers are in parallel, that is, the inputs of the three are the same. Then, the results of the three maximum pooling processes are concatenated with the results of the 3×3 convolution layer compression channel in the previous step, and the convolution layer compression channel with a convolution kernel size of 1×1 is used again to obtain the feature map D. 4.3 , feature map D 4.3 On the one hand, after upsampling, the second output feature map of the backbone network module is channel-joined, and then the feature map D is obtained by a convolution layer with a convolution kernel size of 1×1. 4.4 , feature map D 4.4 On the one hand, the convolution layer with a convolution kernel size of 1×1 compresses the channel and then performs upsampling, and then performs channel splicing with the first output feature map of the backbone network module, and then uses the convolution layer with a convolution kernel size of 3×3 to compress the channel to obtain the feature map D 4.5 ; Feature map D 4.5 On the one hand, we use a convolutional layer with a convolution kernel size of 3×3 and a stride of 2 to downsample and expand the channel before adding it to the feature map D 4.4 The channel splicing is performed, and the result of the splicing of the two channels is compressed using a convolution layer with a convolution kernel size of 3×3 to obtain the feature map D 4.6 , feature map D 4.6 On the one hand, we use a convolutional layer with a convolution kernel size of 3×3 and a stride of 2 to downsample and expand the channel before adding it to the feature map D 4.3 Perform channel splicing. The result of channel splicing uses a convolution layer with a convolution kernel size of 3×3 to compress the channel to obtain the feature map D 4.7 , feature map D 4.7 The output feature map D is obtained by adjusting the channel through a convolution layer with a convolution kernel size of 1×1. 4.8.3 ; Feature map D 4.6 On the other hand, the convolution layer with a convolution kernel size of 1×1 is also used to adjust the channel to obtain the output feature map D 4.8.2 , feature map D 4.5 On the other hand, the convolution layer with a convolution kernel size of 1×1 is also used to adjust the channel to obtain the output feature map D 4.8.1 , feature map D 4.8.3 , feature map D 4.8.2 , feature map D 4.8.1 are the three outputs of the fusion feature extraction module.

7. A strawberry powdery mildew rapid detection system, comprising a memory and a processor, wherein the memory stores the DW-YOLOv4 network and disease index calculation described in any one of claims 1 to 6, and the program on the memory is used to execute the above-mentioned detection method when loaded by the processor.

8. The strawberry powdery mildew rapid detection system according to claim 7, characterized in that: The detection system is integrated into smart terminals, such as mobile phones, iPads or laptops. Farmers take photos of leaves, and the system identifies the severity of the disease and gives treatment advice.

Images