Crop leaf area index estimation method based on depth image semantic segmentation

Abstract

The present application provides a crop leaf area index estimation method based on depth image semantic segmentation. The method comprises a crop high-quality image data acquisition system and a leaf area index measurement framework design based on image segmentation. The purpose is to stably and efficiently collect crop image data without high-quality crop images and develop a crop leaf area index (LAI) measurement model based on image semantic segmentation method. Thus, an image semantic segmentation based crop leaf area index measurement algorithm is constructed. The method verifies the high precision and robustness of the image segmentation method based on deep learning in the measurement of crop leaf area index.

Description

Technical Field

[0001] The present invention relates to the technical field of crop leaf area index estimation, and in particular to a method for estimating the leaf area index of broad-leaved non-overlapping crops, specifically a crop leaf area index estimation method based on depth image semantic segmentation. Background Art

[0002] Remote sensing and non-destructive estimation of leaf area index (LAI) has been a challenge over the past few decades. Existing direct and indirect methods are laborious and time-consuming, and difficult to scale up and apply to large-scale modern unmanned farms. Current image processing-based LAI measurements are based on traditional image segmentation models or algorithms, which have low accuracy and large errors. There is a lack of high-precision end-to-end model methods based on deep learning. This paper proposes a semantic segmentation framework based on end-to-end deep learning to achieve high-precision crop image leaf area index measurement for large-scale and rapid estimation of crop LAI. The end-to-end, simple and efficient characteristics of this method enable it to be rapidly scaled up and applied to large-scale modern unmanned farms, helping to improve the level of agricultural intelligence and automation. Summary of the Invention

[0003] The purpose of the present invention is to solve the problems in the prior art and propose a crop leaf area index estimation method based on deep image semantic segmentation. The method includes two parts: a high-quality crop image data acquisition system and a leaf area index measurement framework design based on image segmentation. The purpose is to be able to stably and efficiently acquire crop image data and develop a crop leaf area index (LAI) measurement model based on image semantic segmentation methods in the absence of high-quality crop images. In this way, a crop leaf area index measurement algorithm based on image semantic segmentation is constructed. The present invention uses cucumber crops to implement crop image acquisition, framework verification, model training, and other tasks.

[0004] The present invention is achieved through the following technical solutions. The present invention proposes a crop leaf area index estimation method based on deep image semantic segmentation, the method comprising:

[0005] First, high-quality crop image data is collected according to the crop high-quality image data acquisition system. After sampling, the semantic areas of crop pixels are manually labeled and an image semantic segmentation dataset is produced. Based on the produced dataset, an end-to-end crop leaf area index measurement deep learning model framework, namely the LAI measurement framework, is designed. The LAI measurement framework consists of four parts, namely the backbone network Backbone, segmentation map Segmentation Map, mask prediction MaskPrediction and leaf area index prediction LAI Prediction; the backbone network part is used to complete the segmentation of the crop image, generate a feature map, use the generated feature map to predict the mask, generate the segmentation result, and calculate the crop LAI prediction result based on the predicted mask; wherein, in the training stage, the generated segmentation result is compared with the true value, namely Loss Seg, and the predicted LAI is compared with the LAI in the image, namely Loss LAI; in the inference stage, the mask prediction result and leaf area measurement result are directly generated to realize crop LAI measurement and segmentation.

[0006] Furthermore, the crop high-quality image data acquisition system consists of five parts: a crop growth environment, lighting, image acquisition, data transmission and data storage server; the crop growth environment provides the soil, water and nutrients required for crop growth; the lighting part provides the light required for crops to carry out photosynthesis, and at the same time provides lighting conditions for high-definition crop image acquisition; the image acquisition part is used to collect video data streams of the crop growth process in real time; the data transmission part stores the data streams collected by the camera on a disk through a network video recorder (NVR); the data storage server is used to read the camera's video stream from the NVR in real time, and at the same time uses relevant codes and scripts to realize automated image data sampling and storage at equal intervals.

[0007] Furthermore, the leaf area index is defined as the area of ​​all leaves per unit area of ​​land, and its formula is:

[0008]

[0009] Furthermore, based on the camera imaging principle, the formula for calculating leaf area through image is obtained:

[0010]

[0011] in Indicates the actual area of ​​crops, represents the actual area of ​​the calibration object, and represents the area of ​​the calibration object in the image, Indicates the area of ​​the crop in the image.

[0012] Furthermore, the loss function used in model training is shown in the formula:

[0013]

[0014] in is the MAE loss function, is the cross entropy loss function.

[0015] Furthermore, the image acquisition part consists of two high-definition cameras. During the data acquisition process, the camera automatically captures an image every 5 minutes and stores the captured image in a disk.

[0016] Furthermore, the estimation method also includes an evaluation part, and the evaluation indicators adopted by the evaluation part include image semantic segmentation accuracy evaluation indicators and regression evaluation indicators.

[0017] Furthermore, the image semantic segmentation accuracy evaluation index is the mean intersection over union (mIoU), which refers to the ratio of the intersection and union of the predicted value and the true value; the regression evaluation indicators include mean absolute error (MAE), mean square error (MSE), root mean square error (RMSE) and R 2 Regression score function is a regression model evaluation indicator.

[0018] The present invention proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the crop leaf area index estimation method based on depth image semantic segmentation when executing the computer program.

[0019] The present invention proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the crop leaf area index estimation method based on depth image semantic segmentation.

[0020] Beneficial effects of the present invention:

[0021] The present invention designs a crop leaf area index estimation method based on deep image semantic segmentation, which includes a crop high-quality image acquisition system and an end-to-end framework for crop leaf area index measurement based on image segmentation method; mIoU and MAE MSE RMSE, R 2 Indicators such as image segmentation and crop leaf area measurement are used to comprehensively reflect the image segmentation effect and the accuracy of crop leaf area measurement.

[0022] Experimental test results show that the deep learning-based image segmentation method has good accuracy and robustness, demonstrating the advantages and effectiveness of deep learning-based image segmentation models in image segmentation and verifying the high accuracy and robustness of deep learning-based image segmentation methods in measuring crop leaf area index. At the same time, the high-quality crop image acquisition system designed by this invention also provides model developers with a highly automated data acquisition solution that can provide high-quality image data, improve the efficiency of high-quality crop image data acquisition and model development, and promote the research and application of related AI models in agriculture. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0024] Figure 1 Block diagram of the crop high-quality image data acquisition system;

[0025] Figure 2 Schematic diagram of crop image data annotation method;

[0026] Figure 3 Schematic diagram of crop leaf area measurement method based on graph paper;

[0027] Figure 4 Design a schematic diagram for the leaf area index measurement framework based on deep image semantic segmentation;

[0028] Figure 5 Schematic diagram of the experimental results of leaf area estimation based on deep image semantic segmentation;

[0029] Figure 6 Schematic diagram of the visualization of the segmentation effect based on the deep image semantic segmentation model, where (a) is the semantic segmentation experiment result of the crop leaf image under graffiti (leaf area estimation), and (b) is the semantic segmentation experiment result of the original crop leaf image;

[0030] Figure 7 Schematic diagram of large-scale farmland crop leaf area index measurement based on deep image semantic segmentation;

[0031] The numbers in the figure are: 701 - camera; 702 - embedded leaf area index calculation platform; 703 - UAV rotor. DETAILED DESCRIPTION

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0033] The present invention proposes a crop leaf area index estimation method based on depth image semantic segmentation, the method comprising:

[0034] First, high-quality crop image data is collected according to the crop high-quality image data acquisition system. After sampling, the semantic areas of crop pixels are manually labeled and an image semantic segmentation dataset is produced. Based on the produced dataset, an end-to-end crop leaf area index measurement deep learning model framework, namely the LAI measurement framework, is designed. The LAI measurement framework consists of four parts, namely the backbone network Backbone, segmentation map Segmentation Map, mask prediction MaskPrediction and leaf area index prediction LAI Prediction; the backbone network part is used to complete the segmentation of the crop image, generate a feature map, use the generated feature map to predict the mask, generate the segmentation result, and calculate the crop LAI prediction result based on the predicted mask; wherein, in the training stage, the generated segmentation result is compared with the true value, namely Loss Seg, and the predicted LAI is compared with the LAI in the image, namely Loss LAI; in the inference stage, the mask prediction result and leaf area measurement result are directly generated to realize crop LAI measurement and segmentation.

[0035] This paper designs a high-quality crop image data automatic acquisition system platform. The main purpose of this system design is to automatically acquire high-quality data. The system architecture is as follows: Figure 1Specifically, the crop high-quality image data acquisition system consists of five parts: a crop growth environment, lighting, image acquisition, data transmission, and a data storage server. The crop growth environment provides the soil, water, nutrients, and other substances required for crop growth. The lighting part provides the light required for photosynthesis for crops and provides good lighting conditions for high-definition crop image acquisition. The image acquisition part consists of two high-definition cameras, which are used to collect video data streams of the crop growth process in real time. During the data acquisition process, the camera automatically captures an image every 5 minutes and stores the captured image on disk. The data transmission part stores the data stream captured by the camera on disk via a network video recorder (NVR). The data storage server is used to read the camera video stream from the NVR in real time and uses relevant codes and scripts to realize automatic image data sampling and storage at equal intervals.

[0036] This paper designs an end-to-end crop leaf area index measurement deep learning model framework, the specific implementation block diagram is as follows Figure 4 As shown in the figure, the purpose of designing this framework is to achieve end-to-end crop leaf area measurement. Since the actual land area is known, the core of calculating the leaf area index is to calculate the leaf area. Therefore, this framework calculates the leaf area through image semantic segmentation methods to achieve the calculation of the leaf area index. Several classic image segmentation models were selected to evaluate the performance of the designed method, including FCN, Unet, UnetPlusPlus, MAnet, DeepLabV3Plus, and Linknet based on fully convolutional neural networks. The loss function used in all model training is shown in the formula:

[0037]

[0038] in is the MAE loss function, is the cross entropy loss function.

[0039] The leaf area index is defined as the area of ​​all leaves per unit land, and its formula is:

[0040]

[0041] According to the camera imaging principle, the formula for calculating leaf area through image is obtained:

[0042]

[0043] in Indicates the actual area of ​​crops, represents the actual area of ​​the calibration object, and represents the area of ​​the calibration object in the image, Indicates the area of ​​the crop in the image.

[0044] In order to evaluate the measurement performance of the designed model, the present invention selects the mIoU and R of each model in different test images. 2 As evaluation indicators. mIoU and R 2 The larger the better, the smaller the MAE, MSE, RMSE and other indicators are, the better the model prediction performance is. In order to reflect the robustness of the model, the present invention tests each model on multiple images.

[0045] (1) Image semantic segmentation accuracy evaluation index: Mean Intersection over Union (mIoU): The mean intersection over union (mIoU) refers to the ratio of the intersection and union of the predicted value and the true value. The specific calculation method is shown in the following formula:

[0046]

[0047] In the formula, N represents the number of samples, P represents the predicted value, and G represents the true value.

[0048] (2) Regression evaluation indicators: mean absolute error (MAE), mean square error (MSE), root mean square error (RMSE), R 2 Regression model evaluation indicators such as regression score function. Its calculation formula is as follows:

[0049]

[0050]

[0051]

[0052]

[0053] Where: n represents the number of samples, y i represents the true value, represents y i The average value of Represents the predicted value.

[0054] Example

[0055] The present invention proposes a crop leaf area index estimation method based on depth image semantic segmentation, the method comprising:

[0056] First, according to Figure 1 The crop high-quality image data acquisition system shown in the figure collects high-quality crop image data, samples it, manually annotates the semantic areas of crop pixels, and produces an image semantic segmentation dataset. The training dataset annotation process and method are as follows: Figure 2The leaf area measurement process and method are as shown in Figure 3 As shown. Based on the prepared data set, an end-to-end LAI measurement framework is designed. Figure 4 The leaf area index measurement framework based on image segmentation shown in the figure consists of four parts, namely backbone network (Backbone), segmentation map (Segmentation Map), mask prediction (Mask Prediction), leaf area index prediction (LAI Prediction). The main function of the backbone network is to complete the segmentation of the crop image and generate a feature map. The generated feature map is used to predict the mask and generate the segmentation result. At the same time, the crop LAI prediction result is calculated based on the predicted mask. Among them, in the training stage, the generated segmentation result is compared with the true value, namely Loss Seg, and the predicted LAI is compared with the LAI in the image, namely Loss LAI; in the inference stage, the model will directly generate the mask prediction result and the leaf area measurement result to realize the crop LAI measurement and segmentation. Since only one crop is measured in the leaf area measurement process, in order to verify the feasibility and effect of the proposed framework, it is necessary to graffiti the leaves other than the leaves of a cucumber crop in the image. Specifically, the training and test validation data used are shown in Table 1. The experimental results of leaf area estimation based on deep image semantic segmentation are shown in Figure 1. Figure 5 As shown in the figure, the segmentation effect visualization results of graffiti and original images based on the deep image semantic segmentation model are as follows Figure 6 The leaf area estimation error evaluation of all models is shown in Table 2. The effects of the alpha parameter on segmentation accuracy and leaf area estimation are shown in Tables 3 and 4, respectively.

[0057] Table 1 CLAI dataset

[0058]

[0059] Table 2 Error analysis results of deep learning on the test set

[0060]

[0061]

[0062] Table 3 Impact of alpha on segmentation accuracy mIoU

[0063]

[0064] Table 4 Effect of alpha on leaf area

[0065]

[0066] The method proposed in the present invention can be used on a drone to estimate the leaf area index of large-scale farmland crops. Specifically, the crop images captured by the camera 701 on the drone are input into the embedded leaf area index calculation platform 702. The leaf area index of the entire farmland crop is obtained by inference calculation based on the trained model. The specific schematic diagram is shown in FIG. Figure 7 shown.

[0067] The present invention proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the crop leaf area index estimation method based on depth image semantic segmentation when executing the computer program.

[0068] The present invention proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the crop leaf area index estimation method based on depth image semantic segmentation.

[0069] The memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DRRAM). It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0070] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disc (SSD)).

[0071] During implementation, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor or by instructions in the form of software. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it will not be described in detail here.

[0072] It should be noted that the processor in the embodiments of the present application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiment can be completed by an integrated logic circuit of the hardware in the processor or by instructions in the form of software. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0073] The above is a detailed introduction to the crop leaf area index estimation method based on deep image semantic segmentation proposed in the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A crop leaf area index estimation method based on deep image semantic segmentation, characterized by: The method comprises: First, high-quality crop image data is collected according to the crop high-quality image data acquisition system. After sampling, the semantic areas of crop pixels are manually labeled and produced into an image semantic segmentation dataset. Based on the produced dataset, an end-to-end crop leaf area index measurement deep learning model framework, namely the LAI measurement framework, is designed. The LAI measurement framework consists of four parts, namely the backbone network Backbone, segmentation map Segmentation Map, mask prediction MaskPrediction and leaf area index prediction LAI Prediction; the backbone network part is used to complete the segmentation of the crop image, generate a feature map, use the generated feature map to predict the mask, generate the segmentation result, and calculate the crop LAI prediction result based on the predicted mask; wherein, in the training stage, the generated segmentation result is compared with the true value, namely Loss Seg, and the predicted LAI is compared with the LAI in the image, namely Loss LAI; in the inference stage, the mask prediction result and leaf area measurement result are directly generated to realize crop LAI measurement and segmentation; The loss function used in model training is shown in the formula: in is the MAE loss function, is the cross entropy loss function.

2. The method according to claim 1, wherein: The high-quality crop image data acquisition system consists of five parts: a crop growth environment, lighting, image acquisition, data transmission, and a data storage server. The crop growth environment provides the soil, water, and nutrients required for crop growth. The lighting part provides the light required for crops to photosynthesize, and also provides lighting conditions for high-definition crop image acquisition. The image acquisition part is used to collect video data streams of the crop growth process in real time. The data transmission part stores the data streams collected by the camera on a disk via a network video recorder (NVR). The data storage server is used to read the camera video stream from the NVR in real time, and uses relevant codes and scripts to realize automated image data sampling and storage at equal intervals.

3. The method according to claim 1, wherein: The leaf area index is defined as the area of ​​all leaves per unit land, and its formula is: 。 4. The method according to claim 3, wherein: According to the camera imaging principle, the formula for calculating leaf area through image is obtained: in Indicates the actual area of ​​crops, represents the actual area of ​​the calibration object, and , represents the area of ​​the calibration object in the image, Indicates the area of ​​the crop in the image.

5. The method according to claim 1, wherein: The image acquisition part consists of two high-definition cameras. During the data acquisition process, the camera automatically captures an image every 5 minutes and stores the captured image in the disk.

6. The method according to claim 1, wherein: The estimation method also includes an evaluation part, and the evaluation indicators adopted by the evaluation part include an image semantic segmentation accuracy evaluation indicator and a regression evaluation indicator.

7. The method according to claim 6, characterized in that: The image semantic segmentation accuracy evaluation index is the mean intersection over union (mIoU), which refers to the ratio of the intersection and union of the predicted value and the true value; the regression evaluation index includes the mean absolute error (MAE), the mean square error (MSE), the root mean square error (RMSE), and the Regression score function is a regression model evaluation indicator.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium for storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images