Method and device for extracting character of plant heading period

Abstract

The invention provides a plant heading period character extraction method and device. The plant heading period character extraction method comprises the steps that an unmanned aerial vehicle is adopted to collect a heading image in the plant heading process, and the heading image is preprocessed; constructing a data set of the heading image; carrying out training, verification and testing on the data set by using an Ulteriics library so as to select an optimal heading model; the method has the beneficial effects that the measurement efficiency is improved, the cost is saved, the data accuracy and reliability are improved, and better support and help are provided for rice production.

Description

Technical Field

[0001] The present invention relates to the field of agriculture, in particular to a method and device for extracting traits of the heading stage of plants based on an unmanned aerial vehicle (UAV). Background Art

[0002] In the prior art, the methods for extracting traits of the heading stage usually include: Field observation method: By observing the growth of plants in the field and recording the heading stage of the plants. This method requires regular on-site inspections in the farmland and making relevant records. Photo analysis method: Using a photographing device to take pictures of the growth of plants, and then analyzing the photos through image processing software to calculate the heading stage of the plants. This method can be carried out remotely, but requires clear photos and accurate software analysis. Sensor monitoring method: Using various sensors to monitor the growth of plants, such as soil humidity, temperature, light, etc., and combining these data with the growth law of plants to predict the heading stage of the plants. This method requires the installation and maintenance of sensors, but can monitor the growth of plants in real time. Gene detection method: Using gene detection technology to detect the gene expression of plants, and combining the known relationship between gene expression and the heading stage to infer the heading stage of the plants. This method requires a relatively high technical level and cost.

[0003] However, in the prior art, the methods for extracting traits of the heading stage have the following defects: High consumption of manpower and time: The field observation method requires a large amount of manpower and time for on-site inspections and records. Especially in large-scale farmland, it is difficult to achieve real-time monitoring and accurate records. Great influence by environment and weather: The photo analysis method and the sensor monitoring method are greatly affected by the environment and weather. For example, insufficient light, obstacles, rain and fog, etc. will all affect the accuracy and reliability of the monitoring results. High cost of sensor installation and maintenance: The sensor monitoring method requires the installation and maintenance of various sensors, with a relatively high cost, and requires professional personnel for operation and maintenance. Immaturity of gene detection technology: Although the gene detection method can more accurately predict the heading stage of plants, this technology is not yet mature and popular enough at present, and the cost is relatively high. Difficulty in data processing and analysis: For the photo analysis method and the sensor monitoring method, a large amount of data needs to be processed, and accurate analysis and processing are required, which requires professional technology and equipment support.

[0004] In order to overcome the above-mentioned defects existing in the prior art, there is an urgent need in the art for a high-throughput rice heading stage trait extraction method and device based on an unmanned aerial vehicle. The high-throughput rice heading stage trait extraction method based on an unmanned aerial vehicle can improve the measurement efficiency, save costs, improve the accuracy and reliability of data, and provide better support and help for rice production. Summary of the Invention

[0005] A brief overview of one or more aspects is given below to provide a basic understanding of these aspects. This overview is not an exhaustive survey of all contemplated aspects and is neither intended to identify key or decisive elements of all aspects nor to define the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that follows.

[0006] To overcome the above-mentioned deficiencies in the prior art, the present invention provides a method for extracting traits of the heading stage of a plant, including: collecting heading images during the heading process of the plant using a drone, and preprocessing the heading images; constructing a dataset of the heading images; using the Ultralytics library to train, validate, and test on the dataset to select the optimal heading model; and detecting and extracting the plant heading traits during the complete heading stage of the plant.

[0007] In one embodiment, preferably, the preprocessing of the heading images includes: extracting the heading images of the plants in the central plot; combining the CIVE vegetation index with the Otsu threshold method to obtain a plant mask; summing the pixel values of the heading images of the plants in the central plot, and using a preset window moving average; and obtaining the actual length, width, and pixel values of the central plot according to the actual density of the plant planting, so as to obtain and crop the heading images of the central plot.

[0008] In one embodiment, preferably, the construction of the dataset of the heading images includes: dividing the heading images with a picture pixel of 3800×2000 into small pixel images with a sliding window size and a stride of 1000×1000 pixels by a sliding window method; and semi-automatically annotating the heading images of the small pixel images using video tags and an arbitrary composition model.

[0009] In one embodiment, preferably, the construction of the dataset of the heading images further includes: the arbitrary composition model improves the complete annotation speed of the heading plants based on user prompts; and after a small number of target detection models are annotated, pseudo-annotations are generated using this model, and the better model is continuously iteratively used to accelerate the annotation.

[0010] In one embodiment, preferably, the use of the Ultralytics library to train, validate, and test on the dataset to select the optimal heading model includes: using the Ultralytics library with default parameters to implement the training, validation, and testing of YOLOv8-X on the constructed dataset. When finally selecting the model, the optimal model is selected by comprehensively considering three evaluation indicators of mAP, R 2 and RMSE. Any open-source target detection model can be used to replace YOLOv8.

[0011] In one embodiment, preferably, detecting and extracting the heading traits of the plant during the heading stage of the plant includes: extracting the plot from each original image, and using the sliding window method to cut it into small images. The size of the sliding window is 1000×1000 pixels, and the stride is 750×750 pixels. Use YOLOv8-X to predict each small image, and the confidence threshold and intersection over union (IoU) threshold for non-maximum suppression are 0.3 and 0.5 respectively. Then use non-maximum suppression to merge and remove redundant prediction results, and the IoU threshold is 0.25. Calculate the number of bounding boxes to obtain the number of panicles in the image plot.

[0012] In one embodiment, preferably, using the Ultralytics library to train, validate, and test on the dataset to select the optimal heading model includes: plotting the panicle number-date curve. Define the maximum panicle number as 100%, and according to 80%, 50%, 30%, and 10% of the panicle numbers, the 10% heading date can be obtained using formula (1):

[0013]

[0014] x = date, y ∈ {80%, 50%, 30%, 10%} (2)

[0015] In one embodiment, preferably, using the Ultralytics library to train, validate, and test on the dataset to select the optimal heading model includes: using the Ultralytics library with default parameters to implement the training, validation, and testing of YOLOv8-X on the constructed dataset. When finally selecting the model, comprehensively consider the mAP, R 2 and RMSE, three evaluation metrics to select the optimal model. Any open-source object detection model can be used instead of YOLOv8.

[0016] On the other hand, the present invention also provides a device for extracting the traits of the plant heading stage, including: a memory; and a processor coupled to the memory, the processor being configured to execute the method for extracting the traits of the plant heading stage described in any one of the above.

[0017] The present invention also provides a computer-readable medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the method for extracting the traits of the plant heading stage described in any one of the above. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] After reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings, the above features and advantages of the present invention can be better understood. In the drawings, the components are not necessarily drawn to scale, and components with similar related characteristics or features may have the same or similar reference numerals.

[0019] Figure 1 It is a schematic flowchart of a method for extracting the traits of the heading stage of a plant shown according to one aspect of the present invention;

[0020] Figure 2A It is an image of a plot obtained by a drone shown according to an embodiment of the present invention;

[0021] Figure 2B It is a small image after cutting obtained by a drone shown according to an embodiment of the present invention;

[0022] Figure 3 It is a black-and-white image of a plot shown according to an embodiment of the present invention;

[0023] Figure 4 It is a semi-automatic plant annotation interface shown according to an embodiment of the present invention;

[0024] Figure 5 It is a schematic diagram of multi-plant trait extraction and annotation shown according to an embodiment of the present invention;

[0025] Figure 6 It is a schematic diagram of the full-cycle curve of the heading stage shown according to an embodiment of the present invention;

[0026] Figure 7A 、 7B They are respectively accuracy evaluation diagrams of the artificial heading stage and the 10% and 30% heading stages shown according to an embodiment of the present invention; and

[0027] Figure 8 It is a schematic diagram of the device structure of a device for extracting the traits of the heading stage of a drone plant shown according to another aspect of the present invention. Detailed implementation manners

[0028] The following specific embodiments illustrate the implementation manners of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. Although the description of the present invention will be introduced in conjunction with the preferred embodiments, this does not mean that the features of this invention are limited to this implementation manner. On the contrary, the purpose of introducing the invention in conjunction with the implementation manner is to cover other alternatives or modifications that may be extended based on the claims of the present invention. In order to provide a deep understanding of the present invention, many specific details will be included in the following description. The present invention can also be implemented without these details. In addition, in order to avoid confusing or obscuring the key points of the present invention, some specific details will be omitted in the description.

[0029] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0030] In addition, the terms "upper", "lower", "left", "right", "top", "bottom", "horizontal" and "vertical" used in the following description should be understood as the directions shown in the paragraph and the related drawings. Such relative terms are only used for the convenience of description and do not mean that the device described therein must be manufactured or operated in a specific direction, and therefore should not be understood as limiting the present invention.

[0031] It is understood that although the terms "first", "second", "third", etc. may be used herein to describe various components, regions, layers and / or parts, these components, regions, layers and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers and / or parts. Therefore, the first component, region, layer and / or part discussed below may be referred to as a second component, region, layer and / or part without departing from some embodiments of the present invention.

[0032] In order to overcome the above-mentioned defects of the prior art, the present invention provides a high-throughput rice heading trait extraction method and device based on drone. The high-throughput rice heading trait extraction method based on drone can improve measurement efficiency, save costs, improve data accuracy and reliability, and provide better support and assistance for rice production.

[0033] This article takes rice as an example for a detailed description.

[0034] Figure 1 The figure is a schematic diagram of the method flow of a method for extracting traits at the heading stage of a plant according to one aspect of the present invention.

[0035] Please refer to Figure 1 The method 100 for extracting plant heading traits provided by the present invention comprises:

[0036] Step 101: using a drone to collect heading images of plants during heading, and pre-processing the heading images.

[0037] In a preferred embodiment, the preprocessing of the heading image may include: extracting the heading image of the plants in the central plot; combining the CIVE vegetation index with the Otsu threshold method to obtain a plant mask; summing the pixel values of the heading image of the plants in the central plot and using a preset window moving average; and obtaining the actual length, width and pixel values of the central plot according to the actual density of the plant planting, so as to obtain and crop the heading image of the central plot.

[0038] For example, first use a drone equipped with a variable-focus digital camera to collect images centered on the plot for 40 days to cover the heading stage, with a frequency of once a day, which can be reduced as appropriate. The finally collected images are as Figure 2A shown.

[0039] Figure 2A is the plot image obtained by the drone according to an embodiment of the present invention.

[0040] As Figure 2A shown, the images collected by the drone contain plants in other plots. First, the image of the central plot needs to be extracted, as Figure 2B shown.

[0041] Figure 3 is the black-and-white image of the plot according to an embodiment of the present invention.

[0042] Combining the CIVE vegetation index with the Otsu threshold method can easily obtain a plant mask ( Figure 3 ), where the 0 pixel value represents the soil area and the 1 pixel value represents the vegetation area. After that, sum the pixel values of each row and column and use moving average for smoothing (window size is 100). According to the actual density of planting, the actual length, width and corresponding pixel values of the plot (3800×2000) can be obtained. Considering the length and width and finding the minimum value of the sum of pixel values, the plot boundary ( Figure 3 ) can be obtained. Cropping from the original image can obtain the plot image ( Figure 3 ).

[0043] Please continue to refer to Figure 1 , the method 100 for extracting the traits of the plant heading stage provided by the present invention further includes:

[0044] Step 102: Construct a dataset of the heading images.

[0045] In a preferred embodiment, the construction of the dataset of the heading images may include: dividing the heading image with a picture pixel of 3800×2000 into small pixel images with a sliding window size and a stride of 1000×1000 pixels by a sliding window method; and semi-automatically annotating the heading images of the small pixel images by using video tags and any composition model.

[0046] Further, constructing the dataset of the heading images may further include: any component model enhances the complete annotation speed of heading plants based on user prompts; and after a small number of target detection models are annotated, pseudo-annotations are generated using this model, and the better model is continuously iteratively used to accelerate the annotation.

[0047] Figure 4 is a semi-automatic plant annotation interface illustrated according to an embodiment of the present invention.

[0048] Semi-automatic annotation is implemented using Label Studio and Segment Anything Model (SAM). LabelStudio is open-source data annotation software ( Figure 4 ), and SAM can achieve complete annotation of rice panicles based on the point prompts given by users. Combining the two can improve the annotation speed. In addition, after training a target detection model by annotating a small number of pictures, pseudo-annotations are generated using this model, and using the above semi-automatic annotation process can further accelerate the annotation.

[0049] In one embodiment, using this process to annotate about 2,000 pictures can obtain a model that meets the prediction accuracy. And the training set is divided into a training set, a validation set, and a test set according to a ratio of 8:1:1.

[0050] Please continue to refer to Figure 1 , the method 100 for extracting traits of the plant heading stage provided by the present invention further includes:

[0051] Step 103: Use the Ultralytics library to train, validate, and test on the dataset.

[0052] In a preferred embodiment, the use of the Ultralytics library to train, validate, and test on the dataset to select the optimal heading model may include: using the Ultralytics library with default parameters to implement the training, validation, and testing of YOLOv8-X on the constructed dataset. Finally, when selecting the model, the optimal model is selected by comprehensively considering three evaluation metrics: mAP, R 2 and RMSE. Any open-source target detection model can be used instead of YOLOv8.

[0053] Please continue to refer to Figure 1 , the method 100 for extracting traits of the plant heading stage provided by the present invention further includes:

[0054] Step 104: Detect and extract the plant heading traits in the complete plant heading stage.

[0055] In a preferred embodiment, similar to when constructing the dataset, for each original image, the plot is extracted, and the sliding window method is used to cut it into small images. The size of the sliding window is 1000×1000 pixels, and the stride is 750×750 pixels. YOLOv8-X is used to predict each small image, and the confidence and intersection over union (IoU) thresholds for non-maximum suppression are 0.3 and 0.5 respectively. Then, non-maximum suppression is used to merge and remove redundant prediction results, and the IoU threshold is 0.25. The number of bounding boxes is calculated to obtain the number of rice panicles in the plot of the image. As Figure 5 shown, Figure 5 is a schematic diagram of multi-plant trait extraction and annotation illustrated according to an embodiment of the present invention.

[0056] Preferably, the use of the Ultralytics library to train, validate, and test on the dataset to select the optimal heading model includes: plotting the rice panicle number-date curve. Defining the maximum rice panicle number as 100%, based on 80%, 50%, 30%, and 10% of the rice panicle numbers, the 10% heading date can be obtained using formula (1):

[0057]

[0058] x = date, y ∈ {80%, 50%, 30%, 10%} (1)

[0059] Figure 6 is a schematic diagram of the full-cycle curve of the heading date illustrated according to an embodiment of the present invention. As Figure 6 shown, in one embodiment, for a single plot, through the above steps, the number of panicles corresponding to all dates can be obtained, and the rice panicle number-date curve can be plotted. Defining the maximum rice panicle number as 100%, based on 80%, 50%, 30%, and 10% of the rice panicle numbers, the 10% heading date can be obtained using formula (1), etc. Example of full panicle period extraction,

[0060] Figure 7A 、 7B are schematic diagrams of the accuracy evaluation of the artificial heading date and the 10% and 30% heading dates illustrated according to an embodiment of the present invention.

[0061] As Figure 7A 、 7B shown, the artificial investigation of the heading date is used to evaluate the 10% and 30% heading dates. As shown in Figures 2-4, the 10% and 30% heading date traits investigated in this process and the artificial investigation of the heading date trait R 2 reach the levels of 0.9387 and 0.9301, which are extremely reliable.

[0062] Figure 8 is a schematic diagram of the device structure of the device for extracting the heading date traits of the drone plants illustrated according to another aspect of the present invention.

[0063] According to another aspect of the present invention, an embodiment of a device 800 for extracting traits of the heading stage of a plant by an unmanned aerial vehicle is also provided herein.

[0064] As Figure 8 shown, the above-mentioned device 800 for extracting traits of the heading stage of a plant by an unmanned aerial vehicle provided in this embodiment may include a memory 801 and a processor 802 coupled to the memory 801. The processor 802 may be configured to implement any of the above-mentioned methods for extracting traits of the heading stage of a plant by an unmanned aerial vehicle.

[0065] According to another aspect of the present invention, an embodiment of a computer storage medium is also provided herein.

[0066] A computer program is stored on the computer storage medium. When the computer program is executed by a processor, the steps of any of the above-mentioned methods for extracting traits of the heading stage of a plant by an unmanned aerial vehicle can be implemented.

[0067] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and the design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

[0068] The processors described in this case can be implemented using electronic hardware, computer software, or any combination thereof. Whether such a processor is implemented as hardware or software will depend upon the particular application and the overall design constraints imposed on the system. As an example, the processors, any part of a processor, or any combination of processors presented in this disclosure can be implemented using a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a state machine, gated logic, discrete hardware circuits, and other suitable processing components configured to perform the various functions described throughout this disclosure. The functionality of the processors, any part of a processor, or any combination of processors presented in this disclosure can be implemented using software executed by a microprocessor, a microcontroller, a DSP, or other suitable platform.

[0069] The steps of the methods or algorithms described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read from, and write to, the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside as discrete components in a user terminal.

[0070] The foregoing description of the disclosure has been provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for extracting traits of the heading stage of a plant, comprising: Collecting heading images during the heading process of the plant using a drone and preprocessing the heading images; Constructing a dataset of the heading images; Using the Ultralytics library to perform training, validation, and testing on the dataset; And Detecting and extracting the plant heading traits during the complete heading stage of the plant.

2. The trait extraction method according to claim 1, wherein The preprocessing of the heading images includes: Extracting the heading images of the plants in the central plot; Combining the CIVE vegetation index with the Otsu threshold method to obtain a plant mask; Summing the pixel values of the heading images of the plants in the central plot and using a preset window moving average; and obtaining the actual length, width, and pixel values of the central plot according to the actual density of the plant planting, so as to obtain and crop the heading images of the central plot.

3. The trait extraction method according to claim 2, wherein The constructing of the dataset of the heading images includes: Dividing the heading images with a pixel size of 3800×2000 into small pixel images with a sliding window size and stride of 1000×1000 pixels by the sliding window method; Semi-automatically annotating the heading images of the small pixel images using video tags and an arbitrary composition model.

4. The trait extraction method according to claim 3, characterized in that, The constructing of the dataset of the heading images further includes: The arbitrary composition model is based on user prompts to improve the complete annotation speed of the heading plants; and After annotating a small number of object detection models, using the model to generate pseudo-annotations and continuing to iteratively use the better model to accelerate the annotation.

5. The trait extraction method according to claim 1, wherein The using of the Ultralytics library to perform training, validation, and testing on the dataset to select the optimal heading model includes: Use the Ultralytics library and default parameters to implement the training, validation, and testing of YOLOv8-X on the constructed dataset. When finally selecting the model, comprehensively consider the three evaluation metrics of mAP, R 2 and RMSE to select the optimal model. Any open-source object detection model can be used to replace YOLOv8..

6. The trait extraction method according to claim 1, wherein The detecting and extracting the plant heading traits during the complete heading stage of the plant includes: Extracting the plot from each original image and also cutting it into small images using the sliding window method. The sliding window size is 1000×1000 pixels and the stride is 750×750 pixels. Using YOLOv8-X to predict each small image, and the confidence and intersection over union thresholds for non-maximum suppression are 0.3 and 0.5 respectively. Then using non-maximum suppression to merge and remove redundant prediction results, and the intersection over union threshold is 0.

25. Calculating the number of bounding boxes to obtain the number of panicles in the image plot.

7. The trait extraction method according to claim 1, wherein The using of the Ultralytics library to perform training, validation, and testing on the dataset to select the optimal heading model includes: Drawing a panicle number-date curve. Defining the maximum panicle number as 100%, and according to 80%, 50%, 30%, and 10% of the panicle numbers, the 10% heading stage can be obtained using formula (1):

8. The trait extraction method according to claim 1, wherein The using of the Ultralytics library to perform training, validation, and testing on the dataset to select the optimal heading model includes: Use the Ultralytics library and default parameters to implement the training, validation, and testing of YOLOv8-X on the constructed dataset. Finally, when selecting the model, comprehensively consider the three evaluation metrics of mAP, R 2 and RMSE to select the optimal model. Any open-source object detection model can be used to replace YOLOv8.

9. An apparatus for extracting traits of the heading stage of a plant, comprising: A memory; And A processor coupled to the memory, the processor being configured to execute the method for extracting traits of the heading stage of a plant according to any one of claims 1 to 8.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for extracting traits of the heading stage of a plant according to any one of claims 1 to 8.

Images