Plant leaf counting method and system based on deep learning, and electronic equipment
Through a deep learning-based method, the optimized YOLOv7 model is trained using multi-angle RGB image sets to solve the problem of low leaf counting accuracy in dwarf plants and achieve higher counting accuracy.
Patent Information
- Application Number
- CN202311789246.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-25
- Publication Date
- 2025-07-18
AI Technical Summary
The prior art is difficult to improve the counting accuracy of leaves of dwarf plants, especially when the leaves are complex and overlapping, the counting error is relatively large.
Using a deep learning-based method, the optimized YOLOv7 model is trained using multiple RGB images at different angles, including the backbone network, the neck network and the prediction network. Through feature matching and feature deduplication, the accuracy of blade count is improved.
The counting accuracy of the leaves of dwarf plants is improved and errors are reduced, especially when the leaves are complex and overlapping.
Smart Images
Figure CN120339154A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of deep learning, and particularly to a method, system and electronic device for counting plant leaves based on deep learning. Background Art
[0002] Currently, most of the research on plant leaf counting targets tall plants or non-tall plants with simple leaf layers. It has always been difficult to improve the measurement accuracy of leaf counting for short plants with severely overlapping and complex-layered leaves (such as strawberry plants). In particular, the complex hierarchical structure and a large number of overlaps of the leaves make it difficult to calculate the number of leaves. Summary of the Invention
[0003] The purpose of the present invention is to provide a method, system and electronic device for counting plant leaves based on deep learning, which improves the counting accuracy of leaves of short plants.
[0004] To achieve the above purpose, the present invention provides the following solutions:
[0005] A method for counting plant leaves based on deep learning includes:
[0006] Obtaining a set of RGB images of a target plant; the target plant is a short plant to be counted for leaves; the set of RGB images includes multiple RGB images at different angles;
[0007] Based on the set of RGB images of the target plant and a leaf counting model, obtaining the number of leaves of the target plant; the leaf counting model is obtained by training an optimized YOLOv7 model using a set of RGB images of multiple short plants and the labels of each RGB image in the corresponding set of RGB images, the label being the actual number of leaves, and the optimized YOLOv7 model includes: a backbone network, a neck network and a prediction network, the neck network includes 2 optimized Ince_Block structures, the prediction network includes 4 optimized detection heads, and the optimized Ince_Block structure includes 7 ordinary convolutional layers, 4 dilated convolutional layers and 2 pooling layers.
[0008] Optionally, obtaining a set of RGB images of a target plant includes:
[0009] Collecting multiple RGB images of the target plant through a drone and a ground camera respectively to obtain an initial set of RGB images;
[0010] Performing feature matching on all RGB images in the initial set of RGB images to determine a set of images to be removed; the set of images to be removed includes N identical RGB images, N>1;
[0011] Remove N - 1 RGB images from the initial RGB image set to obtain the RGB image set of the target plant.
[0012] Optionally, use the ORB algorithm to perform feature matching on all RGB images in the initial RGB image set.
[0013] Optionally, use the RANSAC algorithm to remove N - 1 RGB images from the initial RGB image set to obtain the RGB image set of the target plant.
[0014] Optionally, the backbone network includes: 4 multi - branch stacking modules and 3 transition modules.
[0015] Optionally, the neck network further includes: 6 multi - branch stacking modules, 3 transition modules and 4 ordinary convolutional layers.
[0016] Optionally, the training process of the leaf counting model includes:
[0017] Construct the optimized YOLOv7 model;
[0018] Obtain the RGB image set of multiple dwarf plants and the labels of each RGB image in the corresponding RGB image set;
[0019] Use the RGB image set of each dwarf plant as the input and the labels of each RGB image in the corresponding RGB image set as the output to train the optimized YOLOv7 model to obtain the leaf counting model.
[0020] A plant leaf counting system based on deep learning includes:
[0021] An RGB image set acquisition module for acquiring the RGB image set of the target plant; the target plant is a dwarf plant to be counted for leaves; the RGB image set includes multiple RGB images at different angles;
[0022] A leaf counting module for obtaining the number of leaves of the target plant based on the RGB image set of the target plant and the leaf counting model; the leaf counting model is obtained by training the optimized YOLOv7 model with the RGB image set of multiple dwarf plants and the labels of each RGB image in the corresponding RGB image set, the label is the actual number of leaves, the optimized YOLOv7 model includes: a backbone network, a neck network and a prediction network, the neck network includes 2 optimized Ince_Block structures, the prediction network includes 4 optimized detection heads, and the optimized Ince_Block structure includes 7 ordinary convolutional layers, 4 dilated convolutional layers and 2 pooling layers.
[0023] An electronic device includes a memory and a processor. The memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the above-mentioned plant leaf counting method based on deep learning.
[0024] Optionally, the memory is a readable storage medium.
[0025] According to the specific embodiments provided by the present invention, the following technical effects are disclosed by the present invention:
[0026] The present invention discloses a plant leaf counting method, system and electronic device based on deep learning. First, an RGB image set of a target plant is obtained; the target plant is a dwarf plant to be counted for leaves; the RGB image set includes multiple RGB images at different angles; then, based on the RGB image set of the target plant and a leaf counting model, the number of leaves of the target plant is obtained; the leaf counting model is obtained by training an optimized YOLOv7 model using the RGB image set of multiple dwarf plants and the labels of each RGB image in the corresponding RGB image set, and the label is the actual number of leaves. The optimized YOLOv7 model includes: a backbone network, a neck network and a prediction network. The neck network includes 2 optimized Ince_Block structures, and the prediction network includes 4 optimized detection heads. The optimized Ince_Block structure includes 7 ordinary convolutional layers, 4 dilated convolutional layers and 2 pooling layers. The present invention counts the leaves of dwarf plants through multiple RGB images at different angles and a leaf counting model obtained by training based on the optimized YOLOv7 model, thereby improving the counting accuracy of the leaves of dwarf plants. Description of the Drawings
[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.
[0028] Figure 1 It is a schematic flowchart of a plant leaf counting method based on deep learning provided in Embodiment 1 of the present invention;
[0029] Figure 2 It is a schematic structural diagram of an optimized YOLOv7 model;
[0030] Figure 3 It is a schematic structural diagram of an optimized Ince_Block structure. Detailed Embodiments
[0031] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.
[0032] The purpose of the present invention is to provide a plant leaf counting method, system and electronic device based on deep learning, aiming to improve the counting accuracy of leaves of dwarf plants.
[0033] To make the above objects, features and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0034] Embodiment 1
[0035] Figure 1 It is a schematic flowchart of the plant leaf counting method based on deep learning provided in Embodiment 1 of the present invention. As Figure 1 shown, the plant leaf counting method based on deep learning in this embodiment includes:
[0036] Step 101: Obtain an RGB image set of the target plant; the target plant is a dwarf plant to be counted for leaves; the RGB image set includes multiple RGB images at different angles.
[0037] As an optional implementation manner, Step 101 includes:
[0038] Step 1011: Collect multiple RGB images of the target plant through a drone and a ground camera respectively to obtain an initial RGB image set.
[0039] Specifically, compared with using manual on-site shooting of plant pictures, using a drone to shoot RGB images can improve the shooting speed more. Coupled with the RGB images taken by the ground camera, it can comprehensively shoot the dwarf plants to be counted for leaves from multiple directions.
[0040] Using a drone can improve the shooting speed of plants, but the drone must fly at a height of more than two meters above the plants, otherwise the wind generated by the rotors will blow the plant leaves and affect the image quality. The drone can shoot top views and oblique views at a certain angle, but cannot shoot side views of plants. The ground camera can shoot side views of plants.
[0041] Step 1012: Perform feature matching on all the RGB images in the initial RGB image set to determine an image set to be removed; the image set to be removed includes N identical RGB images, where N>1.
[0042] As an alternative implementation, the ORB algorithm is used to perform feature matching on all RGB images in the initial RGB image set.
[0043] Specifically, when using a drone and a ground camera to cooperate in photographing images of plants, overlapping images may appear due to the positional relationship. Compared with manual duplicate removal, the method of feature matching can improve the efficiency and accuracy of duplicate removal more effectively.
[0044] The ORB algorithm is used for feature extraction and matching. This algorithm can detect key points in the RGB image and calculate the descriptor for each key point.
[0045] In the ORB algorithm, first, the image is subjected to Gaussian smoothing to remove the influence of noise, and then the FAST algorithm is used to detect key points in the image. The FAST algorithm formula is as follows:
[0046] Let the current pixel be p, and its grayscale value be I p , the threshold be t, and the set of pixel points in the circular neighborhood consisting of 16 pixel points be C. Then the condition for p to be a key point is:
[0047] ∑ q∈C I q -I p >t.
[0048] Where, I q is the q-th pixel point in C.
[0049] Secondly, the ORB algorithm calculates the corresponding descriptor for each key point. The descriptor created by the ORB algorithm only contains 1s and 0s and is called a binary descriptor. This descriptor represents the intensity pattern around the key point. Therefore, multiple descriptors can be used to identify a larger area or even a specific object in the image.
[0050] Finally, the FLANN algorithm is used to match the features extracted by the ORB algorithm. The core idea of the FLANN algorithm is to accelerate the search by establishing an index structure for the data. The specific process is as follows: First, the data set is preprocessed to establish an index structure; then the query points are preprocessed to establish a query structure; and the data points closest to the query points are searched for in the index structure.
[0051] Step 1013: Remove N - 1 RGB images in the image set to be removed from the initial RGB image set to obtain the RGB image set of the target plant.
[0052] As an alternative implementation, the RANSAC algorithm is used to remove N - 1 RGB images in the image set to be removed from the initial RGB image set to obtain the RGB image set of the target plant.
[0053] Specifically, the RANSAC algorithm is used to remove some duplicate leaf images. The RANSAC algorithm is a method based on outlier detection and removal, which can help find and remove unnecessary feature points. The specific process is as follows: First, pair-match all feature points and calculate the distances between them. Then, randomly select a set of feature point pairs, calculate the transformation matrix between them, use the transformation matrix to map all feature points in one leaf image to another leaf image, and calculate the distances between them. According to the distance threshold, judge which feature point pairs are "good" matches and which are "bad" matches. If the number of good matches is greater than a certain threshold, it is considered that this set of feature point pairs can be used to estimate the transformation matrix; otherwise, randomly select another set of feature point pairs and repeat the above steps. Use all good match pairs to estimate the optimal transformation matrix, and use this transformation matrix to map all feature points in one leaf image to another leaf image. Finally, remove the duplicate feature points and retain the remaining feature points.
[0054] Compared with the complex feature extraction process in traditional algorithms, the image feature extraction module based on deep learning technology can conveniently and quickly extract the features of images, including the color, texture, and shape of images, etc. For the leaf counting problem, rich feature information is extracted through a multi-layer image feature extraction module to obtain accurate leaf quantity information.
[0055] Step 102: Based on the RGB image set of the target plant and the leaf counting model, obtain the number of leaves of the target plant; the leaf counting model is obtained by training the optimized YOLOv7 model using the RGB image set of multiple short plants and the labels of each RGB image in the corresponding RGB image set, and the label is the actual number of leaves. The optimized YOLOv7 model includes: a backbone network, a neck network, and a prediction network. The neck network includes 2 optimized Ince_Block structures, and the prediction network includes 4 optimized detection heads. The optimized Ince_Block structure includes 7 ordinary convolutional layers, 4 dilated convolutional layers, and 2 pooling layers.
[0056] Specifically, as Figures 2 - 3 shown, the optimized YOLOv7 model specifically includes: a backbone network, a neck network, and a prediction network.
[0057] As an optional implementation manner, the backbone network includes: 4 multi-branch stacking modules (Multi_Concat_Block) and 3 transition modules (Transition_Block).
[0058] Specifically, the backbone network extracts deep feature information from the image, and the deeper the layer, the more feature information is obtained. The residual blocks inside the multi-branch stacking module use skip connections, which alleviates the problem of gradient disappearance caused by increasing the depth in the deep neural network.
[0059] The backbone network also includes: 4 Conv2D_BN_SiLU modules.
[0060] The 4 optimized detection heads (YoloHead) of the prediction network respectively correspond to convolutional layers with sizes of 20×20, 40×40, 80×80, and 160×160, and respectively implement the counting of large targets, medium targets, small and medium targets, and small targets.
[0061] As an optional implementation manner, the neck network also includes: 6 multi-branch stacking modules, 3 transition modules, and 4 ordinary convolutional layers (Conv).
[0062] The neck network can further extract image feature information, and at the same time fuse feature information of different scales to enrich the feature expression ability.
[0063] Specifically, the 7 ordinary convolutional layers in the optimized Ince_Block structure include: 3 CBL1*1, 3 CBL3*3, and 1 CBL7*7; the 4 dilated convolutional layers include: 2 DCBL1*1 and 2 DCBL3*3; the 2 pooling layers are respectively: MaxPool and AvgPool. Figure 3 In it, r1, r3, p3, c, and s1 all represent the parameters of the convolutional layer (ordinary convolutional layer and dilated convolutional layer), Concat represents concatenation, and Sigmoid is the Sigmoid function.
[0064] Specifically, the optimized YOLOv7 model also includes: 4 general convolutional layers (RepConv) and 3 upsampling modules (UpSampling2D).
[0065] As an optional implementation manner, the training process of the leaf counting model includes:
[0066] Construct an optimized YOLOv7 model.
[0067] Obtain the RGB image set of multiple short plants and the labels of each RGB image in the corresponding RGB image set.
[0068] Taking the RGB image set of each short plant as the input and the labels of each RGB image in the corresponding RGB image set as the output, train the optimized YOLOv7 model to obtain the leaf counting model.
[0069] Specifically, the actual training and use process includes:
[0070] (1) Annotate the collected images.
[0071] (2) Configure the model parameters.
[0072] (3) Train the model, and use the loss function to perform forward propagation and backward propagation to update the model parameters.
[0073] (4) Save the trained model for model detection.
[0074] Specifically, for the collected pictures, after removing duplicates, use the Labelimg tool to annotate the remaining pictures to obtain the annotation information of each picture for subsequent model training. Before training, it is necessary to configure the model parameters, including the configuration of the learning rate, batch size, number of iterations, optimizer, etc. of the model, and select appropriate parameters that will neither cause the model to converge slowly nor cause the model loss value to fluctuate too much. After the parameter settings are completed, input the original images and annotation information into the network for training, obtain the loss value of the model for this batch of training through forward propagation using the loss function, and then update the parameters through backward propagation. After multiple trainings, obtain the final model weights. Finally, based on the trained model weights, detect the target images to obtain the information of the target images.
[0075] Embodiment 2
[0076] The plant leaf counting method system based on deep learning in this embodiment includes:
[0077] An RGB image set acquisition module for acquiring an RGB image set of a target plant; the target plant is a dwarf plant to be counted for leaves; the RGB image set includes multiple RGB images at different angles.
[0078] A leaf counting module for obtaining the number of leaves of the target plant based on the RGB image set of the target plant and the leaf counting model; the leaf counting model is trained by using the RGB image set of multiple dwarf plants and the labels of each RGB image in the corresponding RGB image set, and the label is the actual number of leaves. The optimized YOLOv7 model includes: a backbone network, a neck network, and a prediction network. The neck network includes 2 optimized Ince_Block structures, and the prediction network includes 4 optimized detection heads. The optimized Ince_Block structure includes 7 ordinary convolutional layers, 4 dilated convolutional layers, and 2 pooling layers.
[0079] Embodiment 3
[0080] An electronic device includes a memory and a processor. The memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the plant leaf counting method based on deep learning in Embodiment 1.
[0081] As an alternative implementation, the memory is a readable storage medium.
[0082] In this specification, each embodiment is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and reference can be made to the description of the method part for related parts.
[0083] In this article, specific examples are used to elaborate on the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.
Claims
1. A method for counting plant leaves based on deep learning, characterized in that, The method includes: Obtaining an RGB image set of a target plant; the target plant is a dwarf plant to be subjected to leaf counting; the RGB image set includes multiple RGB images at different angles; Based on the RGB image set of the target plant and a leaf counting model, obtaining the number of leaves of the target plant; the leaf counting model is obtained by training an optimized YOLOv7 model using the RGB image sets of multiple dwarf plants and the labels of each RGB image in the corresponding RGB image sets, the label is the actual number of leaves, and the optimized YOLOv7 model includes: a backbone network, a neck network, and a prediction network, the neck network includes 2 optimized Ince_Block structures, the prediction network includes 4 optimized detection heads, and the optimized Ince_Block structure includes 7 ordinary convolutional layers, 4 dilated convolutional layers, and 2 pooling layers.
2. The method for counting plant leaves based on deep learning according to claim 1, wherein Obtaining an RGB image set of a target plant includes: Collecting multiple RGB images of the target plant through a drone and a ground camera respectively to obtain an initial RGB image set; Performing feature matching on all RGB images in the initial RGB image set to determine an image set to be removed; the image set to be removed includes N identical RGB images, N>1; Removing N-1 RGB images in the image set to be removed from the initial RGB image set to obtain the RGB image set of the target plant.
3. The method for counting plant leaves based on deep learning according to claim 2, wherein Using the ORB algorithm to perform feature matching on all RGB images in the initial RGB image set.
4. The method for counting plant leaves based on deep learning according to claim 2, wherein Using the RANSAC algorithm to remove N-1 RGB images in the image set to be removed from the initial RGB image set to obtain the RGB image set of the target plant.
5. The method for counting plant leaves based on deep learning according to claim 1, characterized in that The backbone network includes: 4 multi-branch stacking modules and 3 transition modules.
6. The method for counting plant leaves based on deep learning according to claim 1, wherein The neck network further includes: 6 multi-branch stacking modules, 3 transition modules, and 4 ordinary convolutional layers.
7. The method for counting plant leaves based on deep learning according to claim 1, wherein The training process of the leaf counting model includes: Constructing the optimized YOLOv7 model; Obtaining the RGB image sets of multiple dwarf plants and the labels of each RGB image in the corresponding RGB image sets; Taking the RGB image sets of each dwarf plant as inputs and the labels of each RGB image in the corresponding RGB image sets as outputs, training the optimized YOLOv7 model to obtain the leaf counting model.
8. A plant leaf counting system based on deep learning, characterized in that, The system includes: An RGB image set acquisition module for obtaining an RGB image set of a target plant; the target plant is a dwarf plant to be subjected to leaf counting; the RGB image set includes multiple RGB images at different angles; A leaf counting module, configured to obtain the number of leaves of the target plant based on the RGB image set of the target plant and a leaf counting model; the leaf counting model is obtained by training an optimized YOLOv7 model using the RGB image set of multiple dwarf plants and the labels of each RGB image in the corresponding RGB image set, the label being the actual number of leaves, and the optimized YOLOv7 model includes: a backbone network, a neck network, and a prediction network, the neck network includes 2 optimized Ince_Block structures, the prediction network includes 4 optimized detection heads, and the optimized Ince_Block structure includes 7 ordinary convolutional layers, 4 dilated convolutional layers, and 2 pooling layers.
9. An electronic device, characterized in that, It includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the deep learning-based plant leaf counting method according to any one of claims 1 to 7.
10. An electronic device according to claim 9, characterized in that, The memory is a readable storage medium.