Visual-based plant canopy coverage calculation method, system and device

Abstract

The application discloses a kind of based on vision's plant canopy coverage calculation method, system and device, method includes: through top surface image acquisition device and side surface image acquisition device obtain initial top surface plant image, initial side surface plant image and rotating side surface plant image;Through plant leaf extraction, obtain initial side surface leaf image, rotating side surface leaf image and top surface leaf image;Through skeleton line extraction, obtain initial skeleton line image and rotating skeleton line image, and then obtain the plant stem coordinates in initial top surface plant image;Through leaf vertex coordinate set and plant stem coordinates, obtain the angle with the plane where side surface image acquisition device is located;Through the angle, the plant to be measured is rotated and image acquisition is carried out to obtain top surface plant image set, and the plant canopy coverage is obtained by projection and area analysis.The side surface and top surface of plant are imaged and analyzed in the application, improve calculation accuracy and calculation efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a plant canopy coverage calculation method, system and device based on vision. BACKGROUND

[0002] The plant canopy coverage refers to the percentage between the projected area of the plant canopy and the land area, and is closely related to the growth stage and health degree of the plant. In the early growth stage of the plant, the plant canopy coverage gradually increases until it reaches a high level in the middle and late growth stage. If the plant canopy coverage grows slowly or is lower than the normal level, it may mean that the plant growth is affected by adverse factors such as nutrient deficiency, pest and disease invasion and drought. Analysis based on the plant canopy coverage can effectively adjust the management measures and promote the growth of the plant. Studies have shown that there is a positive correlation between the yield of many crops and the plant canopy coverage. Therefore, monitoring and analyzing the canopy coverage can estimate the yield potential in advance during the growth season of the plant, providing an important reference for agricultural production. At the same time, the size of the plant canopy coverage will affect the evaporation of water in the soil and the absorption and utilization of nutrients by the plant. For the field with high plant canopy coverage, the water evaporation is relatively slow, and the irrigation frequency and water quantity can be appropriately reduced to avoid waste of water resources. When the plant canopy coverage is large, the fertilization scheme needs to be reasonably adjusted to meet the growth needs of the plant and improve the fertilizer utilization rate. Based on this, the analysis of the plant canopy coverage is of great significance for understanding the growth status of the plant and protecting the ecological environment, and is an important content in the process of agricultural and ecological research.

[0003] The existing method for measuring the plant canopy coverage based on vision usually collects an image parallel to the top of the plant by an image collection device, and then calculates the canopy coverage of the plant. However, due to the imaging principle of the image collection device, when the plant is located at a position close to the image collection device, the imaging is large, and when the plant is located at a position far away from the image collection device, the imaging is small. When the plant leaves are distributed on the side of the plant, they present a certain curve in the image, that is, the distances between the points in the plant leaves and the image collection device are different. The plant leaves at the positions close to the image collection device have a certain distortion in the image due to the imaging principle, and the existing method does not correct the distortion existing in the image, resulting in low calculation accuracy of the plant canopy coverage. SUMMARY

[0004] The present application provides a plant canopy coverage calculation method, system and device based on vision to overcome the shortcomings in the prior art.

[0005] To solve the above technical problems, the present application solves the problems by the following technical solutions:

[0006] A plant canopy coverage calculation method based on vision, comprising the following steps:

[0007] An initial top plant image and an initial side plant image of the plant to be tested are acquired, a rotated side plant image is acquired after the plant to be tested is horizontally rotated, and plant leaf extraction is performed to obtain a top leaf image, an initial side leaf image, and a rotated side leaf image;

[0008] An initial side stem image and a rotated side stem image are obtained based on the initial side plant image, the initial side leaf image, the rotated side plant image, and the rotated side leaf image, and an initial skeleton line image and a rotated skeleton line image are obtained through skeleton line extraction;

[0009] Parallel straight lines are constructed, the intersection between the initial skeleton line image and the rotated skeleton line image and the parallel straight lines is analyzed, the initial stem position and the rotated stem position are obtained, and then the plant stem coordinates in the initial top plant image are obtained;

[0010] A leaf vertex coordinate set is obtained through the top leaf image, a leaf vector set is formed based on the plant stem coordinates and the leaf vertex coordinate set, and the angle between the leaf vector and the imaging plane of the side view image acquisition device is obtained;

[0011] The plant to be tested is rotated based on the angle, when the leaf vector corresponding to the plant leaf is parallel to the imaging plane of the side view image acquisition device, the top plant image of the corresponding plant leaf is obtained, and the horizontal projection total area is obtained by projecting to the horizontal plane, and then the plant canopy coverage is obtained.

[0012] As an implementable manner, the plant leaf extraction comprises the following steps:

[0013] A plant image set is acquired, and a plant leaf region is labeled to form a plant data set, and the plant data set is divided into a training set and a verification set;

[0014] The plant leaf detection pre-training model is trained, tested, and verified based on the training set and the verification set to obtain a plant leaf detection model;

[0015] The initial side plant image, the rotated side plant image, and the initial top plant image are detected and analyzed by the plant leaf detection model to obtain the initial side leaf region, the rotated side leaf region, and the top leaf region, and the initial side leaf image, the rotated side leaf image, and the top leaf image are obtained in combination with the initial side plant image, the rotated side plant image, and the initial top plant image.

[0016] As an implementable manner, the initial side stem image and the rotated side stem image are obtained based on the initial side plant image, the initial side leaf image, the rotated side plant image, and the rotated side leaf image, comprising the following steps:

[0017] The initial lateral plant image and the initial lateral leaf image are binarized and subtracted to obtain the initial lateral stem image.

[0018] The rotated side plant image and the rotated side leaf image are binarized and then subtracted to obtain the rotated side stem image.

[0019] As one possible implementation, the step of obtaining the initial skeleton line image and the rotated skeleton line image through skeleton line extraction includes the following steps:

[0020] Extract the non-zero pixels from the initial side stem image and the rotated side stem image to obtain the initial side point set and the rotated side point set;

[0021] Obtain the initial neighborhood pixel set and the rotation neighborhood pixel set of the initial side point set and the rotation side point set respectively. Set a pixel threshold. If the number of pixels connected to the current initial side point or the current rotation side point in the initial pixel set or the rotation pixel set meets the pixel threshold, then the current initial side point or the current rotation side point is the plant endpoint.

[0022] Traverse the initial side point set and the rotated side point set to obtain the initial plant endpoint set and the rotated plant endpoint set, and obtain the corresponding thickness data. Based on the first preset condition and the second preset condition, judge and cut the thickness data to obtain the initial skeleton line image and the rotated skeleton line image.

[0023] As one possible implementation, the step of acquiring the corresponding thickness data, judging and cutting the thickness data through a first preset condition and a second preset condition to obtain an initial skeleton line image and a rotated skeleton line image includes the following steps:

[0024] The thickness data at the initial plant endpoint and the rotated plant endpoint are obtained separately, as shown below:

[0025]

[0026] If the thickness data meets the preset threshold and the neighboring pixel values ​​and thickness data of the initial plant endpoint and the rotating plant endpoint meet the first preset condition, the initial plant endpoint and the rotating plant endpoint are cut until the second preset condition is met, and the initial skeleton line image and the rotating skeleton line image are obtained.

[0027] in, Represents the pixel value of a pixel. Represents thickness data. Indicates the neighborhood range. Represents the first pixel in the neighborhood pixel set. The x-coordinate of each pixel Represents the first pixel in the neighborhood pixel set. The ordinate of each pixel This represents the change in the ordinate of the neighboring pixels.

[0028] As one possible implementation, the step of analyzing the intersection points between the initial skeleton line image and the rotated skeleton line image and the parallel straight lines to obtain the initial stem position and the rotated stem position includes the following steps:

[0029] Preset parallel lines, and obtain the number of first and second intersection points of the parallel lines with the initial skeleton line image and the rotated skeleton line image, as shown below:

[0030]

[0031]

[0032] The first and second intersection point sets are obtained by sliding parallel lines, and the maximum value of the first and second intersection points is obtained.

[0033] Based on the first parallel line and the second parallel line corresponding to the maximum value of the first intersection point and the maximum value of the second intersection point, respectively, the positions of the first parallel line and the second parallel line in the initial skeleton line image and the rotated skeleton line image are obtained, and the initial stem position and the rotated stem position are obtained.

[0034] in, Indicates parallel lines. Represents variables, Indicates the first The number of first or second intersection points corresponding to the parallel lines. Represents the Dirac function, Indicates the first The x-coordinate of each pixel This indicates the number of pixels.

[0035] As one possible implementation, obtaining the angle between the blade vector and the imaging plane where the side view image acquisition device is located includes the following steps:

[0036] Based on the leaf vertex coordinate set and the plant stem coordinates, a leaf vector set is constructed, wherein the direction of the leaf vector points to the leaf vertex coordinate;

[0037] Set the direction vector, obtain the angle between the blade vector and the direction vector, and obtain the angle between the blade vector and the imaging plane where the side view image acquisition device is located, as shown below:

[0038]

[0039]

[0040] in, , Represents the direction vector. Represents the blade vector. Indicates the first The x-coordinate of each leaf vertex. Indicates the first The ordinate of the vertex coordinates of each leaf. The x-coordinate represents the coordinates of the plant stem. The vertical coordinate represents the coordinates of the plant stem. Indicates the included angle. Indicates reverse cut, To represent cosine, This indicates the number of leaf vertex coordinates. This indicates the number of leaves on the plant.

[0041] As one possible implementation method, the plant canopy coverage rate is obtained through the following steps:

[0042] Obtain the mapping matrix, combine it with the actual leaf thickness, and project the top surface of the plant image onto the horizontal plane to obtain the set of horizontal coordinate points of the plant, as shown below:

[0043]

[0044] Using the set of horizontal coordinate points of the plant and the shoelace formula, the horizontal projected area of ​​each leaf in the top plant image is obtained, as shown below:

[0045]

[0046] Based on the horizontal projected area of ​​each plant's leaves, the total horizontal projected area is obtained. Combined with the area of ​​the measured region, the plant canopy coverage rate is obtained.

[0047] in, Indicates the first The first top-side plant image The first in the leaves of the plant The x-coordinate of the horizontal coordinate point of the plant at each point Indicates the first The first top-side plant image The first in the leaves of the plant The horizontal coordinates and vertical coordinates of the plant at each point. No. The first top-side plant image The first in the leaves of the plant The x-coordinate of each pixel Indicates the first The first top-side plant image The first in the leaves of the plant a vertical coordinate of a pixel point, a first a first an actual leaf thickness of a first plant leaf, a horizontal projection area of a first plant leaf, a horizontal projection area of a first plant leaf, a coordinate point number of a first plant leaf, 、 、 、 a change ratio coefficient.

[0048] As an implementation manner, the initial top-view plant image is collected by a top-view image collection device located at the top of the plant to be measured, the initial side-view plant image is collected by a side-view image collection device located at the side of the plant to be measured, and the rotated side-view plant image is collected by the side-view image collection device located at the side of the plant to be measured after the plant to be measured is horizontally rotated by 90°;

[0049] The imaging plane of the top-view image collection device is parallel to the upper surface of the plant, and the imaging center of the top-view image collection device and the center of the plant are located on the same straight line.

[0050] The imaging plane of the side-view image collection device is perpendicular to the upper surface of the plant, and the imaging center of the side-view image collection device, the imaging center of the top-view image collection device and the center of the plant are located on the same plane.

[0051] A vision-based plant canopy coverage calculation system comprises an image acquisition module, a skeleton line extraction module, a stem coordinate acquisition module, an angle calculation module and a canopy coverage calculation module.

[0052] The image acquisition module acquires an initial top-view plant image and an initial side-view plant image of a plant to be measured, acquires a rotated side-view plant image after the plant to be measured is horizontally rotated, and extracts plant leaves to obtain a top-view leaf image, an initial side-view leaf image and a rotated side-view leaf image.

[0053] The skeleton line extraction module obtains an initial side-view stem image and a rotated side-view stem image based on the initial side-view plant image, the initial side-view leaf image, the rotated side-view plant image and the rotated side-view leaf image, and obtains an initial skeleton line image and a rotated skeleton line image through skeleton line extraction.

[0054] The stem coordinate acquisition module constructs a parallel straight line, analyzes the intersection between the initial skeleton line image and the rotated skeleton line image and the parallel straight line, obtains an initial stem position and a rotated stem position, and further obtains plant stem coordinates in the initial top-view plant image.

[0055] The included angle calculation module obtains a leaf vertex coordinate set through the top surface leaf image, forms a leaf vector set based on the plant stem coordinates and the leaf vertex coordinate set, and respectively obtains the included angle between the leaf vector and the imaging plane of the side view image acquisition device;

[0056] The canopy coverage calculation module rotates the plant to be measured based on the included angle, respectively obtains the top surface plant image of the corresponding plant leaf when the leaf vector of the corresponding plant leaf is parallel to the imaging plane of the side view image acquisition device, projects to the horizontal plane, obtains the total horizontal projection area, and further obtains the canopy coverage of the plant.

[0057] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method as follows:

[0058] An initial top surface plant image and an initial side surface plant image of a plant to be measured are obtained, a rotated side surface plant image is obtained after the plant to be measured is horizontally rotated, and plant leaf extraction is performed to obtain a top surface leaf image, an initial side surface leaf image, and a rotated side surface leaf image;

[0059] An initial side surface stem image and a rotated side surface stem image are obtained based on the initial side surface plant image, the initial side surface leaf image, the rotated side surface plant image, and the rotated side surface leaf image, and an initial skeleton line image and a rotated skeleton line image are obtained through skeleton line extraction;

[0060] A parallel straight line is constructed, the intersection between the initial skeleton line image and the rotated skeleton line image and the parallel straight line is analyzed, the initial stem position and the rotated stem position are obtained, and further the plant stem coordinates in the initial top surface plant image are obtained;

[0061] A leaf vertex coordinate set is obtained through the top surface leaf image, a leaf vector set is formed based on the plant stem coordinates and the leaf vertex coordinate set, and the included angle between the leaf vector and the imaging plane of the side view image acquisition device is respectively obtained;

[0062] The plant to be measured is rotated based on the included angle, the top surface plant image of the corresponding plant leaf is respectively obtained when the leaf vector of the corresponding plant leaf is parallel to the imaging plane of the side view image acquisition device, projected to the horizontal plane, the total horizontal projection area is obtained, and further the canopy coverage of the plant is obtained.

[0063] A plant canopy coverage calculation device based on vision includes a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor implements the method as follows when the computer program is executed:

[0064] An initial top plant image and an initial side plant image of the plant to be measured are acquired, a rotated side plant image is acquired after the plant to be measured is horizontally rotated, and plant leaf extraction is performed to obtain a top leaf image, an initial side leaf image and a rotated side leaf image;

[0065] Based on the initial side plant image, the initial side leaf image, the rotated side plant image and the rotated side leaf image, an initial side stem image and a rotated side stem image are obtained, and an initial skeleton line image and a rotated skeleton line image are obtained through skeleton line extraction;

[0066] A parallel straight line is constructed, the intersection between the initial skeleton line image and the rotated skeleton line image and the parallel straight line is analyzed, the initial stem position and the rotated stem position are obtained, and then the plant stem coordinates in the initial top plant image are obtained;

[0067] A set of leaf vertex coordinates is obtained through the top leaf image, a set of leaf vectors is formed based on the plant stem coordinates and the set of leaf vertex coordinates, and the angle between the leaf vector and the imaging plane of the side image acquisition device is obtained;

[0068] The plant to be measured is rotated based on the angle, when the leaf vector corresponding to the plant leaf is parallel to the imaging plane of the side image acquisition device, the top plant image of the corresponding plant leaf is obtained and projected to the horizontal plane to obtain the total horizontal projection area, and then the plant canopy coverage rate is obtained.

[0069] The present application has the following technical effects:

[0070] The present application acquires images of the top and side of the plant to be measured, and through plant leaf segmentation, skeleton line extraction and image analysis, the plant to be measured is rotated by the rotating device to correct the image, so that the plant leaf of the plant to be measured is parallel to the side image acquisition device, the top plant image is obtained, the horizontal projection area is obtained through horizontal plane projection and area calculation, and then the plant canopy coverage rate is obtained. The present application can correct the distortion of the image, more accurately calculate the horizontal projection area of the plant to be measured, and improve the calculation accuracy of the plant canopy coverage rate. BRIEF DESCRIPTION OF DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0072] Figure 1is a flowchart of the method of the present application;

[0073] Figure 2 is a schematic diagram of the system of the present application;

[0074] Figure 3 is a schematic diagram of the image acquisition device of the present application;

[0075] Figure 4 is a schematic diagram of the top view image of the plant to be measured of the present application;

[0076] Figure 5 is a schematic diagram of the side view image of the plant to be measured of the present application;

[0077] Figure 6 is a schematic diagram of the side stem image of the present application;

[0078] Figure 7 is a schematic diagram of the leaf image of the plant to be measured of the present application;

[0079] Figure 8 is a schematic diagram of the coordinate projection mapping of the present application. DETAILED DESCRIPTION

[0080] The present application will be further described in detail below in conjunction with the embodiments. The following embodiments are an explanation of the present application and the present application is not limited to the following embodiments.

[0081] Embodiment 1:

[0082] A visual-based plant canopy coverage calculation method, as shown in Figure 1 , includes the following steps:

[0083] S100, obtaining an initial top plant image and an initial side plant image of a plant to be measured, obtaining a rotated side plant image after rotating the plant to be measured horizontally, and performing plant leaf extraction to obtain a top leaf image, an initial side leaf image, and a rotated side leaf image;

[0084] S200, obtaining an initial side stem image and a rotated side stem image based on the initial side plant image, the initial side leaf image, the rotated side plant image, and the rotated side leaf image, and obtaining an initial skeleton line image and a rotated skeleton line image through skeleton line extraction;

[0085] S300, constructing a parallel straight line, analyzing the intersection between the initial skeleton line image and the rotated skeleton line image and the parallel straight line, obtaining an initial stem position and a rotated stem position, and further obtaining a plant stem coordinate in the initial top plant image;

[0086] S400, obtaining a set of leaf vertex coordinates from the top surface leaf image, forming a set of leaf vectors based on the plant stem coordinates and the set of leaf vertex coordinates, and obtaining an angle between the leaf vector and the imaging plane of the side view image acquisition device;

[0087] S500, rotating the plant to be measured based on the angle, obtaining a top surface plant image of the corresponding plant leaf when the leaf vector of the corresponding plant leaf is parallel to the imaging plane of the side view image acquisition device, projecting the top surface plant image to a horizontal plane to obtain a total horizontal projection area, and further obtaining a plant canopy coverage rate.

[0088] The present application obtains an initial side leaf image, a rotated side leaf image and a top surface leaf image by image acquisition of the top surface and the side surface of the plant to be measured and extraction of the plant leaf, obtains plant stem coordinates in the initial top surface image through skeleton line extraction and analysis, obtains an angle between the plant leaf and the side view image acquisition device, rotates the plant to be measured based on the angle and acquires a top surface plant image, realizes image correction of the plant to be measured, further projects to a horizontal plane and calculates the area to obtain a horizontal projection area, and combines the area of the measurement region to obtain a plant canopy coverage rate. The present application solves the distortion caused by the positional relationship between the plant to be measured and the image acquisition device, realizes high-precision calculation of the plant canopy coverage rate, helps to understand the growth status of the plant to be measured, and has important research significance for promoting agricultural development.

[0089] The present embodiment provides an image acquisition device for a plant to be measured, as shown in Figure 3 The present embodiment provides an image acquisition device for a plant to be measured, as shown in

[0090] The plant to be measured is fixed on the rotating device 3, the light supplementing environment in the box is adjusted, the top surface of the plant to be measured is imaged by the top view image acquisition device 1 to obtain an initial top surface plant image , asFigure 4 As shown in the side view image acquisition device 2, the side of the plant to be measured is imaged to obtain the initial side plant image , the plant to be measured is rotated by 90° by the rotating device 3, the rotating mode in this embodiment includes clockwise and counterclockwise rotation, and the side of the plant to be measured after rotation is imaged by the side view image acquisition device 2 to obtain the rotated side plant image , wherein the initial side plant image or the rotated side plant image is as shown in Figure 5 .

[0091] The initial top plant image , the initial side plant image and the rotated side plant image are processed by a deep learning method to extract plant leaves. In this embodiment, yolov5 framework is used. yolov5 inherits the excellent tradition of YOLO series, significantly improves the inference speed through parallel processing and fp16 inference technology, and uses deep separable convolution, mobilenetv3 module and optimized network structure to reduce the number of model layers and parameter amount, and further improve the detection speed; yolov5 uses an efficient feature extractor, adopts residual connection, attention mechanism and spatial pyramid pooling technology, enhances the feature extraction capability, and can extract more rich feature information from the plant image; at the same time, the objective function of the target detection task is optimized, the GIoU loss function is used, the difference between the predicted box and the real box is more accurately measured, the precision of the model is improved, the plant leaf detection model obtained by training is used to extract plant leaves, and the initial side leaf image , the rotated side leaf image and the top leaf image are obtained, which specifically includes the following steps:

[0092] Step 1: obtain a plant image set and label the plant leaf area to form a plant data set, divide the plant data set into a training set and a validation set according to 8:2;

[0093] Step 2: adopt yolov5 framework, construct boundary box loss function by GIoU Loss, use stochastic gradient descent as optimizer to improve model convergence speed, train and validate model based on plant data set and boundary box loss function to obtain plant leaf detection model;

[0094] Step 3: detect and analyze the initial side plant image , the rotated side plant image and the initial top plant image by the plant leaf detection model to obtain the initial side leaf area, the rotated side leaf area and the top leaf area;

[0095] Step 4: mapping the initial side blade area, the rotated side blade area and the top blade area to the initial side plant image, the rotated side plant image and the initial top plant image to obtain an initial side blade image , a rotated side blade image and a top blade image .

[0096] Step 5: performing image binarization on the initial side plant image and the initial side blade image and subtracting the images to obtain an initial side stem image Step 6: performing image binarization on the rotated side plant image and the rotated side blade image and subtracting the images to obtain a rotated side stem image , wherein the initial side stem image or the rotated side stem image is as shown in Figure 6 .

[0097] Step 7: based on the initial side stem image and the rotated side stem image respectively, performing skeleton line extraction by edge cutting to obtain an initial skeleton line image and a rotated skeleton line image, including the following steps:

[0098] Step 1: obtaining non-zero pixel points in the initial side stem image and the rotated side stem image to form an initial side point set and a rotated side point set ;

[0099] Step 2: obtaining eight-neighbor pixel values of the initial side points and the rotated side points respectively , wherein and , if the number of pixel values connected to the current initial side point or the rotated side point in the eight-neighbor pixel values is greater than a pixel threshold, i.e. at most two pixel values in the eight-neighbor are connected to the current pixel, then the current initial side point or the rotated side point is a plant endpoint, and in the embodiment, the pixel threshold is 2, and the specific formula is as follows:

[0100]

[0101] Step 3: traversing the initial side point set and the rotated side point set by the method of step 2 to obtain an initial plant endpoint set and a rotated plant endpoint set, and counting the number of adjacent pixel values in the up-down direction of the initial plant endpoint and the rotated plant endpoint to obtain thickness data at the initial plant endpoint and the rotated plant endpoint, which is represented as follows:

[0102]

[0103] Step 4: Starting from the initial plant end points and the rotated plant end points, traversing in the direction connected to the end points, and the initial traversal direction being the horizontal direction, traversing the initial plant end point set and the rotated plant end point set to obtain the thickness data of all initial plant end points and rotated plant end points;

[0104] Step 5: If the eight-neighborhood pixel value and the thickness data corresponding to the initial plant end point and the rotated plant end point satisfy the first preset condition, cutting the initial plant end point and the rotated plant end point until the second preset condition is satisfied, completing the cutting of the current initial plant end point and the rotated plant end point, traversing the initial plant end point set and the rotated plant end point set, repeatedly cutting and judging to obtain the initial skeleton line image and the rotated skeleton line image , wherein the first preset condition and the second preset condition are as follows:

[0105]

[0106] , wherein, represents the pixel value of a pixel point, represents the horizontal coordinate of a pixel point, represents the vertical coordinate of a pixel point, represents the thickness data, represents the neighborhood range, represents the horizontal coordinate of the th pixel point in the neighborhood pixel point set, represents the vertical coordinate of the th pixel point in the neighborhood pixel point set, represents the vertical coordinate change value in the neighborhood pixel point set.

[0107] The initial stem position and the rotated stem position are obtained by the straight line sliding intersection method of the preset parallel straight line, and then the plant stem coordinates in the initial top view plant image are obtained. The plant stem coordinates refer to the stem data of the plant to be measured in the initial top view plant image. If the stem is a cylinder in the top view image of the plant to be measured, what is seen is a circle, and the plant stem coordinates are the coordinates of the circle. The horizontal and vertical coordinates of the circle are obtained through the initial stem position and the rotated stem position, and then the plant stem coordinates are obtained. The specific steps include the following steps:

[0108] Step 1: presetting a parallel straight line , obtaining the coordinate data of the initial skeleton line image and the rotated skeleton line image, and calculating the intersection points of the parallel straight line and the initial skeleton line image and the rotated skeleton line image;

[0109] Step 2: sliding the parallel straight line from left to right to obtain the intersection points of the parallel straight line and the initial skeleton line image and the rotated skeleton line image The number of intersections under different values ​​yields the first and second intersection count sets, as shown below:

[0110]

[0111] Step 3: Obtain the maximum value from the first and second intersection point quantity sets to get the maximum value of the first intersection point. and the maximum value at the second intersection point ;

[0112] Step 4: Obtain the first parallel line corresponding to the maximum value of the first intersection point and the second parallel line corresponding to the maximum value of the second intersection point. The position of the first parallel line in the initial skeleton line image is the initial stem position, and the position of the second parallel line in the rotated skeleton line image is the rotated stem position.

[0113] Step 5: Determine the plant stem coordinates in the initial top surface plant image by using the initial stem position and the rotated stem position;

[0114] in, This represents a group of parallel lines. Represents variables, Indicates the first The number of first or second intersection points corresponding to the parallel lines. Represents the Dirac function, Indicates the first The x-coordinate of each pixel This indicates the number of pixels.

[0115] Extract the blade mask image from the top blade image, such as... Figure 7 As shown, the leaf vertices in the top leaf image are obtained based on the leaf mask image, forming a set of leaf vertex coordinates. The process involves the following steps: [The steps are described in the original text, but the provided excerpt ends here.]

[0116] Step 1: Based on the top surface blade image The leaves of each plant were segmented, and the vertex coordinates of each leaf were extracted. ;

[0117] Step 2: Construct a leaf vector using the leaf vertex coordinates and the plant stem coordinates. Leaf vector For the first The vector pointing to the leaf vertex coordinates, formed by the coordinates of the plant leaf and the plant stem, is represented as follows:

[0118]

[0119] Step 3: Assume a direction vector The first vector is calculated by using the dot product and magnitude of the vectors. The angle between the leaf vector corresponding to the leaf of the plant and the imaging plane where the side view image acquisition device is located is specifically represented as follows:

[0120]

[0121]

[0122]

[0123] Step 4: Traverse the coordinates of the vertex of all plant leaves to obtain the angle between each plant leaf and the imaging plane where the side view image acquisition device is located.

[0124] in, Represents the direction vector. Represents the blade vector. Indicates the first The x-coordinate of each leaf vertex. Indicates the first The ordinate of the vertex coordinates of each leaf. The x-coordinate represents the coordinates of the plant stem. The vertical coordinate represents the coordinates of the plant stem. Indicates the included angle. Indicates reverse cut, Represents sine. To represent cosine, This indicates the number of leaf vertex coordinates. This indicates the number of leaves on the plant.

[0125] In this embodiment, by establishing a mapping relationship between the turntable control signal and the included angle, the included angle is converted into a turntable control signal that the rotating device can recognize, and the turntable control signal is sent to the rotating device. The turntable control signal is represented as follows:

[0126]

[0127] in, Indicates turntable control signals. Represents a mapping function. Indicates the included angle.

[0128] After receiving the turntable control signal, the rotating device rotates according to its internal drive mechanism, causing the plant to be tested placed on the rotating device to rotate accordingly. After rotation, the leaf vectors corresponding to the plant leaves should be parallel to the imaging plane of the side image acquisition device. Images of the rotated plant are acquired by the top image acquisition device to obtain a top plant image set. The top plant image set is then projected onto a horizontal plane, and the projected area is calculated to obtain the horizontal projected area. A schematic diagram of the coordinate projection mapping is shown below. Figure 8 As shown, 1 represents the top view image acquisition device, 2 represents the side view image acquisition device, and 3 represents the rotation device. Based on the horizontal projected area and the measurement area, the plant canopy coverage is obtained, including the following steps:

[0129] Step 1: In this embodiment, the intrinsic and extrinsic parameters of the image acquisition device are obtained through camera calibration. The intrinsic parameters describe the internal parameters of the image acquisition device, and the extrinsic parameters describe the relationship between the camera coordinate system and the world coordinate system. By using the intrinsic and extrinsic parameters of the image acquisition device, the correspondence between the three-dimensional geometric position of a point on the plant under test and the two-dimensional geometric position of the corresponding point in the image is determined, thus obtaining the mapping matrix. ;

[0130] Step 2: Obtain plant leaf images from the top plant image set using the plant leaf detection model. For the first leaf image in the plant leaf image... The first leaf of the plant A set of horizontal coordinate points of the plant is obtained by mapping the coordinates of several points to the horizontal plane using a mapping matrix. set of horizontal coordinate points of the plant Coordinates of the top-view plant image and actual blade thickness The relationship between them is represented as follows:

[0131]

[0132]

[0133] Step 3: The coordinate matrix of the vertices of the polygon formed by the set of horizontal coordinate points of the plant is as follows: The horizontal projected area of ​​plant leaves is calculated using the shoelace formula based on the coordinate matrix of polygon vertices, as follows:

[0134]

[0135] Step 4: Traverse all the leaves of the plant to obtain the horizontal projected area of ​​each leaf. Summate the total horizontal projected area to obtain the area of ​​the entire measurement region. Combine the total horizontal projected area to obtain the plant canopy coverage rate. The total horizontal projected area and the plant canopy coverage rate are expressed as follows:

[0136]

[0137]

[0138] in, Indicates the first The first top-side plant image The first of the blades The x-coordinate of the horizontal coordinate point of the plant at each point Indicates the first The first top-side plant image The first of the blades The horizontal coordinates and vertical coordinates of the plant at each point. No. The first top-side plant image The first of the blades The x-coordinate of each pixel Indicates the first The first top-side plant image The first of the blades The ordinate of each pixel Indicates the first The first top-side plant image The actual blade thickness of the blade. Indicates the first The horizontal projected area of ​​the blade Indicates the first The number of coordinate points on each blade. Represents the total horizontal projected area. Indicates the number of leaves. Indicates the area of ​​the measurement region. Indicates the plant canopy coverage. , , This represents the coefficient of variation.

[0139] Example 2:

[0140] A vision-based system for calculating plant canopy coverage, such as... Figure 2 As shown, it includes an image acquisition module 100, a skeleton line extraction module 200, a stem coordinate acquisition module 300, an angle calculation module 400, and a canopy coverage calculation module 500.

[0141] The image acquisition module 100 acquires the initial top surface image and the initial side surface image of the plant to be tested, rotates the plant to be tested horizontally to acquire the rotated side surface image, and extracts the plant leaves to obtain the top surface leaf image, the initial side surface leaf image, and the rotated side surface leaf image.

[0142] The skeleton line extraction module 200 obtains an initial side stem image and a rotated side stem image based on the initial side plant image, the initial side leaf image, the rotated side plant image and the rotated side leaf image, and obtains an initial skeleton line image and a rotated skeleton line image through skeleton line extraction;

[0143] The stem coordinate acquisition module 300 constructs a parallel straight line, analyzes the intersection between the initial skeleton line image and the rotated skeleton line image and the parallel straight line, obtains an initial stem position and a rotated stem position, and further obtains a plant stem coordinate in the initial top plant image;

[0144] The included angle calculation module 400 obtains a leaf vertex coordinate set through the top leaf image, forms a leaf vector set based on the plant stem coordinate and the leaf vertex coordinate set, and respectively obtains the included angle between the leaf vector and the imaging plane of the side view image acquisition device;

[0145] The canopy coverage calculation module 500 rotates the plant to be measured based on the included angle, obtains a top plant image of the corresponding plant leaf when the leaf vector of the corresponding plant leaf is parallel to the imaging plane of the side view image acquisition device, projects the top plant image to a horizontal plane, obtains a total horizontal projection area, and further obtains a plant canopy coverage.

[0146] Various changes and modifications can be made to the application without departing from the spirit and scope of the application. All equivalent technical solutions also belong to the scope of the application.

[0147] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the embodiments can be referred to each other.

[0148] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, device or computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0149] The application is described with reference to the flowcharts and / or block diagrams of the method, terminal device (system), and computer program product according to the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate an apparatus for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one flow or a plurality of flows and / or blocks Figure 1 one flow or a plurality of flows and / or blocks

[0150] These computer program instructions can also be stored in a computer-readable memory capable of directing the computer or other programmable data processing terminal device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one flow or a plurality of flows and / or blocks Figure 1 one flow or a plurality of flows and / or blocks

[0151] These computer program instructions can also be loaded into a computer or other programmable data processing terminal device, so that a series of operation steps are performed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one flow or a plurality of flows and / or blocks Figure 1 one flow or a plurality of flows and / or blocks

[0152] It should be noted that:

[0153] The phrase "one embodiment" or "an embodiment" appearing in the specification means that a specific feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Therefore, the phrase "one embodiment" or "an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment.

[0154] In addition, it should be noted that the specific embodiments described in the specification, the shape of the zero, the name of the component, etc. can be different. Any equivalent or simple change made in accordance with the structure, features and principles described in the patent concept of the present application is included in the protection scope of the present application. Those skilled in the art can make various modifications or supplements to the described specific embodiments or use similar ways to replace them, as long as they do not deviate from the structure of the present application or exceed the scope defined by the present claims.

Claims

1. A method for calculating plant canopy coverage based on vision, characterized in that, The method comprises the following steps: Obtaining an initial top-view plant image and an initial side-view plant image of a plant to be tested, obtaining a rotated side-view plant image by horizontally rotating the plant to be tested, and extracting plant leaves to obtain a top-view leaf image, an initial side-view leaf image, and a rotated side-view leaf image, wherein the rotated side-view plant image is obtained by horizontally rotating the plant to be tested by 90° and collected by a side-view image collection device located on the side of the plant to be tested, and the initial top-view plant image is collected by a top-view image collection device located on the top of the plant to be tested; Obtaining an initial side stem image and a rotated side stem image based on the initial side-view plant image, the initial side-view leaf image, the rotated side-view plant image, and the rotated side-view leaf image, and obtaining an initial skeleton line image and a rotated skeleton line image by skeleton line extraction; Constructing a parallel straight line, analyzing the intersection between the initial skeleton line image and the rotated skeleton line image and the parallel straight line, obtaining an initial stem position and a rotated stem position, and further obtaining a plant stem coordinate in the initial top-view plant image; Predefining a parallel straight line, obtaining a first intersection number and a second intersection number of the parallel straight line and the initial skeleton line image and the rotated skeleton line image, and representing as follows: Obtaining a first intersection number set and a second intersection number set by sliding the parallel straight line, and obtaining a first intersection maximum value and a second intersection maximum value; Obtaining the positions of a first parallel straight line and a second parallel straight line in the initial skeleton line image and the rotated skeleton line image based on the first parallel straight line and the second parallel straight line corresponding to the first intersection maximum value and the second intersection maximum value, and obtaining the initial stem position and the rotated stem position; wherein, represents parallel straight lines, represents a variable, represents the first intersection number or the second intersection number corresponding to the represents the Dirac function, represents the horizontal coordinate of the represents the number of pixel points; Obtaining a leaf vertex coordinate set through the top-view leaf image, forming a leaf vector set based on the plant stem coordinate and the leaf vertex coordinate set, and respectively obtaining the angles between the leaf vectors and the imaging plane of the side-view image collection device; Rotating the plant to be tested based on the angles, obtaining a top-view plant image of a corresponding plant leaf when the leaf vector of the corresponding plant leaf is parallel to the imaging plane of the side-view image collection device, projecting the top-view plant image to a horizontal plane, obtaining a total horizontal projection area, and further obtaining a plant canopy coverage rate.

2. The vision-based canopy cover calculation method of claim 1, wherein, The plant leaf extraction comprises the following steps: Obtaining a plant image set and labeling a plant leaf region to form a plant data set, and dividing the plant data set into a training set and a verification set; Training, testing, and verifying a plant leaf detection pre-training model based on the training set and the verification set, and obtaining a plant leaf detection model; Detecting and analyzing the initial side-view plant image, the rotated side-view plant image, and the initial top-view plant image through the plant leaf detection model to obtain an initial side-view leaf region, a rotated side-view leaf region, and a top-view leaf region, and combining the initial side-view plant image, the rotated side-view plant image, and the initial top-view plant image to obtain an initial side-view leaf image, a rotated side-view leaf image, and a top-view leaf image.

3. The vision-based canopy cover calculation method of claim 1, wherein, The method for obtaining an initial side stem image and a rotated side stem image based on the initial side-view plant image, the initial side-view leaf image, the rotated side-view plant image, and the rotated side-view leaf image comprises the following steps: The initial side plant image and the initial side leaf image are binarized and subtracted to obtain an initial side stem image; The rotated side plant image and the rotated side leaf image are binarized and subtracted to obtain a rotated side stem image.

4. The vision-based canopy cover calculation method of claim 1, wherein, The initial skeleton line image and the rotated skeleton line image are obtained by the following steps: Non-zero pixel points in the initial side stem image and the rotated side stem image are extracted to obtain an initial side point set and a rotated side point set; Initial neighborhood pixel point sets and rotated neighborhood pixel point sets of the initial side point set and the rotated side point set are obtained, and a pixel threshold is preset; if the number of pixel points connected to the current initial side point or the current rotated side point in the initial pixel point set or the rotated pixel point set meets the pixel threshold, the current initial side point or the current rotated side point is a plant endpoint; The initial side point set and the rotated side point set are traversed to obtain an initial plant endpoint set and a rotated plant endpoint set, and corresponding thickness data are obtained; the thickness data are judged and cut according to first and second preset conditions to obtain the initial skeleton line image and the rotated skeleton line image.

5. The vision-based canopy cover calculation method of claim 4, wherein, The initial skeleton line image and the rotated skeleton line image are obtained by the following steps: The thickness data at the initial plant endpoints and the rotated plant endpoints are obtained and represented as follows: If the thickness data meet a preset threshold and the neighborhood pixel values and the thickness data of the initial plant endpoints and the rotated plant endpoints meet the first preset condition, the initial plant endpoints and the rotated plant endpoints are cut until the second preset condition is met, and the initial skeleton line image and the rotated skeleton line image are obtained; wherein, represents a pixel value of a pixel point, represents thickness data, represents a neighborhood range, represents an abscissa of a pixel point in a neighborhood pixel point set, represents an ordinate of a pixel point in a neighborhood pixel point set, represents an ordinate of a pixel point in a neighborhood pixel point set, represents an ordinate of a pixel point in a neighborhood pixel point set, represents an ordinate change value in a neighborhood pixel point set. 6.The vision-based canopy coverage calculation method according to claim 1, wherein, The angle between the leaf vector and the imaging plane of the side view image acquisition device is obtained by the following steps: Based on the leaf vertex coordinate set and the plant stem coordinate, a leaf vector set is constructed, wherein the direction of the leaf vector points to the leaf vertex coordinate; A direction vector is set, the angle between the leaf vector and the direction vector is obtained, and the angle between the leaf vector and the imaging plane of the side view image acquisition device is obtained and represented as follows: wherein , denotes a direction vector, denotes a blade vector, denotes the horizontal coordinate of the coordinate of the nth blade vertex, denotes the vertical coordinate of the coordinate of the nth blade vertex, denotes the horizontal coordinate of the coordinate of the nth blade vertex, denotes the vertical coordinate of the coordinate of the nth blade vertex, denotes the horizontal coordinate of the coordinate of the plant stem, denotes the vertical coordinate of the coordinate of the plant stem, denotes the included angle, denotes the inverse tangent, denotes the cosine, denotes the number of blade vertex coordinates, denotes the number of plant blades.

7. The vision-based canopy cover calculation method of claim 1, wherein, The plant canopy coverage is obtained by the following steps: A mapping matrix is obtained, the actual leaf thickness is combined, the top plant image is projected to a horizontal plane to obtain a plant horizontal coordinate point set, and represented as follows: The horizontal projection area of each plant leaf in the top plant image is obtained by using a shoelace formula based on the plant horizontal coordinate point set, and represented as follows: Based on the horizontal projection area of each plant leaf, a total horizontal projection area is obtained, and the plant canopy coverage is obtained by analyzing the area of the measurement region; wherein, represents the horizontal coordinate of the plant horizontal coordinate point of the i-th point in the j-th plant leaf in the i-th top-view plant image, represents the vertical coordinate of the plant horizontal coordinate point of the i-th point in the j-th plant leaf in the i-th top-view plant image, represents the horizontal coordinate of the i-th pixel point in the j-th plant leaf in the i-th top-view plant image, represents the vertical coordinate of the i-th pixel point in the j-th plant leaf in the i-th top-view plant image, represents the actual leaf thickness of the j-th plant leaf in the i-th top-view plant image, represents the horizontal projection area of the j-th plant leaf, represents the number of coordinate points of the j-th plant leaf, represents a change ratio coefficient.​​​​​​​​​​​​​​​​​​ 8.The vision-based plant canopy coverage calculation method according to claim 1, wherein, The imaging plane of the top view image acquisition device is parallel to the upper surface of the plant, and the imaging center of the top view image acquisition device and the center of the plant are located on the same straight line; The imaging plane of the side view image acquisition device is perpendicular to the upper surface of the plant, and the imaging center of the side view image acquisition device, the imaging center of the top view image acquisition device and the center of the plant are located on the same plane. ​ 9. A vision-based plant canopy coverage calculation system, comprising: The image acquisition module, the skeleton line extraction module, the stem coordinate acquisition module, the angle calculation module and the canopy coverage calculation module are included. The image acquisition module acquires an initial top plant image and an initial side plant image of a plant to be measured, acquires a rotated side plant image after rotating the plant to be measured horizontally, and performs plant leaf extraction to obtain a top leaf image, an initial side leaf image and a rotated side leaf image. The rotated side plant image is acquired by rotating the plant to be measured by 90 degrees horizontally and collected by a side-view image collection device located at the side of the plant to be measured. The initial top plant image is collected by a top-view image collection device located at the top of the plant to be measured. The skeleton line extraction module obtains an initial side stem image and a rotated side stem image based on the initial side plant image, the initial side leaf image, the rotated side plant image and the rotated side leaf image, and obtains an initial skeleton line image and a rotated skeleton line image through skeleton line extraction. The stem coordinate acquisition module constructs a parallel straight line, analyzes the intersection between the initial skeleton line image and the rotated skeleton line image and the parallel straight line, obtains the initial stem position and the rotated stem position, and further obtains the plant stem coordinates in the initial top plant image. A preset parallel straight line is obtained. The first intersection number and the second intersection number of the parallel straight line and the initial skeleton line image and the rotated skeleton line image are obtained, and are expressed as follows: The first intersection number set and the second intersection number set are obtained by sliding the parallel straight line, and the first intersection maximum value and the second intersection maximum value are obtained. Based on the first parallel straight line and the second parallel straight line corresponding to the first intersection maximum value and the second intersection maximum value, the positions of the first parallel straight line and the second parallel straight line in the initial skeleton line image and the rotated skeleton line image are obtained, and the initial stem position and the rotated stem position are obtained. wherein, represents parallel straight lines, represents a variable, represents the first represents the first or second intersection point number corresponding to the parallel straight lines, represents the Dirac function, represents the first represents the horizontal coordinate of the pixel point, represents the pixel point number; The angle calculation module obtains a leaf vertex coordinate set through the top leaf image, forms a leaf vector set based on the plant stem coordinates and the leaf vertex coordinate set, and respectively obtains the angle between the leaf vector and the imaging plane of the side-view image collection device. The canopy coverage calculation module rotates the plant to be measured based on the angle. When the leaf vector corresponding to the plant leaf is parallel to the imaging plane of the side-view image collection device, the top plant image of the corresponding plant leaf is acquired and projected to the horizontal plane to obtain the total horizontal projection area, and further obtain the plant canopy coverage.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1-9. The computer program is executed by the processor to realize the method of any one of claims 1 to 8. 11.A vision-based plant canopy coverage calculation device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein, The processor executes the computer program to realize the method of any one of claims 1 to 8.

Images