Method, system and device for calculating plant canopy coverage rate based on vision

Abstract

The invention discloses a vision-based plant canopy coverage rate calculation method, system and device. The method comprises the following steps: acquiring an initial top surface plant image, an initial side surface plant image and a rotary side surface plant image through a top surface image acquisition device and a side surface image acquisition device; through plant leaf extraction, an initial side leaf image, a rotary side leaf image and a top leaf image are obtained; through skeleton line extraction, obtaining an initial skeleton line image and a rotating skeleton line image, and further obtaining plant stalk coordinates in the initial top surface plant image; acquiring an included angle with a plane where the side image acquisition device is located through the leaf vertex coordinate set and the plant stalk coordinate; and rotating the to-be-detected plant through the included angle, carrying out image acquisition to obtain a top surface plant image set, and carrying out projection and area analysis to obtain the plant canopy coverage rate. According to the invention, images of the side surfaces and the top surfaces of the plants are collected and analyzed, and the calculation accuracy and the calculation efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method, system and device for calculating plant canopy coverage based on vision. Background Art

[0002] Plant canopy coverage refers to the ratio of the projected area of ​​the plant canopy to the land area and is closely related to the plant's growth stage and health. During the early stages of plant growth, canopy coverage gradually increases, reaching higher levels in the middle and late stages of growth. Slow or subnormal growth may indicate that plant growth is being impacted by adverse factors such as nutrient deficiencies, pests and diseases, and drought. Analysis based on canopy coverage can effectively adjust management measures and promote plant growth. Studies have shown a positive correlation between the yield of many crops and canopy coverage. Therefore, monitoring and analyzing canopy coverage can help predict yield potential well in advance during the growing season, providing an important reference for agricultural production. Furthermore, canopy coverage affects soil water evaporation and nutrient absorption by plants. For fields with higher canopy coverage, water evaporation is relatively slow, so irrigation frequency and water volume can be appropriately reduced to avoid water waste. When canopy coverage is high, fertilization programs need to be adjusted to meet plant growth needs and improve fertilizer utilization. Based on this, the analysis of plant canopy coverage is of great significance for understanding plant growth conditions and protecting the ecological environment, and is an important content that cannot be obtained in the process of agricultural and ecological research.

[0003] Existing methods for visually measuring plant canopy coverage typically involve an image capture device positioned parallel to the top of the plant to capture an image, which is then used to calculate the plant's canopy coverage. However, due to the imaging principle of the image capture device, the image is larger when the plant is close to the device and smaller when the plant is farther away. When leaves are distributed along the side of the plant, they appear curved in the image. This means that the distances between each point on the leaf and the image capture device vary. At locations closer to the device, the leaf image is distorted due to the imaging principle. Existing methods do not correct for this distortion, resulting in low accuracy in calculating the plant's canopy coverage. Summary of the Invention

[0004] In view of the shortcomings of the prior art, the present invention provides a method, system and device for calculating plant canopy coverage based on vision.

[0005] In order to solve the above technical problems, the present invention is solved by the following technical solutions: A method for calculating plant canopy coverage based on vision includes the following steps: Acquire an initial top surface plant image and an initial side surface plant image of the plant to be tested, horizontally rotate the plant to be tested to obtain a rotated side surface plant image, and perform plant leaf extraction to obtain a top surface leaf image, an initial side surface leaf image, and a rotated side surface leaf image; Obtaining an initial side stem image and a rotated side stem image based on the initial side plant image, the initial side leaf image, the rotated side plant image, and the rotated side leaf image, and obtaining an initial skeleton line image and a rotated skeleton line image by skeleton line extraction; Constructing parallel straight lines, analyzing the intersections between the initial skeleton line image and the rotated skeleton line image and the parallel straight lines, obtaining the initial stem position and the rotated stem position, and then obtaining the plant stem coordinates in the initial top surface plant image; The leaf vertex coordinate set is obtained through the top leaf image, and the leaf vector set is formed based on the plant stem coordinates and the leaf vertex coordinate set, and the angle between the leaf vector and the imaging plane where the side view image acquisition device is located is obtained respectively; The plant to be tested is rotated based on the included angle. When the leaf vector of the corresponding plant leaf is parallel to the imaging plane where the side view image acquisition device is located, the top surface plant image of the corresponding plant leaf is obtained and projected onto the horizontal plane to obtain the total horizontal projection area, and then the plant canopy coverage rate is obtained.

[0006] As an embodiment, the plant leaf extraction comprises the following steps: Obtain a plant image set and annotate the plant leaf area to form a plant dataset, and divide the plant dataset into a training set and a validation set; The plant leaf detection pre-training model is trained, tested, and verified based on the training set and the validation set to obtain a plant leaf detection model; The initial side plant image, the rotated side plant image and the initial top plant image are detected and analyzed through the plant leaf detection model to obtain the initial side leaf area, the rotated side leaf area and the top leaf area. The initial side plant image, the rotated side plant image and the initial top plant image are combined to obtain the initial side leaf image, the rotated side leaf image and the top leaf image.

[0007] As an implementable method, obtaining the initial side stem image and the rotated side stem image based on the initial side plant image, the initial side leaf image, the rotated side plant image, and the rotated side leaf image includes the following steps: Binarize the initial side plant image and the initial side leaf image, and obtain the initial side stem image by image subtraction; The rotating side plant image and the rotating side leaf image are binarized and subtracted to obtain the rotating side stem image.

[0008] As an implementable method, obtaining the initial skeleton line image and the rotated skeleton line image by skeleton line extraction includes the following steps: Extracting non-zero pixel points in the initial side stem image and the rotated side stem image to obtain an initial side point set and a rotated side point set; Obtaining the initial neighborhood pixel point set and the rotation neighborhood pixel point set of the initial side point set and the rotation side point set respectively, presetting a pixel threshold, and if the number of pixels in the initial pixel point set or the rotation pixel point set connected to the current initial side point or the current rotation side point meets the pixel threshold, then the current initial side point or the current rotation side point is the plant endpoint; The initial side point set and the rotated side point set are traversed to obtain the initial plant endpoint set and the rotated plant endpoint set, and the corresponding thickness data are obtained. The thickness data is judged and cut according to the first preset condition and the second preset condition to obtain the initial skeleton line image and the rotated skeleton line image.

[0009] As an implementable method, the corresponding thickness data is obtained, and the thickness data is judged and cut according to the first preset condition and the second preset condition to obtain the initial skeleton line image and the rotated skeleton line image, including the following steps: The thickness data at the initial plant endpoint and the rotation plant endpoint are obtained respectively, as shown below:

[0010] If the thickness data satisfies a preset threshold and the neighborhood pixel values ​​and thickness data of the initial plant endpoint and the rotated plant endpoint meet a first preset condition, the initial plant endpoint and the rotated plant endpoint are cut until a second preset condition is met, thereby obtaining an initial skeleton line image and a rotated skeleton line image; in, Represents the pixel value of a pixel, Indicates thickness data, represents the neighborhood range, Indicates the number of neighborhood pixels The horizontal coordinate of the pixel point, Indicates the number of neighborhood pixels The vertical coordinate of the pixel point, Indicates the change value of the vertical coordinate of the neighborhood pixel points.

[0011] As an implementable method, analyzing the intersections between the initial skeleton line image and the rotated skeleton line image and the parallel straight lines to obtain the initial stem position and the rotated stem position includes the following steps: Preset parallel lines, obtain the first and second intersection points of the parallel lines with the initial skeleton line image and the rotated skeleton line image, expressed as follows:

[0012]

[0013] Obtaining a first set of intersection points and a second set of intersection points by sliding parallel straight lines, and obtaining a maximum value of the first intersection point and a maximum value of the second intersection point; Based on the first parallel straight line and the second parallel straight line corresponding to the first intersection maximum value and the second intersection maximum value respectively, the positions of the first parallel straight line and the second parallel straight line in the initial skeleton line image and the rotated skeleton line image are obtained to obtain the initial stem position and the rotated stem position; in, represents parallel straight lines, Represents a variable, Indicates the The number of first intersection points or second intersection points corresponding to the parallel straight lines, represents the Dirac function, Indicates the The horizontal coordinate of the pixel point, Indicates the number of pixels.

[0014] As an implementable method, obtaining the angle between the blade vector and the imaging plane where the side view image acquisition device is located includes the following steps: Based on the leaf vertex coordinate set and the plant stem coordinates, a leaf vector set is constructed, wherein the direction of the leaf vector points to the leaf vertex coordinates; Set the direction vector, obtain the angle between the blade vector and the direction vector, and obtain the angle between the blade vector and the imaging plane where the side view image acquisition device is located, which is expressed as follows:

[0015]

[0016] in, , represents the direction vector, represents the leaf vector, Indicates the The horizontal coordinates of the vertex coordinates of the leaves, Indicates the The vertical coordinates of the leaf vertices, represents the horizontal coordinate of the plant stem, represents the vertical coordinate of the plant stem, represents the angle, represents the inverse tangent, represents cosine, represents the number of leaf vertex coordinates, Indicates the number of leaves on the plant.

[0017] As an implementable method, the plant canopy coverage is obtained by the following steps: Obtain the mapping matrix, combine it with the actual leaf thickness, project the top surface plant image onto the horizontal plane, and obtain the plant horizontal coordinate point set, which is expressed as follows:

[0018] The horizontal projection area of ​​each plant leaf in the top surface plant image is obtained by using the plant horizontal coordinate point set and the shoelace formula, which is expressed as follows:

[0019] Based on the horizontal projection area of ​​each plant's leaves, the total horizontal projection area was obtained, and combined with the measurement area for analysis, the plant canopy coverage was obtained; in, Indicates the The top surface plant image The first leaf of the plant The horizontal coordinate of the plant point is the horizontal coordinate of the point, Indicates the The top surface plant image The first leaf of the plant The horizontal coordinate of the plant point is the vertical coordinate of the point, No. The top surface plant image The first leaf of the plant The horizontal coordinate of the pixel point, Indicates the The top surface plant image The first leaf of the plant The vertical coordinate of the pixel point, Indicates the The top surface plant image The actual leaf thickness of the plant leaves, Indicates the The horizontal projection area of ​​the leaves of a plant, Indicates the The number of coordinate points of a plant leaf, 、 、 Indicates the change proportional coefficient.

[0020] As an embodiment, the initial top surface plant image is acquired by a top view image acquisition device located at the top of the plant to be measured, the initial side surface plant image is acquired by a side view image acquisition device located at the side of the plant to be measured, and the rotated side surface plant image is acquired by rotating the plant to be measured horizontally by 90° and then acquired by a side view image acquisition device located at the side of the plant to be measured; The imaging plane of the top-view image acquisition device is parallel to the upper surface of the plant, and the imaging center of the top-view image acquisition device and the center of the plant are located in a straight line; The imaging plane of the side view image acquisition device is perpendicular to the upper surface of the plant, and the imaging center of the side view image acquisition device, the imaging center of the top view image acquisition device and the center of the plant are located in the same plane.

[0021] A vision-based plant canopy coverage calculation system includes an image acquisition module, a skeleton line extraction module, a stem coordinate acquisition module, an angle calculation module, and a canopy coverage calculation module; The image acquisition module acquires an initial top surface plant image and an initial side surface plant image of the plant to be tested, horizontally rotates the plant to be tested to acquire a rotated side surface plant image, and extracts plant leaves to obtain a top surface leaf image, an initial side surface leaf image, and a rotated side surface leaf image; The skeleton line extraction module obtains the initial side stem image and the rotated side stem image based on the initial side plant image, the initial side leaf image, the rotated side plant image and the rotated side leaf image, and obtains the initial skeleton line image and the rotated skeleton line image through skeleton line extraction; The stem coordinate acquisition module constructs parallel straight lines, analyzes the intersections between the initial skeleton line image and the rotated skeleton line image and the parallel straight lines, obtains the initial stem position and the rotated stem position, and further obtains the plant stem coordinates in the initial top surface plant image; The angle calculation module obtains a leaf vertex coordinate set from the top leaf image, forms a leaf vector set based on the plant stem coordinates and the leaf vertex coordinate set, and obtains the angle between the leaf vector and the imaging plane where the side view image acquisition device is located; The canopy coverage calculation module rotates the plant to be measured based on the included angle. When the leaf vector of the corresponding plant leaf is parallel to the imaging plane where the side view image acquisition device is located, the top surface plant image of the corresponding plant leaf is obtained and projected onto the horizontal plane to obtain the total horizontal projection area, and then the plant canopy coverage is obtained.

[0022] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following method: Acquire an initial top surface plant image and an initial side surface plant image of the plant to be tested, horizontally rotate the plant to be tested to obtain a rotated side surface plant image, and perform plant leaf extraction to obtain a top surface leaf image, an initial side surface leaf image, and a rotated side surface leaf image; Obtaining an initial side stem image and a rotated side stem image based on the initial side plant image, the initial side leaf image, the rotated side plant image, and the rotated side leaf image, and obtaining an initial skeleton line image and a rotated skeleton line image by skeleton line extraction; Constructing parallel straight lines, analyzing the intersections between the initial skeleton line image and the rotated skeleton line image and the parallel straight lines, obtaining the initial stem position and the rotated stem position, and then obtaining the plant stem coordinates in the initial top surface plant image; The leaf vertex coordinate set is obtained through the top leaf image, and the leaf vector set is formed based on the plant stem coordinates and the leaf vertex coordinate set, and the angle between the leaf vector and the imaging plane where the side view image acquisition device is located is obtained respectively; The plant to be tested is rotated based on the included angle. When the leaf vector of the corresponding plant leaf is parallel to the imaging plane where the side view image acquisition device is located, the top surface plant image of the corresponding plant leaf is obtained and projected onto the horizontal plane to obtain the total horizontal projection area, and then the plant canopy coverage rate is obtained.

[0023] A device for calculating plant canopy coverage based on vision includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the following method is implemented: Acquire an initial top surface plant image and an initial side surface plant image of the plant to be tested, horizontally rotate the plant to be tested to obtain a rotated side surface plant image, and perform plant leaf extraction to obtain a top surface leaf image, an initial side surface leaf image, and a rotated side surface leaf image; Obtaining an initial side stem image and a rotated side stem image based on the initial side plant image, the initial side leaf image, the rotated side plant image, and the rotated side leaf image, and obtaining an initial skeleton line image and a rotated skeleton line image by skeleton line extraction; Constructing parallel straight lines, analyzing the intersections between the initial skeleton line image and the rotated skeleton line image and the parallel straight lines, obtaining the initial stem position and the rotated stem position, and then obtaining the plant stem coordinates in the initial top surface plant image; The leaf vertex coordinate set is obtained through the top leaf image, and the leaf vector set is formed based on the plant stem coordinates and the leaf vertex coordinate set, and the angle between the leaf vector and the imaging plane where the side view image acquisition device is located is obtained respectively; The plant to be tested is rotated based on the included angle. When the leaf vector of the corresponding plant leaf is parallel to the imaging plane where the side view image acquisition device is located, the top surface plant image of the corresponding plant leaf is obtained and projected onto the horizontal plane to obtain the total horizontal projection area, and then the plant canopy coverage rate is obtained.

[0024] The present invention has significant technical effects due to the adoption of the above technical solutions: The present invention captures images of the top and side surfaces of the plant to be tested, and then performs plant leaf segmentation, skeleton line extraction, and image analysis. The image is then rotated by a rotating device to correct the image, aligning the leaves of the plant to be tested with the side image capture device. The top surface image of the plant is captured, and the horizontal projection area is calculated through horizontal plane projection and area calculation to obtain the plant canopy coverage. The method of the present invention corrects image distortion, more accurately calculates the horizontal projection area of ​​the plant to be tested, and thus improves the accuracy of calculating the plant canopy coverage. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 It is a schematic flow diagram of the method of the present invention; Figure 2 It is an overall schematic diagram of the system of the present invention; Figure 3 Schematic diagram of the image acquisition device of the present invention; Figure 4 Schematic diagram of the top surface image of the plant to be tested according to the present invention; Figure 5 Schematic diagram of the side image of the plant to be tested according to the present invention; Figure 6 It is a schematic diagram of the side stem image of the present invention; Figure 7 Schematic diagram of the image of the leaves of the plant to be tested according to the present invention; Figure 8 It is a schematic diagram of coordinate projection mapping of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described in detail below with reference to the examples. The following examples are intended to explain the present invention but the present invention is not limited to the following examples.

[0028] Example 1: A method for calculating plant canopy coverage based on vision, such as Figure 1 As shown, the following steps are included: S100, obtaining an initial top surface plant image and an initial side surface plant image of the plant to be tested, horizontally rotating the plant to be tested to obtain a rotated side surface plant image, and performing plant leaf extraction to obtain a top surface leaf image, an initial side surface leaf image, and a rotated side surface leaf image; S200, obtaining an initial side stem image and a rotated side stem image based on the initial side plant image, the initial side leaf image, the rotated side plant image, and the rotated side leaf image, and obtaining an initial skeleton line image and a rotated skeleton line image by skeleton line extraction; S300, constructing parallel straight lines, analyzing the intersections between the initial skeleton line image and the rotated skeleton line image and the parallel straight lines, obtaining the initial stem position and the rotated stem position, and then obtaining the plant stem coordinates in the initial top surface plant image; S400, obtaining a leaf vertex coordinate set from a top leaf image, forming a leaf vector set based on the plant stem coordinates and the leaf vertex coordinate set, and obtaining angles between the leaf vectors and an imaging plane where a side view image acquisition device is located; S500. Rotate the plant to be measured based on the included angle. When the leaf vector of the corresponding plant leaf is parallel to the imaging plane where the side view image acquisition device is located, obtain the top surface plant image of the corresponding plant leaf and project it onto the horizontal plane to obtain the total horizontal projection area, and then obtain the plant canopy coverage rate.

[0029] The present invention acquires an initial side leaf image, a rotated side leaf image, and a top leaf image by performing top surface image acquisition and side image acquisition on the plant to be tested, extracts and analyzes the plant leaves, obtains the coordinates of the plant stem in the initial top surface image, obtains the angle between the plant leaves and the side image acquisition device, rotates the plant to be tested based on the angle, and acquires the top surface plant image, thereby achieving image correction of the plant to be tested, and then obtains the horizontal projection area by projecting onto a horizontal plane and calculating the area, and obtains the plant canopy coverage rate in combination with the area of ​​the measurement area. The method of the present invention solves the distortion caused by the positional relationship between the plant to be tested and the image acquisition device, and achieves high-precision calculation of the plant canopy coverage rate, which helps to understand the growth status of the plant to be tested and has important research significance for promoting agricultural development.

[0030] This embodiment provides an image acquisition device for a plant to be tested, such as Figure 3As shown, 1 represents a top view image acquisition device, 2 represents a side view image acquisition device, and 3 represents a rotating device. In this embodiment, to ensure that a clear and well-exposed image of the plant to be measured is obtained, the image acquisition device is installed in a box. A circular opening is opened on the rotating device 3 for fixing the plant to be measured. The rotating device 3 is controlled by a servo motor, which can realize 360° rotation of the rotating device and can stop at any desired angle. The top view image acquisition device 1 is installed on the top of the box, and the imaging plane of the top view image acquisition device 1 is parallel to the plane of the rotating device 3. The imaging center of the top view image acquisition device 1 and the center of the circular opening in the rotating device 3 are located on the same straight line. The side view image acquisition device 2 is installed on the side of the box, and the imaging plane of the side view image acquisition device 2 is perpendicular to the plane of the rotating device 3. The imaging center of the side view image acquisition device 2, the imaging center of the top view image acquisition device 1, and the center of the circular opening in the rotating device 3 are located on the same plane.

[0031] The plant to be tested is fixed on the rotating device 3, the fill light environment in the box is adjusted, and the top surface of the plant to be tested is imaged by the top view image acquisition device 1 to obtain the initial top surface plant image. ,like Figure 4 As shown, the side image acquisition device 2 is used to acquire images of the side of the plant to be tested, and an initial side plant image is obtained. The plant to be tested is rotated 90° by the rotating device 3. The rotation modes in this embodiment include clockwise and counterclockwise rotation. The side image acquisition device 2 is used to capture the image of the side of the rotated plant to be tested, and a rotated side plant image is obtained. , where the initial side plant image or the rotated side plant image is as follows Figure 5 shown.

[0032] The initial top surface plant image is processed by deep learning method , initial side plant image and rotating side plant images , to extract plant leaves. In this embodiment, the yolov5 framework is used. Yolov5 inherits the fine tradition of the YOLO series. Through parallel processing and fp16 inference technology, it significantly improves the inference speed, and adopts deep separable convolution, mobilenetv3 module and optimized network structure to reduce the number of model layers and parameters, further improving the detection speed; yolov5 uses an efficient feature extractor, and adopts residual connection, attention mechanism and spatial pyramid pooling and other technologies to enhance feature extraction capabilities, and can extract richer feature information from plant images; at the same time, it optimizes the objective function of the target detection task, uses the GIoU loss function to more accurately measure the difference between the predicted box and the real box, improves the accuracy of the model, and extracts plant leaves by training the obtained plant leaf detection model to obtain the initial side leaf image. , rotating side blade image and top leaf images , specifically including the following steps: Step 1: Obtain a plant image set and mark the plant leaf area to form a plant dataset, which is divided into a training set and a validation set in an 8:2 ratio. Step 2: Using the yolov5 framework, the bounding box loss function is constructed through GIoU Loss. Stochastic gradient descent is used as the optimizer to improve the model convergence speed. The model is trained and verified based on the plant dataset and the bounding box loss function to obtain a plant leaf detection model. Step 3: Use the plant leaf detection model to detect the initial side plant image , rotate the side plant image and initial top surface plant image Perform detection and analysis to obtain the initial side blade area, the rotating side blade area, and the top blade area; Step 4: Map the initial side leaf area, the rotated side leaf area, and the top leaf area to the initial side plant image, the rotated side plant image, and the initial top plant image to obtain the initial side leaf image. , rotating side blade image and top leaf images .

[0033] The initial side plant image and the initial side leaf image are binarized and the initial side stem image is obtained by image subtraction. The rotating side plant image and the rotating side leaf image are binarized and the rotating side stem image is obtained by image subtraction. , where the initial side stalk image or the rotated side stalk image is as follows Figure 6 shown.

[0034] Based on the initial side stem image and the rotated side stem image, skeleton line extraction is performed by edge cutting method to obtain the initial skeleton line image and the rotated skeleton line image, including the following steps: Step 1: Obtain the initial side stem image and the non-zero pixel points in the rotated side stem image to form the initial side point set and the rotation side point set ; Step 2: Get the eight-neighborhood pixel values ​​of the initial side point and the rotated side point respectively ,in, and If the number of pixel values ​​in the eight neighborhoods connected to the current initial side point or the rotated side point is greater than the pixel threshold, that is, at most two pixel values ​​in the eight neighborhoods are connected to the current pixel, then the current initial side point or the rotated side point is the plant endpoint. In this embodiment, the pixel threshold is 2, and the specific formula is as follows:

[0035] Step 3: Traverse the initial side point set using the method in step 2 and the rotation side point set , get the initial plant endpoint set and the rotated plant endpoint set, count the number of adjacent pixel values ​​in the upper and lower directions of the initial plant endpoint and the rotated plant endpoint, and get the thickness data at the initial plant endpoint and the rotated plant endpoint, which is expressed as follows:

[0036] Step 4: Taking the initial plant endpoint and the rotated plant endpoint as the starting point, and the direction connected to the endpoint as the traversal direction, with the initial traversal direction being the horizontal direction, traverse the initial plant endpoint set and the rotated plant endpoint set to obtain the thickness data of all the initial plant endpoints and the rotated plant endpoints; Step 5: If the eight-neighborhood pixel values ​​and thickness data corresponding to the initial plant endpoint and the rotated plant endpoint meet the first preset condition, the initial plant endpoint and the rotated plant endpoint are cut until the second preset condition is met, and the cutting of the current initial plant endpoint and the rotated plant endpoint is completed. The initial plant endpoint set and the rotated plant endpoint set are traversed, and the cutting and judgment are repeated to obtain the initial skeleton line image. and rotated skeleton line images , wherein the first preset condition and the second preset condition are expressed as follows:

[0037] in, Represents the pixel value of a pixel, Represents the horizontal coordinate of the pixel point, Indicates the vertical coordinate of the pixel point, Indicates thickness data, represents the neighborhood range, Indicates the number of neighborhood pixels The horizontal coordinate of the pixel point, Indicates the number of neighborhood pixels The vertical coordinate of the pixel point, Indicates the change value of the vertical coordinate of the neighborhood pixel points.

[0038] Preset parallel straight lines, obtain the initial stem position and the rotated stem position by the straight line sliding intersection method, and then obtain the initial top surface plant image The plant stem coordinates in the initial top-view plant image refer to the stem data of the plant to be measured in the top-view plant image. That is, in the top-view image of the plant to be measured, if the stem is a cylinder, then what is seen in the top-view image is a circle. The plant stem coordinates are the coordinates of the circle. The horizontal coordinates and vertical coordinates of the circle are obtained by the initial stem position and the rotated stem position, respectively, and then the plant stem coordinates are obtained. The specific steps include: Step 1: Preset parallel lines , obtaining the coordinate data of the initial skeleton line image and the rotated skeleton line image, and obtaining the intersection points of the parallel straight lines with the initial skeleton line image and the rotated skeleton line image by calculation; Step 2: Slide the parallel lines from left to right to get The number of intersection points under different values ​​is used to obtain the first intersection point number set and the second intersection point number set, which are expressed as follows:

[0039] Step 3: Get the maximum value of the first intersection number set and the second intersection number set to get the maximum value of the first intersection and the second intersection maximum ; Step 4: Obtain a first parallel straight line corresponding to the maximum value of the first intersection point and a second parallel straight line corresponding to the maximum value of the second intersection point. The position of the first parallel straight line in the initial skeleton line image is the initial stem position, and the position of the second parallel straight line in the rotated skeleton line image is the rotated stem position. Step 5: Determine the plant stem coordinates in the initial top surface plant image using the initial stem position and the rotated stem position; in, represents a set of parallel lines, Represents a variable, Indicates the The number of first intersection points or second intersection points corresponding to the parallel straight lines, represents the Dirac function, Indicates the The horizontal coordinate of the pixel point, Indicates the number of pixels.

[0040] Extract the leaf mask image of the top leaf image, such as Figure 7 As shown, based on the leaf mask image, the leaf vertices in the top leaf image are obtained to form a leaf vertex coordinate set. Based on the obtained angle between the straight line where the leaf vertex coordinates and the plant stem coordinates are located and the plane where the side view image acquisition device is located, the following steps are included: Step 1: Based on the top leaf image , segment the leaves of each plant and extract the vertex coordinates of each independent plant leaf ; Step 2: Construct the leaf vector using the leaf vertex coordinates and the plant stem coordinates , leaf vector For the The vector pointing to the leaf vertex coordinates formed by the leaf and stem coordinates of a plant is expressed as follows:

[0041] Step 3: Assume the direction vector , calculate the first The angle between the leaf vector corresponding to a plant leaf and the imaging plane where the side view image acquisition device is located is specifically expressed as follows:

[0042]

[0043]

[0044] Step 4: Traverse the vertex coordinates of all plant leaves to obtain the angles between all plant leaves and the imaging plane where the side view image acquisition device is located; in, represents the direction vector, represents the leaf vector, Indicates the The horizontal coordinates of the vertex coordinates of the leaves, Indicates the The vertical coordinates of the leaf vertices, represents the horizontal coordinate of the plant stem, represents the vertical coordinate of the plant stem, represents the angle, represents the inverse tangent, represents sine, represents cosine, represents the number of leaf vertex coordinates, Indicates the number of leaves on the plant.

[0045] In this embodiment, a mapping relationship exists between the turntable control signal and the included angle, so that the included angle is converted into a turntable control signal that can be recognized by the rotating device, and the turntable control signal is sent to the rotating device, wherein the turntable control signal is represented as follows:

[0046] in, Indicates the turntable control signal, represents the mapping function, Indicates the angle.

[0047] After receiving the turntable control signal, the rotating device rotates according to the internal driving mechanism, so that the plant to be tested placed on the rotating device rotates accordingly. After the rotation, the leaf vector corresponding to the plant leaf should be parallel to the imaging plane of the side image acquisition device. The top view image acquisition device is used to collect images of the rotated plant to be tested, and the top surface plant image set is obtained. The top surface plant image set is projected onto the horizontal plane and the projection area is calculated to obtain the horizontal projection area. The coordinate projection mapping diagram is shown as follows: Figure 8 As shown, 1 represents a top view image acquisition device, 2 represents a side view image acquisition device, and 3 represents a rotation device. Based on the horizontal projection area and the measurement area, the plant canopy coverage is obtained, including the following steps: Step 1: In this embodiment, the camera is calibrated to obtain the intrinsic and extrinsic parameters of the image acquisition device. The intrinsic parameters are used to describe the internal parameters of the image acquisition device, and the extrinsic parameters are used to describe the relationship between the camera coordinate system and the world coordinate system. The corresponding relationship between the three-dimensional geometric position of the point on the plant to be measured and the two-dimensional geometric position of the corresponding point in the image is determined by the intrinsic and extrinsic parameters of the image acquisition device to obtain the mapping matrix ; Step 2: Obtain the plant leaf images in the top surface plant image set through the plant leaf detection model. The first leaf of the plant Points are mapped to the horizontal plane based on the mapping matrix to obtain the horizontal coordinate point set of the plant , plant horizontal coordinate point set and the coordinate points in the top view plant image and the actual blade thickness The relationship between them is expressed as follows:

[0048]

[0049] Step 3: The polygon vertex coordinate matrix formed by the plant horizontal coordinate point set is , the horizontal projection area of ​​the plant leaves is calculated based on the polygon vertex coordinate matrix using the shoelace formula, which is expressed as follows:

[0050] Step 4: Traverse the leaves of the plants and obtain the horizontal projection area of ​​each plant leaf. The total horizontal projection area is obtained by summing up, and the measurement area of ​​the entire measurement area is obtained. Combined with the total horizontal projection area, the plant canopy coverage is obtained. The total horizontal projection area and the plant canopy coverage are expressed as follows:

[0051]

[0052] in, Indicates the The top surface plant image The first leaf The horizontal coordinate of the plant point is the horizontal coordinate of the point, Indicates the The top surface plant image The first leaf The horizontal coordinates of the plant point are the vertical coordinates of the point, No. The top surface plant image The first leaf The horizontal coordinate of the pixel point, Indicates the The top surface plant image The first leaf The vertical coordinate of the pixel point, Indicates the The top surface plant image The actual blade thickness of the blade, Indicates the The horizontal projection area of ​​the blade, Indicates the The number of coordinate points of each leaf, represents the total horizontal projection area, Indicates the number of leaves, Indicates the area of ​​the measurement area, represents the plant canopy coverage, 、 、 Indicates the change proportional coefficient.

[0053] Example 2: A vision-based plant canopy coverage calculation system, such as Figure 2As shown, it includes an image acquisition module 100, a skeleton line extraction module 200, a stem coordinate acquisition module 300, an angle calculation module 400 and a canopy coverage calculation module 500; The image acquisition module 100 acquires an initial top surface plant image and an initial side surface plant image of the plant to be tested, horizontally rotates the plant to be tested to acquire a rotated side surface plant image, and extracts plant leaves to obtain a top surface leaf image, an initial side surface leaf image, and a rotated side surface leaf image; The skeleton line extraction module 200 obtains an initial side stem image and a rotated side stem image based on the initial side plant image, the initial side leaf image, the rotated side plant image and the rotated side leaf image, and obtains an initial skeleton line image and a rotated skeleton line image through skeleton line extraction; The stem coordinate acquisition module 300 constructs parallel lines, analyzes the intersections between the initial skeleton line image and the rotated skeleton line image and the parallel lines, obtains the initial stem position and the rotated stem position, and further obtains the plant stem coordinates in the initial top surface plant image; The angle calculation module 400 obtains a leaf vertex coordinate set from the top leaf image, forms a leaf vector set based on the plant stem coordinates and the leaf vertex coordinate set, and obtains the angle between the leaf vector and the imaging plane where the side view image acquisition device is located; The canopy coverage calculation module 500 rotates the plant to be measured based on the included angle. When the leaf vector of the corresponding plant leaf is parallel to the imaging plane where the side view image acquisition device is located, the top surface plant image of the corresponding plant leaf is obtained and projected onto the horizontal plane to obtain the total horizontal projection area, and then the plant canopy coverage is obtained.

[0054] Various changes and modifications can be made without departing from the spirit and scope of the present invention, and all equivalent technical solutions also fall within the scope of the present invention.

[0055] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referenced to each other.

[0056] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0057] The present invention is described with reference to the flowcharts and / or block diagrams of the method, terminal device (system), and computer program product according to the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the process in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0058] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0059] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0060] It should be noted that: References in this specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Therefore, appearances of the phrases "one embodiment" or "an embodiment" in various places throughout this specification do not necessarily refer to the same embodiment.

[0061] Furthermore, it should be noted that the specific embodiments described in this specification may vary in the shapes and names of their components. Any equivalent or simple variations based on the structure, features, and principles described in the patented concept of this invention are included within the scope of protection of this patent. Persons skilled in the art may make various modifications, additions, or substitutions to the described specific embodiments, and these modifications, as long as they do not deviate from the structure of the invention or exceed the scope defined by the claims, shall fall within the scope of protection of this invention.

Claims

1. A method for calculating plant canopy coverage based on vision, characterized in that: The following steps are involved: Acquire an initial top surface plant image and an initial side surface plant image of the plant to be tested, horizontally rotate the plant to be tested to obtain a rotated side surface plant image, and perform plant leaf extraction to obtain a top surface leaf image, an initial side surface leaf image, and a rotated side surface leaf image; Obtaining an initial side stem image and a rotated side stem image based on the initial side plant image, the initial side leaf image, the rotated side plant image, and the rotated side leaf image, and obtaining an initial skeleton line image and a rotated skeleton line image by skeleton line extraction; Constructing parallel straight lines, analyzing the intersections between the initial skeleton line image and the rotated skeleton line image and the parallel straight lines, obtaining the initial stem position and the rotated stem position, and then obtaining the plant stem coordinates in the initial top surface plant image; The leaf vertex coordinate set is obtained through the top leaf image, and the leaf vector set is formed based on the plant stem coordinates and the leaf vertex coordinate set, and the angle between the leaf vector and the imaging plane where the side view image acquisition device is located is obtained respectively; The plant to be tested is rotated based on the included angle. When the leaf vector of the corresponding plant leaf is parallel to the imaging plane where the side view image acquisition device is located, the top surface plant image of the corresponding plant leaf is obtained and projected onto the horizontal plane to obtain the total horizontal projection area, and then the plant canopy coverage rate is obtained.

2. The method for calculating plant canopy coverage based on vision according to claim 1, characterized in that: The plant leaf extraction comprises the following steps: Obtain a plant image set and annotate the plant leaf area to form a plant dataset, and divide the plant dataset into a training set and a validation set; The plant leaf detection pre-training model is trained, tested, and verified based on the training set and the validation set to obtain a plant leaf detection model; The initial side plant image, the rotated side plant image and the initial top plant image are detected and analyzed through the plant leaf detection model to obtain the initial side leaf area, the rotated side leaf area and the top leaf area. The initial side plant image, the rotated side plant image and the initial top plant image are combined to obtain the initial side leaf image, the rotated side leaf image and the top leaf image.

3. The method for calculating plant canopy coverage based on vision according to claim 1, characterized in that: The method of obtaining the initial side stem image and the rotated side stem image based on the initial side plant image, the initial side leaf image, the rotated side plant image and the rotated side leaf image comprises the following steps: Binarize the initial side plant image and the initial side leaf image, and obtain the initial side stem image by image subtraction; The rotating side plant image and the rotating side leaf image are binarized and subtracted to obtain the rotating side stem image.

4. The method for calculating plant canopy coverage based on vision according to claim 1, characterized in that: The process of obtaining an initial skeleton line image and a rotated skeleton line image by skeleton line extraction includes the following steps: Extracting non-zero pixel points in the initial side stem image and the rotated side stem image to obtain an initial side point set and a rotated side point set; Obtaining the initial neighborhood pixel point set and the rotation neighborhood pixel point set of the initial side point set and the rotation side point set respectively, presetting a pixel threshold, and if the number of pixels in the initial pixel point set or the rotation pixel point set connected to the current initial side point or the current rotation side point meets the pixel threshold, then the current initial side point or the current rotation side point is the plant endpoint; The initial side point set and the rotated side point set are traversed to obtain the initial plant endpoint set and the rotated plant endpoint set, and the corresponding thickness data are obtained. The thickness data is judged and cut according to the first preset condition and the second preset condition to obtain the initial skeleton line image and the rotated skeleton line image.

5. The method for calculating plant canopy coverage based on vision according to claim 4, characterized in that: The method of obtaining the corresponding thickness data, judging and cutting the thickness data according to the first preset condition and the second preset condition, and obtaining the initial skeleton line image and the rotated skeleton line image includes the following steps: The thickness data at the initial plant endpoint and the rotation plant endpoint are obtained respectively, as shown below: If the thickness data satisfies a preset threshold and the neighborhood pixel values ​​and thickness data of the initial plant endpoint and the rotated plant endpoint meet a first preset condition, the initial plant endpoint and the rotated plant endpoint are cut until a second preset condition is met, thereby obtaining an initial skeleton line image and a rotated skeleton line image; in, Represents the pixel value of a pixel, Indicates thickness data, represents the neighborhood range, Indicates the number of neighborhood pixels The horizontal coordinate of the pixel point, Indicates the number of neighborhood pixels The vertical coordinate of the pixel point, Indicates the change value of the vertical coordinate of the neighborhood pixel points.

6. The method for calculating plant canopy coverage based on vision according to claim 1, characterized in that: The intersection points between the initial skeleton line image and the rotated skeleton line image and the parallel straight lines are analyzed to obtain the initial stem position and the rotated stem position. The following steps are involved: Preset parallel lines, obtain the first and second intersection points of the parallel lines with the initial skeleton line image and the rotated skeleton line image, expressed as follows: Obtaining a first set of intersection points and a second set of intersection points by sliding parallel straight lines, and obtaining a maximum value of the first intersection point and a maximum value of the second intersection point; Based on the first parallel straight line and the second parallel straight line corresponding to the first intersection maximum value and the second intersection maximum value respectively, the positions of the first parallel straight line and the second parallel straight line in the initial skeleton line image and the rotated skeleton line image are obtained to obtain the initial stem position and the rotated stem position; in, represents parallel straight lines, Represents a variable, Indicates the The number of first intersection points or second intersection points corresponding to the parallel straight lines, represents the Dirac function, Indicates the The horizontal coordinate of the pixel point, Indicates the number of pixels.

7. The method for calculating plant canopy coverage based on vision according to claim 1, characterized in that: The method of obtaining the angle between the blade vector and the imaging plane where the side view image acquisition device is located comprises the following steps: Based on the leaf vertex coordinate set and the plant stem coordinates, a leaf vector set is constructed, wherein the direction of the leaf vector points to the leaf vertex coordinates; Set the direction vector, obtain the angle between the blade vector and the direction vector, and obtain the angle between the blade vector and the imaging plane where the side view image acquisition device is located, which is expressed as follows: in, , represents the direction vector, represents the leaf vector, Indicates the The horizontal coordinates of the vertex coordinates of the leaves, Indicates the The vertical coordinates of the leaf vertices, represents the horizontal coordinate of the plant stem, represents the vertical coordinate of the plant stem, represents the angle, represents the inverse tangent, represents cosine, represents the number of leaf vertex coordinates, Indicates the number of leaves on the plant.

8. The method for calculating plant canopy coverage based on vision according to claim 1, characterized in that: The plant canopy coverage is obtained by the following steps: Obtain the mapping matrix, combine it with the actual leaf thickness, project the top surface plant image onto the horizontal plane, and obtain the plant horizontal coordinate point set, which is expressed as follows: The horizontal projection area of ​​each plant leaf in the top surface plant image is obtained by using the plant horizontal coordinate point set and the shoelace formula, which is expressed as follows: Based on the horizontal projection area of ​​each plant's leaves, the total horizontal projection area was obtained, and combined with the measurement area for analysis, the plant canopy coverage was obtained; in, Indicates the The top surface plant image The first leaf of the plant The horizontal coordinate of the plant point is the horizontal coordinate of the point, Indicates the The top surface plant image The first leaf of the plant The horizontal coordinate of the plant point is the vertical coordinate of the point, No. The top surface plant image The first leaf of the plant The horizontal coordinate of the pixel point, Indicates the The top surface plant image The first leaf of the plant The vertical coordinate of the pixel point, Indicates the The top surface plant image The actual leaf thickness of the plant leaves, Indicates the The horizontal projection area of ​​the leaves of a plant, Indicates the The number of coordinate points of a plant leaf, 、 、 Indicates the change proportional coefficient.

9. The method for calculating plant canopy coverage based on vision according to claim 1, characterized in that: The method further includes: the initial top surface plant image is acquired by a top view image acquisition device located at the top of the plant to be measured; the initial side surface plant image is acquired by a side view image acquisition device located at the side of the plant to be measured; and the rotated side surface plant image is acquired by rotating the plant to be measured horizontally by 90° and then acquired by a side view image acquisition device located at the side of the plant to be measured; The imaging plane of the top-view image acquisition device is parallel to the upper surface of the plant, and the imaging center of the top-view image acquisition device and the center of the plant are located in a straight line; The imaging plane of the side view image acquisition device is perpendicular to the upper surface of the plant, and the imaging center of the side view image acquisition device, the imaging center of the top view image acquisition device and the center of the plant are located in the same plane.

10. A vision-based plant canopy coverage calculation system, characterized in that: It includes image acquisition module, skeleton line extraction module, stem coordinate acquisition module, angle calculation module and canopy coverage calculation module; The image acquisition module acquires an initial top surface plant image and an initial side surface plant image of the plant to be tested, horizontally rotates the plant to be tested to acquire a rotated side surface plant image, and extracts plant leaves to obtain a top surface leaf image, an initial side surface leaf image, and a rotated side surface leaf image; The skeleton line extraction module obtains the initial side stem image and the rotated side stem image based on the initial side plant image, the initial side leaf image, the rotated side plant image and the rotated side leaf image, and obtains the initial skeleton line image and the rotated skeleton line image through skeleton line extraction; The stem coordinate acquisition module constructs parallel straight lines, analyzes the intersections between the initial skeleton line image and the rotated skeleton line image and the parallel straight lines, obtains the initial stem position and the rotated stem position, and further obtains the plant stem coordinates in the initial top surface plant image; The angle calculation module obtains a leaf vertex coordinate set from the top leaf image, forms a leaf vector set based on the plant stem coordinates and the leaf vertex coordinate set, and obtains the angle between the leaf vector and the imaging plane where the side view image acquisition device is located; The canopy coverage calculation module rotates the plant to be measured based on the included angle. When the leaf vector of the corresponding plant leaf is parallel to the imaging plane where the side view image acquisition device is located, the top surface plant image of the corresponding plant leaf is obtained and projected onto the horizontal plane to obtain the total horizontal projection area, and then the plant canopy coverage is obtained.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

12. A device for calculating plant canopy coverage based on vision, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

Images