A plant leaf inclination angle measurement method based on binocular vision

Abstract

The application discloses a kind of plant leaf inclination angle measurement methods based on binocular vision, comprising: using binocular vision system to shoot scene, and carry out parallax estimation, utilize parallax value, calculate the space slope between each pixel point and adjacent pixel point, according to the jump in the radian, direction between plane, realize image segmentation, extract leaf tip and leaf tail point to the image segmented, according to leaf tip and leaf tail coordinates, obtain the angle of leaf and lens, to the collection and calculation of angle information is obtained.The application can greatly improve the collection efficiency of plant leaf inclination angle parameter, shorten working time, reduce work difficulty, improve the measurement precision of leaf inclination angle data.

Description

Technical Field

[0001] This invention relates to the field of measurement technology, and more specifically to a method for measuring the tilt angle of plant leaves based on binocular vision. Background Technology

[0002] Artificial intelligence and big data have accelerated the development of smart agriculture, and plant phenotyping, as an important subfield, has gradually become a research hotspot. Plant phenotyping is a science that reflects all the physical, physiological, and biochemical characteristics and traits of plants, determined or influenced by genetic and environmental factors, including their structure and composition, growth and development processes and outcomes. By studying plant phenotyping, we can more clearly study the plant growth process, more accurately grasp the development of plant diseases, and more precisely monitor the effects of certain pesticides. Research on plant phenotyping is of great help in organic pesticide research, plant disease and pest classification, and monitoring of plant growth processes.

[0003] In existing plant phenotyping research, most plant phenotyping data measurements suffer from problems such as tedious work, high workload, and low accuracy. Leaf inclination angle, as a fundamental parameter, is a crucial piece of data in plant phenotyping, providing fundamental support for precision agriculture. Leaf inclination angle is an important basis for plant classification, for assessing photosynthetic efficiency, and for calculating pesticide residues. By understanding the leaf inclination angle, we can calculate how much pesticide is theoretically absorbed by the plant and how much is wasted due to gravity flowing into the soil when sprayed at a specific angle. Precise measurement of plant leaf inclination angle allows for clearer identification of plant species, more accurate calculation of theoretical carbon sequestration, and more precise control of pesticide application. Therefore, measuring leaf inclination angle is a fundamental task in precision agriculture. Currently, commonly used instruments for measuring plant leaf inclination angle include ordinary monocular visible light cameras, DHPs (Digital High-Performance Cameras), LiDAR, and binocular visible light cameras.

[0004] Ordinary monocular visible light cameras are used to acquire leaf tilt angles. While the equipment is inexpensive, the workload and time cost are enormous because only one leaf's tilt angle can be captured at a time. During shooting, the camera angle needs careful adjustment to make the camera's imaging plane as parallel to the leaf as possible. The camera's acquisition angle must be recorded simultaneously with the photograph, and the leaf tilt angle parameters are inferred from the angle between the camera and the leaf. This method results in low accuracy and a massive workload.

[0005] The DHPs type of measurement method mainly uses a fisheye lens with a field of view close to or equal to 180 degrees to capture leaf information. This type of method cannot measure the tilt angle of a single leaf, but can only output the average tilt angle of multiple leaves.

[0006] LiDAR primarily uses lidar for data acquisition. By calculating the time it takes for each photon to travel back after emission, the depth of each point in space can be determined. By collecting data around the sampled object, removing interference, and then aggregating the data, the three-dimensional information of the sampled object can be reconstructed, thus accurately obtaining the leaf tilt angle of each leaf. This method offers high accuracy, but LiDAR equipment is extremely expensive, depth information aggregation is time-consuming, and data acquisition is easily affected by environmental interference, making it difficult to promote widely.

[0007] All three methods mentioned above suffer from problems such as high testing difficulty, long testing time, huge data collection workload, and cumbersome and difficult-to-operate data collection process. To address this, Chinese Patent Publication No. CN109146948B, published on July 30, 2021, discloses a vision-based method for quantifying crop growth phenotypic parameters and analyzing their correlation with yield. This method uses a binocular camera to measure leaf tilt angle. Based on the principle of parallax ranging, it converts information in the image into spatial point cloud information with depth. Then, referencing LiDAR-like methods, it uses filtering and aggregation techniques on the measured point cloud information to reconstruct the three-dimensional information of the sampled object. However, the filtering and aggregation processing of the point cloud is computationally intensive, resulting in high computational costs and long testing time. Furthermore, this scheme is based on RGB color segmentation of pixels, making it difficult to handle densely packed leaves. Summary of the Invention

[0008] The purpose of this invention is to overcome the above-mentioned problems and provide a binocular vision-based method for measuring plant leaf inclination angle, which can greatly improve the efficiency of collecting plant leaf inclination angle parameters, shorten working time, reduce working difficulty, and improve the measurement accuracy of leaf inclination angle data.

[0009] The present invention provides a method for measuring the tilt angle of plant leaves based on binocular vision, comprising the following steps:

[0010] (1) Use a binocular vision system to capture the scene and perform disparity estimation.

[0011] Using the checkerboard method, take 2-20 photos of a checkerboard pattern and find the corner points; use an algorithm encapsulated in the open-source library OpenCV to estimate the corner points, and after correction, obtain the camera's intrinsic parameters as shown below:

[0012]

[0013] Among them, f x with f y Let d represent the focal length of the camera in the x and y directions, respectively. x With d y This represents the number of pixels that need to be moved horizontally to the left and vertically upward to convert the Cartesian coordinate system to the image coordinate system.

[0014] Epipolar correction mathematically aligns the camera to the same observation plane, ensuring that the pixel rows on the camera are strictly aligned. It finds a rotation matrix such that after multiplying the left and right cameras by this matrix, the xy coordinate systems of the two camera coordinate systems lie on the same plane, with the x-axis on the same straight line and the y-axis parallel to each other.

[0015] During the shooting process, some distortion and loss of image may occur. The image is corrected by using the camera's intrinsic parameters. The image is transformed by spatial coordinates using the parameter values ​​obtained from camera calibration. First, a mapping relationship is established between the image coordinates and the corresponding point coordinates of the reference image. Then, the coordinates of each pixel are corrected according to the mapping relationship. Finally, the gray value of each pixel after correction is determined.

[0016] The disparity estimation algorithm for binocular vision takes two images as input and outputs a disparity map. In the disparity map, each pixel stores the disparity value between the two related images.

[0017] (2) Using the disparity value, calculate the spatial slope between each pixel and its neighboring pixels.

[0018] The depth corresponding to each pixel can be calculated using the following formula:

[0019]

[0020] Calculate the depth of the spatial point corresponding to each pixel, where b is the baseline length between the optical centers of the two cameras, d is the disparity value between the two matched points, and f x Z(x) represents the focal length of the camera along the x-axis, and Z(x) represents the spatial depth value of the corresponding pixel.

[0021] Use formula Estimate the depth of the corresponding point in space;

[0022] The xyz 3D coordinate system can be viewed as a combination of the xz and y coordinate systems. By slicing the 3D coordinate system according to the pixel column values ​​of x and y respectively, the 3D problem is decomposed into the relationship between z and x when y is a fixed value, and the relationship between z and y when x is a constant value. The spatial slope between adjacent pixels can be calculated using the following formula:

[0023]

[0024] Where tanα is the slope of adjacent spatial points in the image coordinate system along the x-direction, and tanβ is the slope of adjacent spatial points in the image coordinate system along the y-direction. z1 and z2 represent the depth information of two spatial points, x1 and x2 represent the x-coordinate information of two adjacent points along the x-axis, and y1 and y2 represent the y-coordinate information of two adjacent points along the y-axis.

[0025] Using the disparity between two spatial points and the pixel positions of the two spatial points, the tangent angle between the corresponding imaging planes of the two points is calculated as follows:

[0026]

[0027] Where, d A This represents the disparity information stored at the corresponding point on the reference plane for point A in space, x. A f represents the x-axis pixel position information of point A in the camera coordinate system. x Let be the focal length of the two cameras along the x-axis.

[0028] Iterate through all integers x and y, calculate the gradient relationship between z and independent variable y and the gradient relationship between z and independent variable x, and store them in two grayscale images with sizes of (x-1)*y and x*(y-1), respectively. The grayscale values ​​represent the tangent of the angles formed with respect to the imaging plane in the xz coordinate system and the yz coordinate system, respectively.

[0029] Calculate the angle between each pair of adjacent pixels in the gradient map, and use the formula arctan A=α to calculate the change of angle in the same coordinate axis direction, so as to obtain the angle between two adjacent pixels and the imaging plane, which can be applied to image segmentation.

[0030] (3) Achieve image segmentation based on the abrupt changes in curvature and direction between planes.

[0031] Using the x-axis and y-axis as the measurement standard, the angle images corresponding to the x-axis and y-axis are traversed to achieve image segmentation;

[0032] A threshold φ is set for the angle change, and it is determined that when |α A -α B When |≤φ, pixels with two related angles along the x-axis are on the same plane; when |α... A -α B When |>φ, the pixels with two related angles along the x-axis are not on different planes;

[0033] The threshold φ is used to determine whether the connection is along the y-axis. When |β A -β B When |≤φ, two related pixels along the y-axis are in the same plane; when |β...A -β B When |>φ, the two pixels associated in the y-axis direction are in the same plane;

[0034] By calculating the angle change of adjacent pixels, when the angle change of adjacent pixels is less than a threshold, the two pixels are on the same plane; when the angle change of adjacent pixels is greater than the threshold, the two pixels are not on the same plane, and segmentation is performed.

[0035] Iterate through the gradient map segmented in the x-direction, divide each row of pixels into different regions, and use a counter to mark each connected region with the same number;

[0036] When the angular jump in the horizontal direction exceeds the threshold and the number of pixels with the angular jump is greater than the interference threshold, the two planes are not smooth and need to be segmented; if the angular jump in the horizontal direction does not exceed the threshold, or the number of pixels with the angular jump exceeds the threshold is less than the interference threshold, the pixels adjacent to the above pixels are still on the same plane.

[0037] The image is segmented by traversing the x-direction and the y-direction, merging connected pixel blocks.

[0038] Traverse the y-direction and, based on the specified threshold, determine whether adjacent gradients in the y-direction represent a sudden change in the gradient of the leaf; if no sudden change occurs, the pixels of the two labels are connected blocks and are in the same plane, and merge the two pixel blocks.

[0039] Through segmentation and merging, the entire image was segmented based on depth and planar information;

[0040] (4) Extract the leaf tip and leaf tail points from the segmented image.

[0041] Set a threshold to separate the background pixels from the leaf segmentation information;

[0042] In step (3), the image has been segmented, and the pixels in the two-dimensional projection image have been classified. In the current step, the pixel depth calculated in step (1) is used. A threshold is set to filter the pixels and discard the segmented regions with a depth value greater than the threshold.

[0043] For each segmented region, perform image binarization; set the gray level of pixels in each category to 0, and set the gray level of pixels outside the category to 255. Each leaf has a corresponding binarized image.

[0044] Morphological transformation is performed on each binarized image to extract the leaf skeleton from two dimensions; the skeleton is the leaf vein, which connects the leaf tip and leaf tail; the starting and ending pixels of the curve in the skeleton structure are selected, and the spatial points corresponding to these pixels are the leaf tip and leaf tail coordinates of each leaf.

[0045] (5) Calculate the angle between the blade and the lens based on the coordinates of the blade tip and the blade tip;

[0046] Connecting the leaf tip and leaf tail, their projections onto the imaging plane are shown as AB, where d A This represents the disparity information stored at the corresponding point on the reference plane for point A in space, x. A f represents the x-axis pixel position information of point A in the camera coordinate system. x Let be the focal length of the two cameras along the x-axis.

[0047] Through formula

[0048]

[0049] Calculate the slopes of adjacent spatial points in the x and y directions in the image coordinate system; calculate the changes in angles along the same coordinate axis using the formula arctan A=α.

[0050] Knowing the positions and disparity values ​​of two points in the image coordinate system, we can obtain the angle information of the corresponding points.

[0051] The angle between the line connecting the tip and the tip of the leaf and the line projected onto the imaging plane is the angle between the leaf and the lens.

[0052] Calculate the angle between any two spatial points and the projection plane using the following formula:

[0053]

[0054] Where, d A This represents the disparity information stored at the corresponding point on the reference plane for point A in space, x. A f represents the x-axis pixel position information of point A in the camera coordinate system. x Let be the focal length of the two cameras along the x-axis.

[0055] Substitute the pixel positions corresponding to the leaf tip and leaf tail into the equation to calculate the angle between the corresponding leaf blade and the imaging plane.

[0056] (6) Collection and calculation of included angle information

[0057] The blade tilt angle can be calculated from the angles between the blade and the imaging plane, and between the imaging plane and the ground. The formula is as follows:

[0058] ∠α-∠β=∠φ

[0059] Where ∠α is the angle between the blade and the imaging plane, ∠β is the angle between the imaging plane and the ground plane, and ∠φ is the angle between the blade and the ground plane, which is the blade tilt angle we are looking for.

[0060] The above-mentioned method for measuring the tilt angle of plant leaves based on binocular vision, wherein the movable angle leaf tilt angle measurement method is as follows: the angle between the fixed rod and the imaging plane is α, and the fixed rod is perpendicular to the ground plane, then the angle β between the imaging plane and the ground can be calculated according to the geometric formula β = 90° - α;

[0061] The mobile device collects information about the blades, using a gyroscope to collect the angle between the mobile device and the ground;

[0062] When collecting leaf tilt angle information, the binocular vision system used for acquisition can move and rotate flexibly in space. Combining the angle between the leaf and the lens collected by the binocular vision system and the angle between the imaging plane and the ground collected by the gyroscope sensor, a detection algorithm is used to track all leaves in the field of view: each leaf that newly appears in the lens is marked with a new number; the number of the leaves that remain in the imaging plane remains unchanged.

[0063] Using streaming data, information transmitted from the gyroscope sensor is recorded, and the acquisition time of each piece of information is recorded. The current device attitude is matched with the corresponding video information. The video information extraction time is taken evenly, and the blade tilt angle state and data extraction number for each number are recorded. The blade tilt angle values ​​collected each time are added together, divided by the number of data extractions, and the average value is calculated. The calculated average value is the blade tilt angle that is closer to the true value.

[0064] Compared with existing technologies, this invention has significant advantages. As can be seen from the above technical solution: this invention uses a drone to improve data acquisition efficiency; a binocular camera is used to predict the depth of a portion of objects within the photographic range; and a gyroscope sensor is used to measure the angles between the drone / camera and the horizontal plane, thereby calculating the angles between the gyroscope and the binocular camera, and between the gyroscope and the horizontal plane. This combination enables batch acquisition and processing of leaf tilt angles. This invention transforms plant leaf tilt angle measurement into stereoscopic vision reconstruction, using a lower-cost RGB camera instead of an expensive radar, thus reducing the cost of leaf tilt angle measurement. Furthermore, because the method supports the acquisition of information from all leaves within the camera's field of view, this invention supports the simultaneous acquisition of leaf tilt angle information from multiple leaves, greatly saving time, reducing the complexity of leaf tilt angle information acquisition, decreasing workload, and providing more accurate leaf tilt angle measurement results. Attached Figure Description

[0065] Figure 1This is a flowchart of the vision-based leaf tilt angle acquisition method of the present invention;

[0066] Figure 2 This is a schematic diagram illustrating the calculation of the spatial tilt angle between two points in space according to the present invention.

[0067] Figure 3 This is a schematic diagram illustrating the calculation of the blade tilt angle based on the angle between the imaging plane and the ground plane and the blade angle according to the present invention;

[0068] Figure 4 This is a schematic diagram of the process of collecting and estimating plant leaf tilt angle using a mobile device according to the present invention;

[0069] Figure 5 This is a schematic diagram showing the angle between the imaging plane and the rigid body for measuring the blade tilt angle using a rigid body, as described in this invention. Detailed Implementation

[0070] Example 1:

[0071] like Figure 1 As shown, the present invention provides a method for measuring plant leaf tilt angle based on a binocular camera (a fixed-angle leaf tilt angle measurement method), which includes the following steps:

[0072] (1) Use a binocular vision system to capture the scene and perform parallax estimation.

[0073] In imaging systems, commonly used coordinate systems include: image coordinate system, camera coordinate system, and world coordinate system. The image coordinate system is a two-dimensional coordinate system with x and y axes, the top-left corner of the image as the origin, and a unit scale of pixels. Each camera has a corresponding image coordinate system. The camera coordinate system uses the horizontal axis of the imaging plane as the x-axis, the vertical axis as the y-axis, and the spatial depth of the image point from the camera as the z-axis. Each camera also has a corresponding camera coordinate system.

[0074] For each 3D reconstruction task, there is one and only one corresponding world coordinate system, which is used to unify information from different coordinate systems in the same spatial reference standard.

[0075] Binocular ranging technology is based on the principle of parallax in human vision. By obtaining two images from different positions along the same baseline, it can estimate the shape and distance of an object, much like the human eye. Furthermore, the principle of similar triangles allows for the acquisition of three-dimensional structural information about the target.

[0076] Camera calibration technology is essential because any theoretical physical model is an approximation of the real thing based on specific assumptions. However, errors exist in practical applications, and the imaging model of a regular camera is no exception (perspective projection).

[0077] Using the checkerboard method, take 2-20 photos of a checkerboard pattern and find the corner points; use an algorithm encapsulated in the open-source library OpenCV to estimate the corner points, and after correction, obtain the camera's intrinsic parameters as shown below:

[0078]

[0079] Among them, f x with f y Let d represent the focal length of the camera in the x and y directions, respectively. x With d y This represents the number of pixels that need to be moved horizontally to the left and vertically upward to convert the Cartesian coordinate system to the image coordinate system.

[0080] Epipolar correction mathematically aligns the camera to the same observation plane, ensuring that the pixel rows on the camera are strictly aligned. It finds a rotation matrix such that after multiplying the left and right cameras by this matrix, the xy coordinate systems of the two camera coordinate systems lie on the same plane, with the x-axis on the same straight line and the y-axis parallel to each other.

[0081] During the shooting process, some distortion and loss of image quality occurred. The images were corrected using the camera's internal parameters.

[0082] Using the parameter values ​​obtained from camera calibration, the image is transformed into spatial coordinates. First, a mapping relationship is established between the image coordinates and the corresponding point coordinates of the reference image. Then, the coordinates of each pixel are corrected according to the mapping relationship, and finally, the gray value of each pixel after correction is determined.

[0083] Disparity estimation in binocular vision includes four steps: matching cost calculation, cost aggregation, disparity calculation, and disparity optimization, ultimately outputting a disparity map.

[0084] The disparity estimation algorithm for binocular vision takes two images as input and outputs a disparity map. In the disparity map, each pixel stores the disparity value between the two related images. The estimated disparity is preserved and used for subsequent segmentation and leaf tilt angle calculation.

[0085] (2) Using disparity information, calculate the spatial slope information between each pixel and its neighboring pixels.

[0086] The depth information corresponding to each pixel can be calculated using the following formula:

[0087]

[0088] Calculate the depth information of the spatial point corresponding to each pixel, where b is the baseline length between the optical centers of the two cameras, d is the disparity value between the two matched points, and f xZ(x) represents the focal length of the camera in the x-axis direction, Z(x) represents the spatial depth value of the corresponding pixel, and the estimated parallax is preserved for subsequent segmentation and leaf tilt angle calculation.

[0089] Use formula Estimate the depth of the corresponding point in space;

[0090] The xy coordinate system corresponds to the imaging plane, while the reconstructed 3D points are discretely distributed in the xyz 3D coordinate system. Therefore, in the analysis, the xyz 3D coordinate system is considered as a combination of the xz and y coordinate systems. The 3D coordinate system is sliced ​​according to the pixel column values ​​of x and y respectively. The 3D problem is decomposed into the relationship between z and x when y is a fixed value, and the relationship between z and y when x is a constant value. The spatial slope between adjacent pixels can be calculated using the following formula to evaluate the relationship between adjacent pixels in the x and y axes:

[0091]

[0092] Where tanα is the slope of adjacent spatial points in the image coordinate system along the x-direction, and tanβ is the slope of adjacent spatial points in the image coordinate system along the y-direction. z1 and z2 represent the depth information of two spatial points, x1 and x2 represent the x-coordinate information of two adjacent points along the x-axis, and y1 and y2 represent the y-coordinate information of two adjacent points along the y-axis.

[0093] Using the disparity between two spatial points and the pixel positions of the two spatial points, the tangent angle between the corresponding imaging planes of the two points is calculated as follows:

[0094]

[0095] Where, d A This represents the disparity information stored at the corresponding point on the reference plane for point A in space, x. A f represents the x-axis pixel position information of point A in the camera coordinate system. x Let be the focal length of the two cameras along the x-axis.

[0096] Iterate through all integers x and y, calculate the gradient relationship between z and independent variable y and the gradient relationship between z and independent variable x, and store them in two grayscale images with sizes of (x-1)*y and x*(y-1), respectively. The grayscale values ​​represent the tangent of the angles formed with respect to the imaging plane in the xz coordinate system and the yz coordinate system, respectively.

[0097] When performing segmentation tasks, the tangent of the included angle is not intuitive enough and is not linearly related to the pixel angle difference. Therefore, the included angle between each pair of adjacent pixels in the gradient map is calculated, and the change of angle in the same coordinate axis direction is calculated using the formula arctan A=α, so as to obtain the included angle between two adjacent pixels and the imaging plane, which is then applied to image segmentation.

[0098] (3) Achieve image segmentation based on the abrupt changes in curvature and direction between planes.

[0099] Since the image coordinate system and the camera coordinate system coincide in the xy plane, and the x-axis and y-axis coincide, the x-axis and y-axis are used as the measurement standard to traverse the angle images corresponding to the x-axis and y-axis to achieve image segmentation.

[0100] When two different leaf planes intersect, a sudden angle change occurs; therefore, a threshold φ is set for the angle change, and it is determined that when |α... A -α B When |≤φ, pixels with two angles related along the x-axis are on the same plane; when |α1-α2|>φ, pixels with two angles related along the x-axis are on different planes.

[0101] The threshold φ is used to determine whether the connection is along the y-axis. When |β A -β B When |≤φ, two related pixels along the y-axis are in the same plane; when |β... A -β B When |>φ, the two pixels associated in the y-axis direction are not in the same plane;

[0102] By calculating the angle change of adjacent pixels, when the angle change of adjacent pixels is less than a threshold, the two pixels are on the same plane; when the angle change of adjacent pixels is greater than the threshold, the two pixels are not on the same plane, and segmentation is performed.

[0103] Iterate through the gradient map segmented in the x-direction, divide each row of pixels into different regions, and use a counter to mark each connected region with the same number;

[0104] When taking photos, there is some noise interference in the image, which will cause some points to be unable to match during the depth estimation process;

[0105] During the segmentation process, some pixels may be affected by noise, which may cause "mismatch" of matching points, resulting in differences in the estimated depth values.

[0106] Therefore, a threshold is set for interference information. When the angle jump in the horizontal direction exceeds the threshold and the number of pixels with angle jump is greater than the interference threshold, the two planes are not smooth and need to be segmented. If the angle jump in the horizontal direction does not exceed the threshold, or the number of pixels with angle jump exceeds the threshold is less than the interference threshold, the pixels adjacent to the above pixels are still on the same plane.

[0107] The image is segmented by traversing the x-direction, but the connected pixel blocks in the vertical direction appear as strips. Therefore, it is necessary to traverse the y-direction to merge the connected pixel blocks and segment the entire image.

[0108] Traverse the y-direction and, based on the specified threshold, determine whether adjacent gradients in the y-direction represent a sudden change in the gradient of the leaf; if no sudden change occurs, the pixels of the two labels are connected blocks and are in the same plane, and merge the two pixel blocks.

[0109] Through segmentation and merging, the entire image was segmented based on depth and planar information;

[0110] (4) Extract the leaf tip and leaf tail points from the segmented image.

[0111] To accurately extract the leaf tip and leaf tail, a threshold is set to separate the background pixels from the leaf segmentation information.

[0112] In step (3), the image has been segmented, and the pixels in the two-dimensional projection image have been classified. In the current step, the pixel depth calculated in step (1) is used. A threshold is set to filter the pixels and discard the segmented regions with a depth value greater than the threshold.

[0113] For each segmented region, perform image binarization; set the gray level of pixels in each category to 0, and set the gray level of pixels outside the category to 255. Each leaf has a corresponding binarized image.

[0114] Morphological transformation is performed on each binarized image to extract the leaf skeleton from two dimensions; the skeleton is the leaf vein, which connects the leaf tip and leaf tail; the starting and ending pixels of the curve in the skeleton structure are selected, and the spatial points corresponding to these pixels are the leaf tip and leaf tail coordinates of each leaf.

[0115] (5) Calculate the angle between the blade and the lens based on the coordinates of the blade tip and the blade tip;

[0116] Connecting the leaf tip and leaf tail, their projections on the imaging plane (e.g.) Figure 2 As shown in the figure, AB is an example of d. A This represents the disparity information stored at the corresponding point on the reference plane for point A in space, x. Af represents the x-axis pixel position information of point A in the camera coordinate system. x Let be the focal length of the two cameras along the x-axis.

[0117] Through formula

[0118]

[0119] Calculate the slopes of adjacent spatial points in the x and y directions in the image coordinate system; calculate the changes in angles along the same coordinate axis using the formula arctan A=α.

[0120] Knowing the positions and disparity values ​​of two points in the image coordinate system, we can obtain the angle information of the corresponding points.

[0121] The angle between the line connecting the tip and the tip of the leaf and the line projected onto the imaging plane is the angle between the leaf and the lens.

[0122] The angle between any two spatial points and the projection plane can be calculated using the following formula:

[0123]

[0124] Where, d A This represents the disparity information stored at the corresponding point on the reference plane for point A in space, x. A f represents the x-axis pixel position information of point A in the camera coordinate system. x Let be the focal length of the two cameras along the x-axis.

[0125] By substituting the pixel positions corresponding to the leaf tip and leaf tail, the angle between the corresponding leaf and the imaging plane can be calculated.

[0126] (6) Collection and calculation of included angle information

[0127] like Figure 3 As shown, the blade tilt angle can be calculated based on the angle between the blade and the imaging plane, and the angle between the imaging plane and the ground. The calculation formula is:

[0128] ∠α-∠β=∠φ

[0129] Where ∠α is the angle between the blade and the imaging plane, ∠β is the angle between the imaging plane and the ground plane, and ∠φ is the angle between the blade and the ground plane, which is the blade tilt angle we are looking for.

[0130] Example 2:

[0131] like Figure 1 As shown, the present invention provides a method for measuring plant leaf tilt angle based on a binocular camera (a leaf tilt angle measurement method with movable angle), which includes the following steps:

[0132] The preceding steps are the same as in Example 1;

[0133] like Figure 5 As shown, the angle between the fixed rod and the imaging plane is α, and the fixed rod is perpendicular to the ground plane. Therefore, the angle β between the imaging plane and the ground can be calculated using the geometric formula β = 90° - α.

[0134] The mobile device collects information from the blades, using a gyroscope to measure the angle between the mobile device and the ground.

[0135] There are many types of gyroscopes. According to their uses, they can be divided into sensing gyroscopes and indicating gyroscopes. Sensing gyroscopes are used in automatic control systems for the motion of aircraft as sensors for horizontal, vertical, pitch, heading, and angular velocity.

[0136] The principle of a gyroscope is that the direction pointed to by the axis of rotation of a rotating object will not change when it is not affected by external forces; based on this principle, people use it to maintain direction; and then use various methods to read the direction indicated by the axis and automatically transmit the data signal to the control system.

[0137] Use a mobile device to capture images of the blades within the field of view.

[0138] Because the equipment is in motion during the data collection process, using cameras to collect and transmit more information from the leaves results in better data collection quality.

[0139] However, during the movement, the same leaf may appear in the image multiple times. If each frame of data is analyzed, a large amount of data redundancy will be generated.

[0140] By incorporating a gyroscope sensor, leaf tilt angle information can be measured more flexibly. When acquiring this information, the binocular vision system can move and rotate flexibly in space. Combining the angle between the leaf and the lens acquired by the binocular vision system with the angle between the imaging plane and the ground acquired by the gyroscope sensor, the leaf tilt angle information can be calculated more flexibly using the formula ∠α - ∠β = ∠φ. Therefore, a detection algorithm is used to track all leaves within the field of view: each newly appearing leaf in the lens is assigned a new number; leaves remaining in the imaging plane retain their original numbers.

[0141] For the same leaf, data will be collected several times during the movement of the mobile device;

[0142] Therefore, by using streaming data, the information transmitted back from the gyroscope sensor is recorded, and the acquisition time of each piece of information is recorded, and the current device posture is matched with the corresponding video information.

[0143] like Figure 4As shown, the video information extraction time is taken evenly, and the leaf tilt angle state and data extraction count for each number are recorded. The leaf tilt angle values ​​collected each time are added together, divided by the number of data extractions, and the average value is calculated. The calculated average value is the leaf tilt angle that is closer to the true value.

[0144] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments without departing from the technical essence of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for measuring the tilt angle of plant leaves based on binocular vision, comprising the following steps: (1) Use a binocular vision system to capture the scene and perform disparity estimation. Using the checkerboard method, take 2-20 photos of a checkerboard pattern and find the corner points; use an algorithm encapsulated in the open-source library OpenCV to estimate the corner points, and after correction, obtain the camera's intrinsic parameters as shown below: Among them, f x with f y d represents the focal length of the camera along the x-axis and y-axis, respectively. x With d y This represents the number of pixels that need to be moved horizontally to the left and vertically upward to convert the Cartesian coordinate system to the image coordinate system. Epipolar correction mathematically aligns the camera to the same observation plane, ensuring that the pixel rows on the camera are strictly aligned. It finds a rotation matrix such that after multiplying the left and right cameras by this matrix, the xy coordinate systems of the two camera coordinate systems lie on the same plane, with the x-axis on the same straight line and the y-axis parallel to each other. During the shooting process, some distortion and loss of image may occur. The image is corrected by using the camera's intrinsic parameters. The image is transformed by spatial coordinates using the parameter values ​​obtained from camera calibration. First, a mapping relationship is established between the image coordinates and the corresponding point coordinates of the reference image. Then, the coordinates of each pixel are corrected according to the mapping relationship. Finally, the gray value of each pixel after correction is determined. The disparity estimation algorithm for binocular vision takes two images as input and outputs a disparity map. In the disparity map, each pixel stores the disparity value between the two related images. (2) Using the disparity value, calculate the spatial slope between each pixel and its neighboring pixels. The depth corresponding to each pixel can be calculated using the following formula: Calculate the depth of the spatial point corresponding to each pixel, where b is the baseline length between the optical centers of the two cameras, d is the disparity value between the two matched points, and Z(x) is the spatial depth value of the corresponding pixel. Use formula Estimate the depth of the corresponding point in space; The xyz 3D coordinate system can be viewed as a combination of the xz and y coordinate systems. By slicing the 3D coordinate system according to the pixel column values ​​of x and y respectively, the 3D problem is decomposed into the relationship between z and x when y is a fixed value, and the relationship between z and y when x is a constant value. The spatial slope between adjacent pixels can be calculated using the following formula: Wherein, tanα is the slope of adjacent spatial points in the image coordinate system in the x-direction, and tanβ is the slope of adjacent spatial points in the image coordinate system in the y-direction; z1 and z2 represent the depth information of two spatial points, x1 and x2 represent the abscissa information of two adjacent points in the x-axis direction, and y1 and y2 represent the ordinate information of two adjacent points in the y-axis direction. Using the disparity between two spatial points and the pixel positions of the two spatial points, the tangent angle between the corresponding imaging planes of the two points is calculated as follows: Where, d A This represents the disparity information stored at the corresponding point on the reference plane for point A in space, x. A This represents the x-axis pixel position information of point A in the camera coordinate system; Iterate through all integers x and y, calculate the gradient relationship between z and independent variable y and the gradient relationship between z and independent variable x, and store them in two grayscale images with sizes of (x-1)*y and x*(y-1), respectively. The grayscale values ​​represent the tangent of the angles formed with respect to the imaging plane in the xz coordinate system and the yz coordinate system, respectively. Calculate the angle between each pair of adjacent pixels in the gradient map, and use the formula arctanA=α to calculate the change of angle in the same coordinate axis direction, so as to obtain the angle between two adjacent pixels and the imaging plane, which can be applied to image segmentation. (3) Achieve image segmentation based on the abrupt changes in curvature and direction between planes. Using the x-axis and y-axis as the measurement standard, the angle images corresponding to the x-axis and y-axis are traversed to achieve image segmentation; A threshold φ is set for the angle change, and it is determined that when |α A -α B When |≤φ, pixels with two related angles along the x-axis are on the same plane; when |α... A -α B When |>φ, the pixels with two related angles along the x-axis are located on different planes; The threshold φ is used to determine whether the connection is along the y-axis. When |β A -β B When |≤φ, two related pixels along the y-axis are in the same plane; when |β... A -β B When |>φ, the two pixels associated in the y-axis direction are not in the same plane; By calculating the angle change of adjacent pixels, when the angle change of adjacent pixels is less than a threshold, the two pixels are on the same plane; when the angle change of adjacent pixels is greater than the threshold, the two pixels are not on the same plane, and segmentation is performed. Iterate through the gradient map segmented in the x-direction, divide each row of pixels into different regions, and use a counter to mark each connected region with the same number; When the angular jump in the horizontal direction exceeds the threshold and the number of pixels with the angular jump is greater than the interference threshold, the two planes are not smooth and need to be segmented; if the angular jump in the horizontal direction does not exceed the threshold, or the number of pixels with the angular jump exceeds the threshold is less than the interference threshold, the pixels adjacent to the above pixels are still on the same plane. The image is segmented by traversing the x-direction and the y-direction, merging connected pixel blocks. Traverse the y-direction and, based on the specified threshold, determine whether adjacent gradients in the y-direction represent a sudden change in the gradient of the leaf; if no sudden change occurs, the pixels of the two labels are connected blocks and are in the same plane, and merge the two pixel blocks. Through segmentation and merging, the entire image was segmented based on depth and planar information; (4) Extract the leaf tip and leaf tail points from the segmented image. Set a threshold to separate the background pixels from the leaf segmentation information; In step (3), the image has been segmented, and the pixels in the two-dimensional projection image have been classified. In the current step, the pixel depth calculated in step (1) is used. A threshold is set to filter the pixels and discard the segmented regions with a depth value greater than the threshold. For each segmented region, perform image binarization; set the gray level of pixels in each category to 0, and set the gray level of pixels outside the category to 255. Each leaf has a corresponding binarized image. Morphological transformation is performed on each binarized image to extract the leaf skeleton from two dimensions; the skeleton is the leaf vein, which connects the leaf tip and leaf tail; the starting and ending pixels of the curve in the skeleton structure are selected, and the spatial points corresponding to these pixels are the leaf tip and leaf tail coordinates of each leaf. (5) Calculate the angle between the blade and the lens based on the coordinates of the blade tip and blade tip. Connecting the leaf tip and leaf tail, their projections onto the imaging plane are shown as AB. Through formula Calculate the slopes of adjacent spatial points in the x and y directions in the image coordinate system; calculate the changes in angles along the same coordinate axis using the formula arctanA=α. Knowing the positions and disparity values ​​of two points in the image coordinate system, we can obtain the angle information of the corresponding points. The angle between the line connecting the tip and the tip of the leaf and the line projected onto the imaging plane is the angle between the leaf and the lens. Calculate the angle between any two spatial points and the projection plane using the following formula: Substitute the pixel positions corresponding to the leaf tip and leaf tail into the equation to calculate the angle between the corresponding leaf blade and the imaging plane. (6) Collection and calculation of included angle information The blade tilt angle can be calculated from the angles between the blade and the imaging plane, and between the imaging plane and the ground. The formula is as follows: ∠α-∠β=∠φ Where ∠α is the angle between the blade and the imaging plane, ∠β is the angle between the imaging plane and the ground plane, and ∠φ is the angle between the blade and the ground plane, which is the blade tilt angle we are looking for.

2. The method for measuring the tilt angle of plant leaves based on binocular vision as described in claim 1, wherein the movable angle leaf tilt angle measurement method is as follows: the angle between the fixed rod and the imaging plane is α, and the fixed rod is perpendicular to the ground plane, then the angle β between the imaging plane and the ground can be calculated according to the geometric formula β=90°-α; The mobile device collects information about the blades, using a gyroscope to collect the angle between the mobile device and the ground; When collecting leaf tilt angle information, the binocular vision system used for acquisition can move and rotate flexibly in space. Combining the angle between the leaf and the lens collected by the binocular vision system and the angle between the imaging plane and the ground collected by the gyroscope sensor, a detection algorithm is used to track all leaves in the field of view: each leaf that newly appears in the lens is marked with a new number; the number of the leaves that remain in the imaging plane remains unchanged. Using streaming data, information transmitted from the gyroscope sensor is recorded, and the acquisition time of each piece of information is recorded. The current device attitude is matched with the corresponding video information. The video information extraction time is taken evenly, and the blade tilt angle state and data extraction number for each number are recorded. The blade tilt angle values ​​collected each time are added together, divided by the number of data extractions, and the average value is calculated. The calculated average value is the blade tilt angle that is closer to the true value.

Images