Plant growth monitoring method based on binocular vision stereo matching

Abstract

The invention discloses a plant growth monitoring method based on binocular vision stereo matching, and the method comprises the steps: collecting a target plant image, improving the brightness distribution of the plant image through employing an image enhancement algorithm of histogram equalization and dual gamma function weighted fusion, improving the contrast of the image, and obtaining an improved image; formulating a binocular vision stereo matching constraint criterion; constructing a binocular vision stereo matching model, including matching cost calculation, cost aggregation, parallax calculation and parallax optimization; inputting the improved image into a binocular vision stereo matching model to obtain a final target plant growth image; and performing experimental analysis on the final target plant growth image to obtain a target plant generation stage. According to the method, the pixel matching accuracy of a stereo matching algorithm is improved, so that accurate extraction of distance and phenotype parameters of plant leaves in an actual scene is realized, and certain real-time performance of a system is ensured.

Description

Technical Field

[0001] The invention belongs to computer image processing, and in particular relates to a plant growth monitoring method based on binocular vision stereo matching. Background Art

[0002] Plant growth monitoring is an important task in agriculture. Since we can analyze plant growth based on the width and length of the leaves and the number of leaves, leaves are the most common part of plants that need to be monitored. The phenotypic parameters of plant leaves can directly reflect the growth of plants. The growth rate of plants can be judged according to the changes in the phenotypic parameters of plant leaves. Obtaining the length and width parameters of the leaves can effectively monitor the growth status of plants. Through continuous monitoring of the length and width information of plants, effective scientific guidance can be provided for the field management of plants. In practical applications, the parameter measurement of plant leaves usually adopts the method of manual direct measurement, that is, using tools such as tape measures or rulers to measure the length and width of leaves. Manual measurement of parameters is inefficient and greatly affected by human subjective factors, which does not meet the needs of the development of plant leaf parameter measurement.

[0003] With the development of science and technology, the measurement methods of plant leaf parameters are becoming more and more diverse, and various measurement methods have emerged. At present, the measurement methods are mainly divided into contact and non-contact. Traditional contact measurement methods such as tape measurement and caliper measurement are greatly affected by human subjective factors, and have disadvantages such as low efficiency, high cost and low accuracy. The contact measurement method based on three-coordinate measurement has improved the measurement accuracy to a certain extent, but the measurement speed is slow, the contact point is easy to deform, and the measurement equipment is expensive and large in size, which is not suitable for mass production. With the development of stereo measurement technology and the improvement of computer hardware computing power, binocular stereo vision technology is increasingly widely used in fields such as three-dimensional measurement and target positioning. There are also methods for measuring parameters of objects through monocular cameras, but to obtain the real object parameters, it is necessary to use reference objects, and the real-time performance is insufficient. Nowadays, non-contact measurement methods are developing rapidly, with strong real-time performance and high accuracy. In complex environments, non-contact measurement methods can replace manual precise measurement of plant leaf parameters and gradually become the mainstream method of size measurement. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a plant growth monitoring method based on binocular vision stereo matching. The pixel matching accuracy of the stereo matching algorithm is improved to achieve accurate extraction of the distance and phenotypic parameters of plant leaves in actual scenes, and to ensure a certain degree of real-time performance of the system.

[0005] Technical solution: A plant growth monitoring method based on binocular vision stereo matching of the present invention comprises the following steps:

[0006] Step 1, collect the target plant image, use the image enhancement algorithm of histogram equalization and dual gamma function weighted fusion to improve the brightness distribution of the plant image, improve the image contrast, and obtain the improved image;

[0007] Step 2: Build a binocular vision stereo matching model, including matching cost calculation, cost aggregation, disparity calculation, and disparity optimization;

[0008] Step 3: Input the improved image into the binocular vision stereo matching model to obtain the final target plant growth image;

[0009] Step 4: Conduct experimental analysis on the final target plant growth image to obtain the target plant generation stage.

[0010] Furthermore, step 1 specifically includes the following steps:

[0011] Step 1.1, the dual gamma function is a correction algorithm that increases the brightness of low-brightness areas and reduces the brightness of high-brightness areas. Its basic formula is:

[0012]

[0013] In the above formula, x is the gray value of the image, γ is the correction coefficient used to control the image enhancement strength. When γ = 5, Ga is a convex function used to enhance the image brightness, Gb is a concave function used to reduce the image brightness, and G is a weighted fusion of Ga and Gb; this function can distinguish the details hidden in the dark and bright areas;

[0014] Step 1.2, using adaptive parameter settings, considering the local information of the original image, adaptively selecting the γ value according to the image's own information, using the local mean and standard deviation of the image to predict the image brightness, with the central pixel (x, y) and a local window of size M*N, traversing the image, and selecting a square window of odd size, the local mean and local variance in the window are:

[0015]

[0016] In the formula, M and N are the length and width of the window respectively, M*N is the number of pixels in the local window, x(i,j) is the pixel in the window, u is the mean of the pixels in the local window, and σ is the standard deviation in the local window;

[0017] The γ value is designed by the local mean and local variance. Using the designed γ value, for low-contrast images, the formula for setting γ1 is:

[0018]

[0019] For medium contrast images, the formula for γ2 setting is:

[0020]

[0021] Then use the following formula to select the γ value;

[0022]

[0023] Step 1.3, use the set adaptive γ value as the γ value of the dual gamma function to correct the image. The image captured by the acquisition device is an RGB image. By correcting the RGB three channels, it is necessary to convert the image from the RGB color space to the HSV color space, and then divide the HSV image into H channel, S channel and V channel; H is hue, S is saturation, and V is brightness.

[0024] Furthermore, step 1.3 is as follows: first, the image to be corrected is normalized, and then the image is converted to the HSV color space, and the H channel, S channel and V channel are extracted respectively, and the image is corrected by the improved adaptive dual gamma correction function, and then the H channel, S channel and the corrected V channel are fused together and converted to an RGB image to obtain the final corrected image. The weighted fusion formula is:

[0025]

[0026] Wherein, Ga is the image obtained by the convex function in the dual gamma function, Gb is the image obtained by the concave function in the dual gamma function, Gc is the image after histogram equalization, and G is the image after weighted fusion of the three images, that is, the corrected image.

[0027] Furthermore, in step 2, the matching cost calculation specifically includes the following steps:

[0028] Step 2.1.1. Preprocessing

[0029] Binocular vision stereo matching SGBM first preprocesses the left image and the right image through the horizontal sobel operator, and the calculation process is:

[0030] X_Sobel(x,y)=2*[f(x+1,y)-f(x-1,y)]+f(x+1,y+1)-f(x-1,y+1)+f(x+1,y-1)-f(x-1,y-1)

[0031] The above formula is the calculation formula for preprocessing the image using the Sobel operator in the horizontal direction;

[0032] Step 2.1.2, BT cost

[0033] Then, the BT cost is calculated for the preprocessed left and right images and the left and right images that have not been preprocessed by the Sobel operator. The BT cost is a one-dimensional matching cost, and it only needs to be searched on the same row.

[0034] The calculation method of BT cost is:

[0035] For the right image y i Points, constructed using interpolation:

[0036]

[0037] In the above formula, y i1 and i2 is obtained by linear interpolation in y i and i-1 and i and i+1 The pixel points corresponding to the grayscale values ​​between , and then calculate I by the following formula min ,I max and

[0038]

[0039] In the above formula, I R (y i1 ) and I R (y i2 ) is y i1 and i2 The corresponding gray value, I max For I R (y i ), I R (y i1 ) and I R (y i2 ), I min is the minimum value among them, for y i The matching cost of the point;

[0040] For the left image x i Points are constructed by interpolation in the same way:

[0041]

[0042] In the above formula, x i1 and x i2 is obtained by linear interpolation at x i and x i-1 and x i and x i+1 The pixel points corresponding to the grayscale values ​​between , and then calculate I by the following formulamin ,I max and

[0043]

[0044] Therefore, the final BT cost is:

[0045]

[0046] The X_sobel_BT cost and the grayscale image BT cost are weightedly fused using the following formula;

[0047] cost x =BT x_sobel +α*BT gray

[0048] Here is BT x-sobel The weight is 1, BT gray The weight of is α;

[0049] Step 2.1.3, SAD cost

[0050] The SAD matching algorithm sums the absolute values ​​of the corresponding pixel differences in the corresponding windows of the left and right images after stereo correction. The calculation formula is:

[0051]

[0052] Where n is the window size, L is the pixel point in the window in the left image, R is the corresponding pixel point in the corresponding window in the right image, and d is the disparity;

[0053] Step 2.1.4, SAD_BT matching price

[0054] Since the BT cost is a one-dimensional cost, the SAD cost can be calculated by combining it with the SAD cost for neighborhood summation. The calculation formula is:

[0055]

[0056] The above formula is that the SAD_BT cost of point p is the sum of all BT costs of the neighborhood of the point in the window, that is, the neighborhood summation, to obtain the final matching cost.

[0057] Furthermore, in step 2, the cost aggregation is specifically:

[0058] The semi-global algorithm uses eight directions for cost aggregation. In each direction, the idea of ​​dynamic programming is used to accumulate energy in each direction. The matching costs in each direction are added together to obtain the overall matching cost. The formula is as follows:

[0059]

[0060] The first term on the right side of the formula represents the sum of the matching costs of all pixels when the disparity is D, and the second term represents the matching costs of the pixels in the neighborhood N of the pixel point p. p A penalty coefficient P1 is applied to all pixels q within, and P1 is effective only when the disparity difference is 1. The third term indicates that a larger penalty coefficient P2 is applied to pixels with a disparity difference greater than 1. P1 is to adapt to inclined or curved surfaces, and P2 is to retain discontinuities, which are usually related to gradient changes. The calculation formula of P2 is shown in the formula:

[0061]

[0062] The above formula is the sum of the matching costs of all paths. First, complete the aggregation constraints of a single path, and then add up the aggregation constraints of all directions to complete the cost aggregation of eight directions. The minimum cost formula for a certain direction is shown in the formula:

[0063]

[0064] The first item on the right side of the equation is the data storage item, the second item is the smoothing item, first calculate the minimum value of the three cases of no penalty, P1 penalty and P2 penalty, and then add this minimum value to the path. The third item is the cost constraint item to prevent the cost value from being too large. L r (pr,d) represents the aggregation cost when the disparity of the previous pixel in the path is d, L r (pr,d-1) represents the aggregation cost when the disparity of the previous pixel in the path is d-1, L r (pr,d+1) represents the aggregation cost when the disparity of the previous pixel in the path is d+1. represents the minimum value of all costs of the previous pixel in the path, so the cost of all paths is aggregated as follows:

[0065]

[0066] Where s is the aggregate cost of all paths, L r (P, d) is the minimum cost of a path.

[0067] Furthermore, in step 2, the parallax calculation is specifically as follows:

[0068] The Winner Takes All (WTA) algorithm selects the disparity value with the smallest aggregation cost as the optimal disparity. Since the cost calculation during cost aggregation is not completely accurate, it will cause incorrect disparity, so the disparity needs to be optimized.

[0069] Furthermore, in step 2, the parallax optimization is specifically as follows:

[0070] Based on the disparity optimization of the original SGBM stereo matching algorithm (including sub-pixel interpolation and left-right consistency detection, etc.), flood filling is used to first select a pixel point, and then calculate the difference between the pixel point and the surrounding pixels to determine whether the surrounding pixels belong to the connected neighborhood. If the conditions are met, the pixel point is filled, and this pixel point is used as a seed for the next round of filling until the connected area is completely filled; the flood filling algorithm includes the four-domain filling method and the eight-neighborhood filling method;

[0071] The weighted least squares filter WLS is used to better preserve the edge information of the image, and its loss function c(u) is expressed as:

[0072]

[0073] In the formula, g is the input image, u is the output image, and a x and a y are the weight matrices of the gradients in the x and y directions, respectively, p is the pixel coordinate, the first term indicates that the more similar the input image and the result image are, the better, the second term is the regularization term, which achieves image smoothing by minimizing the partial derivative u, and λ is the regularization term parameter, which is used to balance the proportions of the two. The larger λ is, the smoother the image is;

[0074] The above formula can be written in matrix form as follows:

[0075]

[0076] In the above formula, g and u are the vector representations of the input image and output image respectively, A x and A y Is a x and a y is a diagonal matrix with diagonal elements, D x and D y It is a discrete difference operator. In order to minimize C(u), u needs to satisfy the following conditions:

[0077] (I+λL g )u=g

[0078] Among them, L g The formula is:

[0079]

[0080] The weight is a function that decreases monotonically as the gradient increases. The weight function formula of weighted least squares filtering is:

[0081]

[0082] Where l represents the log value of the input image, α determines the gradient, and ε is a very small value.

[0083] Furthermore, step 3 is specifically as follows: measuring the parameters of the object in the three-dimensional space, collecting the left and right images of the object to be measured by the binocular camera, obtaining the disparity map by performing a stereo matching process on the binocular images, thereby obtaining the disparity value of each point, and then obtaining the three-dimensional coordinates of the two points by selecting two key points. The distance between the two points can be obtained according to the distance formula, that is, the required real size, which is:

[0084]

[0085] In the formula, S1 and S2 are two points in the space object, (x1, y1, z1) and (x2, y2, z2) are the three-dimensional coordinates of the two points respectively, and S1 S2 represents the actual distance between the two points. According to this formula, the real distance between the two points of the leaf in the image can be obtained, thereby obtaining the required length and width parameters.

[0086] The present invention also discloses a computer device, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method of the present invention.

[0087] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0088] (1) For traditional gamma correction, the improved adaptive dual gamma function is integrated with histogram equalization. This scheme effectively solves the limitations of traditional gamma correction, such as the need to manually select the γ value and the global correction. It significantly optimizes the brightness distribution and contrast of the leaf image and reduces the impact on edge detection.

[0089] (2) To address the problem that the disparity map obtained by the SGBM stereo matching algorithm has empty areas and unclear leaf contours, the present invention optimizes the disparity map by combining flood filling and weighted least squares filtering, making the contours of plant leaves clearer in the disparity map and reducing the algorithm's mismatching rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 Block diagram of the improved method of the present invention;

[0091] Figure 2 It is a graph of dual gamma function;

[0092] Figure 3 Comparison diagram of the original image and the corrected image;

[0093] Figure 4 This is the experimental comparison chart after weighted fusion;

[0094] Figure 5 This is a grayscale histogram comparison chart of the algorithm result graph;

[0095] Figure 6 is a block diagram of an improved gamma correction algorithm;

[0096] Figure 7 This is an experimental comparison of different enhancement algorithms on hollyhock leaves;

[0097] Figure 8 The canny edge detection result diagram of different enhancement algorithms;

[0098] Fig. 9 This is the overall flow chart of binocular stereo matching;

[0099] Fig.10 Refine the flow chart for binocular stereo matching;

[0100] Fig.11 Graph for flood fill algorithm;

[0101] Fig.12 Binocular image of leaves;

[0102] Fig.13 Comparison chart of different algorithm experiments;

[0103] Fig.14 Left and right images for standard tests;

[0104] Fig.15 Disparity maps obtained for different stereo matching algorithms;

[0105] Fig.16 is the location diagram of key points of the blade;

[0106] Fig.17 Data line chart. DETAILED DESCRIPTION

[0107] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0108] Figure 1 The flowchart of the improved method of the present invention is shown below. As traditional gamma correction requires manual setting of thresholds, different γ values ​​need to be set for images of different brightness. Therefore, when correcting an image, the same correction is applied to all grayscale values ​​in the image. This single global correction has a reduced correction effect for images with non-uniform brightness, resulting in a decrease in image quality. In response to the above problems, the present invention adopts an image enhancement algorithm that combines weighted fusion of histogram equalization and bi-gamma function (Bilateral Gamma Adjustment, BiGA).

[0109] Figure 2This is a graph of the dual gamma function. Since the γ in the dual gamma function needs to be set manually, images of different brightness require different γ values, which will affect the efficiency of the algorithm. In order to be able to adaptively select γ, an adaptive parameter setting is adopted. The algorithm takes into account the local information of the original image, improves the image contrast, increases the flexibility of the algorithm, and improves the performance of the algorithm. In order to improve the problem of excessive image brightness and low contrast caused by daytime image acquisition, and to be able to adaptively select the γ value based on the image's own information, the local mean and standard deviation of the image are used to predict the image brightness.

[0110] The adaptive γ value set above is used as the γ value of the dual gamma function to correct the image. The image collected by the acquisition device is an RGB image. It is difficult to ensure that the R, G, and B channel components increase or decrease in the same proportion by correcting the three RGB channels. Considering the visual characteristics of humans, it is necessary to convert the image from the RGB color space to the HSV color space, and then divide the HSV image into H channel, S channel, and V channel. H is hue, S is saturation, and V is brightness. Since the correction image is mainly to improve the illumination uniformity of the image, it is only necessary to correct the extracted V channel to achieve the desired result. The main process is: first normalize the image to be corrected, then convert the image to the HSV color space, extract the H channel, S channel, and V channel respectively, correct the image through the improved adaptive dual gamma correction function, and then merge the H channel, S channel, and the corrected V channel, and then convert it into an RGB image. Here, hollyhock plants are taken as the object, and the experimental comparison chart is as follows Figure 3 shown. Figure 3 In the figure, (a) is the original image of hollyhock, and (b) is the image corrected by the improved algorithm. The illumination of the original image is relatively bright. After correction, it can be seen that the details of the leaves are clearer and the brightness is lower. However, from the comparison of the histograms of (c) and (d), although the brightness of the pixels in the corrected image is lower, the grayscale values ​​are still concentrated in a small range, and the contrast is greatly reduced.

[0111] The experimental comparison chart after weighted fusion is as follows Figure 4 As shown, Figure 4 In the figure, (a) is the image after histogram equalization. Compared with the original image, the brightness is reduced, but the brightness of the leaf area is too low, and the image saturation is poor, which does not conform to the visual characteristics of the human eye. (b) is the correction result without the fusion of histogram equalization. It can be seen that the image brightness uniformity is much better than the original image, but the contrast is too low. (c) is the corrected image after the weighted fusion of the adaptive dual gamma function and histogram equalization. It can be seen that the image illumination distribution is relatively uniform and the contrast is good, which conforms to the visual characteristics of humans.

[0112] Figure 5Color histogram comparison of algorithm result graphs. Figure 5 In the figure, (a), (b) and (c) are respectively the histogram after histogram equalization, the grayscale histogram without fusion histogram equalization and the grayscale histogram after fusion histogram equalization. From these three figures, it can be seen that the improved algorithm of the present invention improves the contrast while making the image grayscale distribution more uniform and the distribution range wider. In summary, according to the experimental graph and the grayscale histogram, it can be concluded that the improved algorithm of the present invention can improve the brightness distribution of the image and has certain practicality. The flow chart of the improved algorithm of the present invention is shown in FIG. Figure 6 shown.

[0113] In order to verify the feasibility of the improved algorithm, experiments and data comparisons were conducted on the performance of the improved algorithm from both subjective and objective perspectives. In order to reasonably evaluate the performance of the improved algorithm of the present invention, representative correction algorithms were selected, including SSR enhancement, MSRCR enhancement, histogram equalization and improved algorithm (Research on image enhancement method of fruit with uneven illumination based on FPGA [J]. Li Wenjing, Liang Ying, Zhou Ruiyun. Journal of Chizhou University, 2022, 36(06): 20-23.) The algorithm was compared with the improved algorithm of the present invention. The subjective comparison of the results of different algorithms on hollyhock leaves is shown in the figure below. Figure 7 As shown in the figure, the canny edge detection comparison of different algorithms for hollyhock pictures is shown in Figure 8 In order to accurately obtain the length and width parameters of plant leaves, the improved method of the present invention subsequently uses canny edge detection to extract the leaf contour, so as to more accurately calculate the three-dimensional information on the leaf contour. Figure 8 It can be seen from the figure that the improved method of the present invention can obtain a complete blade contour for the canny edge detection algorithm, while the edge contours obtained by other correction algorithms are incomplete. It can be seen that too high image brightness will affect the effect of edge detection. The objective data comparison is shown in Table 1.

[0114] Table 1 Comparison of hollyhock leaf image correction data using different algorithms

[0115]

[0116] It can be seen from Table 1 that the mean of the SSR enhancement and MSRCR enhancement algorithms is too large, which means that the image brightness is too large, which seriously affects the visual effect of the image, and the peak signal-to-noise ratio is low. A grayscale value near 128 indicates that the visual effect of the image is good. The mean of the improved algorithm of the present invention in Table 1 is the closest to 128, so the visual effect of the resulting image of the improved algorithm of the present invention is better than that of other algorithms. Although the improved algorithm of the present invention is slightly lower than the histogram equalization in terms of clarity index, the improved root mean square error (MSE) of the present invention is the lowest, and the peak signal-to-noise ratio (PSNR) is the highest. It can be seen from the objective data that the improved algorithm of the present invention has a certain effectiveness. On the whole, for the corrected image of the present invention, the performance is better than that of the improved algorithms of other algorithms.

[0117] like Fig. 9 As shown in the figure, the binocular stereo matching process includes matching cost calculation, cost aggregation, disparity calculation and disparity optimization; the global matching algorithm does not perform the cost aggregation step, but defines a global energy function containing data terms and smoothing terms. The method of calculating the disparity by minimizing the energy function has a higher matching accuracy, but due to its high computational complexity, it cannot be applied in most scenarios. The improved method of the present invention adopts the SGBM semi-global stereo matching algorithm to obtain the disparity map. Fig.10 This is a refined flowchart of the SGBM stereo matching algorithm.

[0118] Since the present invention measures parameters through binocular vision, parallax is necessary to obtain the three-dimensional information of the object, and the accuracy of the parallax will greatly affect the measurement accuracy of the present invention. Therefore, it is extremely important to obtain an accurate disparity map. However, the disparity map obtained by the SGBM semi-global stereo matching algorithm contains many hole areas, and the leaf contours are unclear, and the pixel mismatch rate is low. Therefore, the present invention proposes an improved solution to optimize the disparity map, and uses a flood filling algorithm and WLS filtering to fill the hole areas in the disparity map and smooth the image.

[0119] (1) Flood filling

[0120] The flood fill method is a filling algorithm that simulates the fluid diffusion process to fill the connected area with a specific color. The algorithm principle is to first select a pixel point, then calculate the difference between the pixel point and the surrounding pixels to determine whether the surrounding pixels belong to the connected neighborhood. If the condition is met, the pixel point is filled and this pixel point is used as a seed for the next round of filling until the connected area is completely filled. The main flood fill algorithms are the four-neighborhood filling method and the eight-neighborhood filling method. See the respective filling directions for details. Fig.11 .from Fig.11As can be seen, Figure (a) is a four-neighborhood filling method. The principle is to use the selected pixel as the starting point to determine whether the adjacent pixels in the four directions of up, down, left, and right need to be filled. As can be seen from Figure (b), the eight-neighborhood filling method uses the selected pixel as the starting point to determine whether the adjacent pixels in the eight directions of up, down, left, right, upper left, lower left, upper right, and lower right need to be filled. As can be seen from the figure, the eight-neighborhood filling method will result in overfilling, which is not required. Therefore, the four-neighborhood filling method can not only reduce the amount of algorithm calculations but also fill the black area.

[0121] In the disparity map, since there is a grayscale mutation between invalid points and valid points, the flood filling method can be used to fill the disparity. Taking a non-zero point in the hole area as the starting point, the connected area is filled to improve the quality of the disparity map. Then the disparity map is processed by weighted least squares filtering.

[0122] In order to verify the feasibility of this improved algorithm, a visual comparison is performed using the SAD matching algorithm, BM matching algorithm, SGBM matching algorithm and the improved algorithm of the present invention; Fig.12 For this experiment, Fig.13 The disparity maps obtained by the SAD matching algorithm, BM matching algorithm, SGBM matching algorithm and the improved algorithm of the present invention are compared; it can be seen that the disparity maps obtained by the SAD matching algorithm and the BM matching algorithm have a lot of mismatching points, and the outline of the blade area is almost invisible, which will directly affect the accuracy of subsequent three-dimensional measurements. The disparity map obtained by the SGBM algorithm has significantly fewer mismatching points, but there are still some hole areas, and the edge contour is intermittent. The disparity map is obtained by combining the flood filling and WLS filtering proposed by the present invention, and the hole areas that existed before the improvement are repaired. The blade outline becomes clearly visible and the image becomes smoother. In order to more accurately evaluate the effectiveness of the improved disparity optimization scheme, the mismatch rate is used here to evaluate the feasibility of the improved scheme. The higher the mismatch rate, the worse the quality of the disparity map, which reduces the accuracy of the disparity value. Therefore, the lower the mismatch rate, the more accurate the disparity value, and the better the quality of the disparity map. The expression of the mismatch rate is shown in the formula:

[0123]

[0124] d(i,j) represents the disparity value obtained by stereo matching, d g (i, j) is the disparity value of the standard test image, δ d Indicates the error value of image matching, which is generally set to 1. Fig.14 The standard test left and right images and their ideal disparity maps selected from the Middlebury dataset are Teddy image and Cones image respectively. Fig.15The disparity maps of two standard test images obtained by SAD matching, BM matching, SGBM matching and the improved method of the present invention are shown in Table 2.

[0125] Table 2 Comparison of mismatching rates of different stereo matching algorithms

[0126]

[0127] It can be seen from the data in Table 2 that the mismatch rate of the improved method of the present invention is the lowest, and the mismatch rate of the SAD stereo matching algorithm is the highest. In summary, the hole area of ​​the disparity map obtained by the improved disparity optimization is filled, which reduces the mismatch points. The WLS filtering of the disparity map can make the image contour information more complete and clearer. Therefore, the present invention can improve the quality of the disparity map.

[0128] Experimental Analysis of Blade Parameter Measurement

[0129] The parameter measurement of plant leaves mainly involves obtaining three-dimensional information through the key points of the object to be measured, and then obtaining the real parameters of the leaves through the distance formula. According to the disparity map obtained by stereo matching, the real distance between two points can be calculated based on triangulation and Euclidean distance. Based on the research and analysis of the first four chapters, this paper builds a binocular vision parameter measurement platform, and completes the parameter measurement of hollyhock leaves based on this platform, and verifies the measurement accuracy.

[0130] Hardware Composition

[0131] A measurement platform consisting of a 4-megapixel USB2.0 binocular camera, a camera support, a soft ruler and a Lenovo laptop computer with Inter(R)Core-i5 was built.

[0132] Software components

[0133] All algorithms were completed on Matlab and Pycharm, and the system interface design was completed using pyqt5.

[0134] Experimental measurements

[0135] In order to more accurately extract the three-dimensional coordinates of a point on the edge of the leaf, the outline of the leaf area in the image can be extracted and the outline of the leaf can be drawn on the image, so that the edge of the leaf can be accurately clicked by the mouse. The workflow is as follows: first, the outlines of the left and right images are extracted respectively by the canny edge detection algorithm, and then the area inside the outline is processed by morphological operation, and then the image is gray-scale flipped, and finally the leaf area in the original image is extracted. After obtaining the edge image, it is necessary to process the inside of the outer contour. The morphological processing scheme can eliminate the internal black blocks. After determining the position of the target area, the target leaf area can be extracted. After obtaining the left and right images of the target area, the outline of the target area can be drawn on the left image, and the disparity value of the point can be output by clicking on the point on the outline.

[0136] Experimental results and analysis

[0137] The disparity map and the left image with the outer contour of the target area are obtained, and the three-dimensional coordinates of the point on the image where the mouse is clicked can be calculated. The researchers selected the key points of leaves A, B, C, D and E on the image, marked them with yellow dots, and calculated the distance between the key points, and then the length and width parameters of the leaf can be calculated. Fig.16 An image with key points marked.

[0138] The three-dimensional coordinates of the key points in the image are printed out, and the length and width of the leaf are calculated using the Euclidean distance formula. Table 3 compares the calculated values ​​of the length and width parameters of leaf A, leaf B, leaf C, leaf D, and leaf E in the image with the data actually measured with a tape measure.

[0139] Table 3 Three-dimensional coordinates of key points in the blade area

[0140]

[0141]

[0142] The three-dimensional coordinates of each key point are obtained. The distance between the key points is recorded, that is, the key point distance between the required length and width parameters, and compared with the parameter values ​​measured manually by a tape measure. The data comparison is shown in Table 4;

[0143] Table 4 Comparison of measured data

[0144]

[0145] In Table 4, the measured value is the value calculated by the algorithm of the present invention, the actual value is the value measured manually with a tape measure, and the error is the percentage of the actual value divided by the measured value. Fig.17The data in the table are presented in a line graph. From the data in the table and the line graph, it can be seen that the error can basically be maintained at about 4%. Therefore, the algorithm of the present invention can basically measure the length and width parameters of the blade and is feasible.

Claims

1. A plant growth monitoring method based on binocular vision stereo matching, characterized in that: The steps include: Step 1, collect the target plant image, and use the image enhancement algorithm of histogram equalization and dual gamma function weighted fusion to improve the brightness distribution of the plant image to obtain the improved image; Step 2: Build a binocular vision stereo matching model, including matching cost calculation, cost aggregation, disparity calculation, and disparity optimization; Step 3: Input the improved image into the binocular vision stereo matching model to obtain the final target plant growth image; Step 4: Conduct experimental analysis on the final target plant growth image to obtain the target plant generation stage.

2. A method for monitoring plant growth based on binocular vision stereo matching according to claim 1, characterized in that: Step 1 specifically includes the following steps: Step 1.1, the dual gamma function is a correction algorithm that increases the brightness of low-brightness areas and reduces the brightness of high-brightness areas. Its basic formula is: In the above formula, x is the gray value of the image, γ is the correction coefficient used to control the image enhancement strength. When γ = 5, Ga is a convex function used to enhance the image brightness, Gb is a concave function used to reduce the image brightness, and G is a weighted fusion of Ga and Gb; this function can distinguish the details hidden in the dark and bright areas; Step 1.2, using adaptive parameter settings, considering the local information of the original image, adaptively selecting the γ value according to the image's own information, using the local mean and standard deviation of the image to predict the image brightness, with the central pixel (x, y) and a local window of size M*N, traversing the image, and selecting a square window of odd size, the local mean and local variance in the window are: In the formula, M and N are the length and width of the window respectively, M*N is the number of pixels in the local window, x(i,j) is the pixel in the window, u is the mean of the pixels in the local window, and σ is the standard deviation in the local window; The γ value is designed by the local mean u and the local variance σ. Using the designed γ value, for low contrast images, The formula for setting γ1 is: For medium contrast images, the formula for γ2 setting is: Then use the following formula to select the γ value; Step 1.3, use the set adaptive γ value as the γ value of the dual gamma function to correct the image. The image captured by the acquisition device is an RGB image. By correcting the RGB three channels, it is necessary to convert the image from the RGB color space to the HSV color space, and then divide the HSV image into H channel, S channel and V channel; H is hue, S is saturation, and V is brightness.

3. A method for monitoring plant growth based on binocular vision stereo matching according to claim 2, characterized in that: Step 1.3 is as follows: first normalize the image to be corrected, then convert the image to the HSV color space, extract the H channel, S channel and V channel respectively, correct the image through the improved adaptive dual gamma correction function, then fuse the H channel, S channel and the corrected V channel, and then convert it to an RGB image to obtain the final corrected image. The weighted fusion formula is: Wherein, Ga is the image obtained by the convex function in the dual gamma function, Gb is the image obtained by the concave function in the dual gamma function, Gc is the image after histogram equalization, and G is the image after weighted fusion of the three images, that is, the corrected image.

4. The plant growth monitoring method based on binocular vision stereo matching according to claim 1, characterized in that: In step 2, the matching cost calculation specifically includes the following steps: Step 2.1.

1. Preprocessing Binocular vision stereo matching SGBM first preprocesses the left image and the right image through the horizontal sobel operator, and the calculation process is: X_Sobel(x,y)=2*[f(x+1,y)-f(x-1,y)]+f(x+1,y+1)-f(x-1,y+1)+f(x+1,y-1)-f(x-1,y-1)The above formula is the calculation formula for the horizontal sobel operator to preprocess the image; Step 2.1.2, BT cost Then, the BT cost is calculated for the preprocessed left and right images and the left and right images that have not been preprocessed by the Sobel operator. The BT cost is a one-dimensional matching cost and only needs to be searched on the same row. The calculation method of BT cost is: For the right image y i Points, constructed using interpolation: In the above formula, y i1 and i2 is obtained by linear interpolation in y i and i-1 and i and i+1 The pixel points corresponding to the grayscale values ​​between , and then calculate I by the following formula min ,I max and In the above formula, I R (y i1 ) and I R (y i2 ) is y i1 and i2 The corresponding gray value, I max For I R (y i ), I R (y i1 ) and I R (y i2 ), I min is the minimum value among them, for y i The matching cost of the point; For the left image x i Points are constructed by interpolation in the same way: In the above formula, x i1 and x i2 is obtained by linear interpolation at x i and x i-1 and x i and x i+1 The pixel points corresponding to the grayscale values ​​between , and then calculate I by the following formula min ,I max and Therefore, the final BT cost is: The X_sobel_BT cost and the grayscale image BT cost are weightedly fused using the following formula; cost x =BT x_sobel +α*BT gray Here is BT x-sobel The weight is 1, BT gray The weight of is α; Step 2.1.3, SAD cost The SAD matching algorithm sums the absolute values ​​of the corresponding pixel differences in the corresponding windows of the left and right images after stereo correction. The calculation formula is: Where n is the window size, L is the pixel point in the window in the left image, R is the corresponding pixel point in the corresponding window in the right image, and d is the disparity; Step 2.1.4, SAD_BT matching price Since the BT cost is a one-dimensional cost, the SAD cost can be calculated by combining it with the SAD cost for neighborhood summation. The calculation formula is: The above formula is that the SAD_BT cost of point p is the sum of all BT costs of the neighborhood of the point in the window, that is, the neighborhood summation, to obtain the final matching cost.

5. The plant growth monitoring method based on binocular vision stereo matching according to claim 1, characterized in that: In step 2, the cost aggregation is specifically as follows: The semi-global algorithm uses eight directions for cost aggregation. In each direction, the idea of ​​dynamic programming is used to accumulate energy in each direction. The matching costs in each direction are added together to obtain the overall matching cost. The formula is as follows: The first term on the right side of the formula represents the sum of the matching costs of all pixels when the disparity is D, and the second term represents the matching costs of the pixels in the neighborhood N of the pixel point p. p A penalty coefficient P1 is applied to all pixels q within, and P1 is effective only when the disparity difference is 1. The third term indicates that a larger penalty coefficient P2 is applied to pixels with a disparity difference greater than 1. P1 is to adapt to inclined or curved surfaces, and P2 is to retain discontinuities, which are usually related to gradient changes. The calculation formula of P2 is shown in the formula: The above formula is the sum of the matching costs of all paths. First, complete the aggregation constraints of a single path, and then add up the aggregation constraints of all directions to complete the cost aggregation of eight directions. The minimum cost formula for a certain direction is shown in the formula: The first item on the right side of the equation is the data storage item, the second item is the smoothing item, first calculate the minimum value of the three cases of no penalty, P1 penalty and P2 penalty, and then add this minimum value to the path. The third item is the cost constraint item to prevent the cost value from being too large. L r (pr,d) represents the aggregation cost when the disparity of the previous pixel in the path is d, L r (pr,d-1) represents the aggregation cost when the disparity of the previous pixel in the path is d-1, L r (pr,d+1) represents the aggregation cost when the disparity of the previous pixel in the path is d+1. represents the minimum value of all costs of the previous pixel in the path, so the cost of all paths is aggregated as follows: Where s is the aggregate cost of all paths, L r (P, d) is the minimum cost of a path.

6. The method for plant growth monitoring based on binocular vision stereo matching according to claim 1, characterized in that: In step 2, the disparity calculation is specifically as follows: Using the winner-takes-all WTA algorithm, the disparity value with the smallest aggregation cost is selected as the optimal disparity.

7. The method for plant growth monitoring based on binocular vision stereo matching according to claim 1, characterized in that: In step 2, the parallax optimization is specifically as follows: Flood filling first selects a pixel point, then calculates the difference between the pixel point and the surrounding pixels to determine whether the surrounding pixels belong to the connected neighborhood. If the condition is met, the pixel point is filled and this pixel point is used as a seed for the next round of filling until the connected area is completely filled. The flood filling algorithm includes the four-domain filling method and the eight-neighborhood filling method. The weighted least squares filter WLS is used to better preserve the edge information of the image, and its loss function c(u) is expressed as: In the formula, g is the input image, u is the output image, and a x and a y are the weight matrices of the gradients in the x and y directions, respectively, p is the pixel coordinate, the first term indicates that the more similar the input image and the result image are, the better, the second term is the regularization term, which achieves image smoothing by minimizing the partial derivative u, and λ is the regularization term parameter, which is used to balance the proportions of the two. The larger λ is, the smoother the image is; The above formula can be written in matrix form as follows: In the above formula, g and u are the vector representations of the input image and output image respectively, A x and A y Is a x and a y is a diagonal matrix with diagonal elements, D x and D y It is a discrete difference operator. In order to minimize C(u), u needs to satisfy the following conditions: (I+λL g )u=g Among them, L g The formula is expressed as: The weight is a function that decreases monotonically as the gradient increases. The weight function formula of weighted least squares filtering is: Where l represents the log value of the input image, α determines the gradient, and ε is a very small value.

8. The method for plant growth monitoring based on binocular vision stereo matching according to claim 1, characterized in that: Step 3 is as follows: measure the parameters of the object in the three-dimensional space, collect the left and right images of the object to be measured by the binocular camera, obtain the disparity map by stereo matching the binocular images, and then obtain the disparity value of each point, and then obtain the three-dimensional coordinates of the two points by selecting two key points. According to the distance formula, the distance between the two points can be obtained, that is, the required real size. The formula is: In the formula, S1 and S2 are two points in the space object, (x1, y1, z1) and (x2, y2, z2) are the three-dimensional coordinates of the two points respectively, and S1 S2 represents the actual distance between the two points. According to this formula, the real distance between the two points of the leaf in the image can be obtained, thereby obtaining the required length and width parameters.

9. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method of claim 1.

Images