Method for measuring material stacking height of grain tank of grain transport vehicle based on binocular vision

Abstract

The invention belongs to the technical field of agricultural machinery, and discloses a method for measuring the stacking height of materials in a grain tank of a grain transport vehicle based on binocular vision, which comprises the following steps of: constructing a binocular vision system which comprises two cameras with the same parameters, installing the binocular vision system on the grain transport vehicle, shooting the grain tank at a fixed angle, and calculating the stacking height of the materials in the grain tank according to parallax caused by the positions of the cameras; the space coordinates of the feature points are calculated through triangulation by using the difference between the pixel coordinate values of the left image and the right image, the depth information of the measured target is obtained, and the object is positioned and measured; through accurate binocular camera calibration and an efficient stereo matching algorithm, rapid and non-contact measurement of the material stacking height is achieved, the error rate is as low as 4% or below, the high-precision operation requirement is met, measurement data are further optimized through introduction of a linear regression model, errors are effectively reduced, and the measurement accuracy is improved. And the stability and the reliability of a measurement result are improved.

Description

Technical Field

[0001] The invention belongs to the technical field of agricultural machinery, and in particular relates to a method for measuring the material stacking height of a grain box of a grain transport vehicle based on binocular vision. Background Art

[0002] The development of intelligent agricultural machinery is an indispensable part of the development of modern agriculture, and has an important impact on modern agriculture. Intelligent agricultural machinery improves production efficiency and quality, reduces production costs and labor intensity, and promotes the sustainable development of agricultural production by carrying advanced technologies such as sensors, cameras, and data analysis systems. This intelligent trend provides important support for the modernization, efficiency, and sustainability of modern agriculture, and has far-reaching significance for achieving food security, improving farmers' living standards, and protecting the environment.

[0003] Research on the field of cooperative harvesting of multiple agricultural machinery is an important direction for the current development of intelligent agricultural machinery, and the unloading of grain by following the vehicle of combined harvesting is one of the key technologies for the cooperative operation of multiple machines. At present, the unloading of grain by following the vehicle of combined harvesting mainly relies on manpower to cooperate and human observation to judge the filling status of the grain box, and its loading rate is low. Machine vision technology is an important branch of the rapid development of artificial intelligence. Combined with image processing, pattern recognition and other technologies, it can realize the recognition of specific objects, as well as the mutual positioning and position tracking of multiple objects. Target recognition, spatial positioning and position tracking technology based on machine vision has become a hot spot and difficulty in the field of intelligent control of agricultural machinery equipment. Binocular vision is an important branch of machine vision. It uses two camera devices to simulate human binocular vision, obtains depth information through computer algorithms, and realizes applications such as distance measurement and object recognition. Binocular vision technology has a wide range of applications in the fields of autonomous driving, robot navigation, industrial measurement, etc.

[0004] In recent years, with the rapid development of machine vision technology, its application in combine harvester systems has become more and more extensive. Machine vision is usually used in unloading and loading systems, and the unloading position is fixed directly above the vehicle, or measured by the vehicle transportation system through real-time display. Tian Ruiming et al. collected stacking images to obtain information such as the edge contour of the stack, stacking angle, material height, etc., calculated the natural stacking angle, and identified the loading state. Zhang Chi used a camera to capture images of the height changes of materials in the compartment, extracted the area of ​​interest, and performed edge extraction and image enhancement. When the edge of the grain is measured to coincide with the edge of the grain truck, it is full. Foreign research on this technology is mostly applied to forage harvesters. For example, John Deere's active filling control system, Claas's automatic filling system, and New Holland's IntelliFillTM system.

[0005] At present, the research on automatic measurement of the material stacking height of grain trucks is still at a low level in China. The measurement of the material stacking height of grain trucks still requires the operator to observe the throwing device and the material stacking height of the grain trucks with the naked eye or simply observe through video. This operation has various defects such as inaccurate throwing, material waste, low utilization rate of grain trucks, and untimely response due to high workload of operators. Summary of the invention

[0006] The purpose of the present invention is to provide a method for measuring the material stacking height of a grain tank of a grain transport vehicle based on binocular vision, so as to solve the problems raised in the above-mentioned background technology.

[0007] To achieve the above object, the present invention provides the following technical solution: A method for measuring the material stacking height of a grain tank of a grain transport vehicle based on binocular vision, comprising the following steps:

[0008] S1. Build a binocular vision system, including two cameras with the same parameters, and install them on the grain transport vehicle. Take pictures of the grain box at a fixed angle. Through the parallax caused by the position of the camera and the difference in the pixel coordinate values ​​of the left and right images, calculate the spatial coordinates of the feature points through triangulation, obtain the depth information of the measured target, and locate and measure the object;

[0009] S2. Calibrate and correct the binocular camera parameters, use Zhang Zhengyou calibration method to obtain the camera's internal and external parameters, and eliminate image distortion;

[0010] S3. Obtain left and right images of the material in the grain box through a binocular camera, calculate the disparity map using a stereo matching algorithm, and then obtain a depth map;

[0011] S4. Calculate the three-dimensional coordinates of the bottom of the grain tank and the material accumulation surface based on the depth map and the position parameters of the binocular camera to obtain the material accumulation height;

[0012] Preferably, the stereo matching algorithm includes a BM algorithm or a SGBM algorithm.

[0013] Preferably, the S1 specifically comprises the steps of:

[0014] S11: Build a binocular vision system, including two cameras with the same parameters, and install them on the grain transport vehicle to shoot the grain box at a fixed angle;

[0015] S12: Let XO1YZ be the world coordinates, coincident with the left camera coordinate system, O1 be the optical center of the left camera, O2 be the optical center of the right camera, the focal lengths of the two cameras be f, the distance b between the optical centers of the left and right cameras be the baseline of the binocular camera, and let the three-dimensional coordinates of point P be (x, y, z);

[0016] S13: When a binocular camera is used to shoot point P, the points formed by the images of point P on the left and right camera planes are recorded as P1(x1, y1) and P2(x2, y2) respectively. The three-dimensional coordinates of point P are obtained based on the formula of similar triangle principle:

[0017]

[0018] Among them, d is the difference between the horizontal coordinates of points P1 and P2 imaged by the left and right cameras.

[0019] Preferably, the S2 specifically comprises the steps of:

[0020] S21: First, use a binocular camera to obtain a pair of chessboard calibration images, and then extract the corner points of each set of images. In the process of extracting corner points, the calibration chessboard is allowed to occupy 50% to 80% of the entire image, and space is reserved at the edges and corners of the image to prevent areas with severe distortion from affecting the calibration results.

[0021] S22: Calculate the intrinsic and extrinsic parameters of the camera using multiple checkerboard images from different perspectives using the camera projection model and linear algebra methods;

[0022] S23: After the binocular camera is calibrated, the original image acquired by the binocular camera is stereo corrected using the internal and external parameters of the camera obtained by the calibration to eliminate image distortion;

[0023] S24: Calculate the depth information and the depth map through the parallax and the known internal and external parameters of the binocular camera, and calculate the distance from the object to the camera.

[0024] Preferably, S3 specifically includes the steps of:

[0025] S31: After obtaining the parallax of the left and right images, restore the image pixel distance according to the binocular distance measurement principle to obtain the distance information of each pixel point, and finally obtain the distance from the stacked materials in the test grain box to the binocular camera;

[0026] S32: By measuring the distance D between the bottom of the vehicle box and the binocular camera when empty and the distance d between the surface of the piled potatoes and the binocular camera when loaded, the stacking height H at different positions when loaded is finally measured by a preset formula.

[0027] Preferably, in S32, the preset formula is:

[0028] H = Dd;

[0029] Where H is the stacking height, D is the measured distance between the bottom of the truck box and the binocular camera when empty, and d is the distance between the surface of the stacked grain box and the binocular camera when loaded.

[0030] Preferably, in S22, the calculation of the intrinsic parameters of the camera includes focal length, principal point, and distortion coefficient.

[0031] Preferably, in S22, the extrinsic parameters include rotation and translation values ​​of the camera relative to the chessboard.

[0032] Preferably, the method of calculating the three-dimensional coordinates of the bottom of the grain box and the material accumulation surface based on the depth map and the position parameters of the binocular camera, and obtaining the material accumulation height includes the following steps:

[0033] S5. Perform linear regression model correction on the measurement results.

[0034] Preferably, in S13, the similar triangle principle formula is:

[0035]

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The binocular vision-based material stacking height measurement method and optimization of the grain tank of a grain transport vehicle proposed in the present invention significantly improves the automation and intelligence level of the combine harvester operation. The method realizes fast and non-contact measurement of the material stacking height through precise binocular camera calibration and efficient stereo matching algorithm. The error rate is as low as less than 4%, which meets the high-precision operation requirements. The introduction of the linear regression model further optimizes the measurement data, effectively reduces the error, and improves the stability and reliability of the measurement results. It also shows great potential in the follow-up unloading operation of the combine harvester, which not only greatly improves the measurement efficiency and reduces the uncertainty and safety risks of manual visual inspection, but also promotes the continuity and efficiency of the harvesting process. In addition, its high-precision characteristics ensure the precise control of material loading and avoid transportation safety hazards and waste of resources caused by excessive loading. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is the principle diagram of triangulation of the present invention;

[0039] Figure 2 It is a schematic diagram of calibration data of the calibration paper of the present invention;

[0040] Figure 3 A schematic diagram of calibration data of the calibration plate of the present invention;

[0041] Figure 4 Schematic diagram of the original binocular image of the present invention;

[0042] Figure 5 is a schematic diagram of an image after correction according to the present invention;

[0043] Figure 6This is a schematic diagram of the original images obtained by the left and right cameras when the grain tank of the present invention is empty;

[0044] Figure 7 It is a depth map and a pseudo-color map after BM and SGBM stereo matching when the grain tank is empty in the present invention;

[0045] Figure 8 It is one of the schematic diagrams of an embodiment of the present invention;

[0046] Fig. 9 This is a second schematic diagram of an embodiment of the present invention;

[0047] Fig.10 Schematic diagram of the relationship between the actual distance of the target and the binocular measurement distance and absolute distance of the present invention;

[0048] Fig.11 It is a schematic diagram of measuring the stacking height of the contents in the grain box according to the present invention;

[0049] Fig.12 This is a schematic diagram of potato pile height data of the present invention;

[0050] Fig.13 It is a scatter plot and trend line schematic diagram of the present invention;

[0051] Fig.14 It is a residual schematic diagram of the present invention;

[0052] Fig.15 It is a schematic diagram of the comparison results of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] See also Figure 1-Figure 15 As shown, the present invention provides the following technical solutions:

[0055] Implementation method 1:

[0056] Step S1: Select two high-resolution, low-distortion industrial cameras and ensure that their parameters (such as focal length, resolution, etc.) are exactly the same.

[0057] The two cameras were mounted on top of the grain truck, pointing downward at about a 45-degree angle toward the grain tank, ensuring full coverage of the interior of the grain tank.

[0058] The camera is mounted on a fixed bracket to ensure that the camera position remains stable during shooting.

[0059] Step S2: Using the Zhang Zhengyou calibration method, the camera's intrinsic parameters (focal length, principal point, distortion coefficient) and extrinsic parameters (rotation and translation values) are calibrated by taking multiple images of the checkerboard calibration plate at different angles and positions.

[0060] The calibration results are used to perform distortion correction and stereo correction on the images acquired by the camera to ensure image quality.

[0061] Step S3: Collect the left and right images of the corn kernels in the grain box, perform stereo matching using the BM (block matching) algorithm, and generate a disparity map.

[0062] According to the disparity map and camera parameters, the depth map, that is, the distance information corresponding to each pixel, is calculated.

[0063] Step S4: In the depth map, identify the lowest point of the bottom of the grain tank and the highest point of the corn kernel accumulation surface.

[0064] According to the camera's position parameters and depth information, the three-dimensional coordinates of the two points are calculated to obtain the stacking height of the corn kernels.

[0065] Step S5: Statistically analyze the results of multiple measurements, establish a linear regression model for correction, and improve measurement accuracy.

[0066] Implementation method 2:

[0067] Step S1: Select a camera that is suitable for the color and texture of wheat to ensure that the wheat accumulation can be clearly photographed.

[0068] Install the camera on both sides of the grain truck and aim it at the grain tank at a vertical angle to reduce measurement errors caused by angle.

[0069] Step S2: Use the SGBM (semi-global block matching) algorithm for camera calibration and correction, because SGBM performs better in dealing with complex textures and lighting changes.

[0070] The calibration process is the same as implementation method 1, but more emphasis is placed on adaptability to lighting changes.

[0071] Step S3: Collect the left and right images of the wheat pile, use the SGBM algorithm to perform stereo matching, and generate a high-precision disparity map.

[0072] The depth map is calculated using the disparity map and camera parameters to obtain detailed distance information of the wheat pile.

[0073] Step S4: In the depth map, the boundary between the bottom of the grain box and the surface of the wheat pile is identified by an edge detection algorithm.

[0074] Calculate the three-dimensional coordinates corresponding to the boundary and obtain the pile height of wheat.

[0075] Step S5: Perform linear regression analysis on the measurement results under different loading amounts, establish a correction model, and improve the stability and accuracy of the measurement results.

[0076] Implementation method three:

[0077] Step S1: Select a camera with powerful image processing capabilities to cope with complex scenes when multiple materials are mixed.

[0078] The camera is mounted on the top center of the grain truck to capture the interior of the grain tank from a bird's-eye view, ensuring full coverage of the field of view.

[0079] Step S2: Using a comprehensive calibration method, combining Zhang Zhengyou's calibration method and specific calibration objects in actual scenes, the camera is accurately calibrated.

[0080] Advanced distortion correction is performed on the images acquired by the camera to eliminate interference caused by the mixing of multiple materials.

[0081] Step S3: Using an improved stereo matching algorithm (such as an algorithm combined with deep learning), the left and right images of various materials are efficiently matched to generate an accurate disparity map.

[0082] The depth map is calculated based on the disparity map and camera parameters to obtain detailed distance information of the mixed material accumulation.

[0083] Step S4: In the depth map, different materials are separated by image segmentation technology, and their stacking heights are calculated respectively.

[0084] The material stacking situation of the entire grain tank can be obtained by comprehensively analyzing the stacking height of each material.

[0085] Step S5: Comprehensively analyze the measurement results of mixed stacking of multiple materials, establish a multivariate linear regression model for correction, and improve measurement accuracy and reliability.

[0086] In addition, in the present invention, the depth information of the object in space is lost when the image is formed on the photosensitive element. Compared with monocular vision, the working principle of binocular stereo vision is similar to that of human eyes. Two cameras with the same parameters are used to shoot the target under the condition of known relative position relationship. The basic principle of binocular vision is to use the difference between the left and right image pixel coordinate values, the difference between the left and right image pixel coordinate values, and the difference between the left and right image pixel coordinate values. The spatial coordinates of the feature points are calculated by triangulation to obtain the depth information of the measured target and realize the positioning and measurement of the object. The principle of triangulation is as follows: Figure 1 As shown, the specific process is as follows:

[0087] Let XO1YZ be the world coordinates, coincident with the left camera coordinate system, O1 be the optical center of the left camera, O2 be the optical center of the right camera, the focal length of the two cameras be f, the distance b between the optical centers of the left and right cameras be the baseline of the binocular camera, and let the three-dimensional coordinates of point P be (x, y, z). When the binocular camera is used to shoot point P, the points formed by point P on the left and right camera planes are recorded as P1(x1, y1) and P2(x2, y2), respectively. According to the principle of similar triangles, we know that:

[0088]

[0089] Then the three-dimensional coordinates of point P can be obtained as:

[0090]

[0091] In formula (2), d is the difference between the horizontal coordinates of point P1 and point P2 obtained by the left and right cameras, which is called parallax. Therefore, as long as the intrinsic parameters and parallax of the camera are obtained, the three-dimensional coordinates of point P in space can be obtained.

[0092] In addition, in the present invention, the binocular camera needs to be calibrated. The specific method of calibration is as follows:

[0093] The binocular camera used in the experiment was produced by Corecom Technology, and its parameters are shown in Table 1.

[0094] Table 1. Parameters of stereo camera

[0095] Items Parameters Pixel 3840x1080 same frame synchronization 4 million pixels Pixel size 3umx3μm Frame rate 60fps Baseline 60mm

[0096] In order to calculate the relative position relationship between the image coordinate system and the world coordinate system, the binocular camera needs to be calibrated to obtain the camera's intrinsic and extrinsic parameters. The camera calibration methods mainly include traditional calibration method, self-calibration method, active vision calibration method and Zhang Zhengyou calibration method. Among them, Zhang Zhengyou calibration method first uses a linear model to find the optimal solution of the camera's intrinsic and extrinsic parameters, and then uses the maximum likelihood estimation method to accurately solve the camera's intrinsic and extrinsic parameters nonlinearly. It has the advantages of simple operation and high accuracy, and is widely used in binocular camera calibration.

[0097] Therefore, the present invention adopts Zhang Zhengyou calibration method and calibrates the binocular camera through the Stereo Camera CalibratorAPP built into MATLAB A. First, the binocular camera is used to obtain the chessboard calibration plate image pair, and then the corner points of each group of images are extracted. In the process of extracting the corner points, the calibration chessboard should be allowed to occupy 50% to 80% of the entire picture as much as possible to ensure accurate measurement of the corner points, while leaving some space at the edges and corners of the image to prevent the severely distorted areas from affecting the calibration results.

[0098] In order to obtain better calibration parameters and reduce radial distortion coefficient and tangential distortion, this paper uses calibration paper and calibration board for calibration. The calibration data are as follows: Figure 2 and Figure 3 As shown, the average back-projection errors of the calibration paper and the calibration plate are 0.41 pixels and 0.15 pixels respectively. The calibration effect of the calibration plate is obviously better than that of the calibration paper. The calibration plate is used for calibration in this experiment.

[0099] The basic principle of Zhang Zhengyou’s calibration method is to calculate the camera’s intrinsic parameters (including focal length, principal point, distortion coefficient) and extrinsic parameters (rotation and translation of the camera relative to the chessboard) through multiple chessboard images of different perspectives, using the camera projection model and linear algebra methods, providing a reliable foundation for subsequent visual algorithms. The parameters of the calibrated stereo camera are shown in Table 2.

[0100] Table 2 Binocular camera parameters

[0101]

[0102] Furthermore, in the present invention, after the binocular camera is calibrated, it needs to be corrected. The specific correction method is as follows:

[0103] The original image acquired by the binocular camera is stereo rectified using the calibrated camera internal and external parameters. Figure 4 This is the original binocular image before correction. It can be seen from the figure that the original image is distorted before correction, the picture is curved, and the farther away from the center of the image, the more serious the distortion. Figure 5 This is the image after correction. It can be seen from the figure and the horizontal line that the distortion caused by the lens is basically eliminated, the epipolar lines of the left and right images are parallel to each other, the longitudinal coordinates of the corresponding points are consistent, and the corresponding pixels have the same position in the vertical direction, which makes it simpler and more effective to match the corresponding points in the subsequent experimental process.

[0104] After the binocular camera is calibrated, the depth map needs to be generated, as follows:

[0105] The result of stereo matching algorithm is the disparity of the scene. The disparity indicates the difference in horizontal displacement of pixels in the left and right images. The larger the disparity value, the closer the object is to the camera, and the smaller the disparity value, the farther the object is from the camera. Through the disparity and the known internal and external parameters of the binocular camera, the depth information and depth map can be calculated, thereby calculating the distance from the object to the camera.

[0106] BM (Block Matching) and SGBM (Semi-Global Block Matching) are the two main methods in stereo matching algorithms. The BM algorithm is simple to calculate, fast, and suitable for real-time applications, but it performs poorly in weak texture areas and edges. SGBM combines local and global matching, improves matching accuracy through multi-directional cost aggregation, and can effectively handle weak texture and edge problems, but has higher computational complexity and is suitable for application scenarios with high accuracy requirements. Based on the OpenCV library, this paper performs BM and SGBM stereo matching on the pre-processed test grain box loading state images, and analyzes the performance and efficiency of the two algorithms in this experiment.

[0107] In this test, it is necessary to measure the distance from the bottom of the box and the surface of the piled materials to the binocular camera.

[0108] like Figure 6 As shown, this image is the original image obtained by the left and right cameras when the grain box is empty. Figure 7 They are the depth map and pseudo-color map after BM and SGBM stereo matching when the grain tank is empty. The pseudo-color map enhances the visual effect and data readability of grayscale images such as depth maps through color mapping technology.

[0109] In addition, in the present invention, the left view and the right view of the grain box three different loading state image pairs in the actual harvesting project are matched using the BM and SGBM stereo matching algorithms. Figure 8 As shown, the depth map and pseudo-color map obtained by stereo matching are as follows Fig. 9 shown.

[0110] In addition, from Figure 7 and Fig. 9 It can be seen that both BM and SGBM matching algorithms can restore the disparity information of binocular images to a certain extent. The depth map of the BM algorithm has many holes, but the overall features of the main contour of the grain box and the information on the height of the material stacking wheel in the grain box are relatively clear, but there are a lot of holes on the ground without depth information; the depth map of the SGBM algorithm has relatively few holes and is less affected by light. There are no large areas of holes on the ground, but a black boundary appears at the junction of the boundary contour of the grain box and the ground.

[0111] The binocular measurement results and analysis are as follows:

[0112] After obtaining the parallax of the left and right images, the image pixel distance can be restored according to the binocular ranging principle to obtain the distance information of each pixel point, and finally obtain the distance from the stacked materials in the test grain box to the binocular camera.

[0113] This experiment uses empty and loaded potato test grain boxes as the target objects, and measures the distance between the bottom of the empty box and the surface of the stacked potatoes to the binocular camera. The camera remains at a constant height relative to the box and the ground, and the distances at different positions in the box are measured by moving the box back and forth. Write a program to click on the pixel points on the depth map, calculate the distance information of the corresponding pixel points and output it. In order to verify the ranging accuracy of the BM and SGBM algorithms, the experiment still uses two different algorithms, BM and SGBM. The materials used in this experiment are potatoes grown in Inner Mongolia. The empty state of the grain box and three different states of potato stacking in the grain box are set respectively. The distance between the potato surface and the binocular camera at 10 different positions is measured by moving the grain box, and the distance measured by the binocular camera is compared with the actual distance.

[0114] When the truck is empty, the distance measurement results of the two algorithms between the binocular camera and the bottom of the grain box are shown in Table 3.

[0115] Table 3 Distance measurement results between the binocular camera and the bottom of the grain box

[0116]

[0117]

[0118] When no-load, the average errors measured by the BM algorithm and the SGBM algorithm are shown in Table 4.

[0119] Table 4

[0120] algorithm Average distance / mm Mean absolute error / mm Average relative error (%) Bm algorithm 1427 66.7 4.90％ SGBM algorithm 1407 47.2 3.47％

[0121] When the truck is loaded, the distance between the potato surface and the binocular camera is measured at 10 different locations under three different stacking conditions. The results are shown in Tables 5, 6, and 7.

[0122] Table 5 Status 1

[0123]

[0124]

[0125] Table 6 State 2

[0126] Number of measurements Measured BM SGBM BM Error SGBM Error 1 1104 1112 1109 0.72 0.45 2 1167 1178 1174 0.94 0.59 3 1174 1192 1184 1.53 0.85 4 1203 1224 1218 1.74 1.24 5 1232 1257 1248 2.02 1.29 6 1240 1274 1259 2.74 1.53 7 1255 1293 1277 3.02 1.75 8 1260 1299 1289 3.09 2.30 9 1290 1332 1328 3.25 2.94 10 1342 1398 1385 4.17 3.20

[0127] Table 7 Status 3

[0128]

[0129]

[0130] In order to more intuitively reflect the accuracy of binocular measurement, the relationship between the actual target distance and the binocular measurement distance and absolute distance is visualized respectively, such as Fig.10 shown.

[0131] From Table 5, Table 6, Table 7 and Fig.10 From (b), (c), and (d), we can see that at close distances, the error between the distance measured by the binocular camera and the actual distance is relatively small; however, as the distance increases, the error gradually increases and the accuracy gradually decreases. This is because when the target is at a close range, the camera can capture more effective pixels, thereby matching a clearer depth map, so the ranging accuracy is higher; on the contrary, when the object is far away from the camera, the camera's shooting range is relatively wide, and the space in its image is relatively complex, which will lead to an inaccurate depth map obtained by matching, thereby increasing the error of the distance information and reducing the number of effective pixels, ultimately resulting in poor matching results and reduced ranging accuracy.

[0132] From Table 3, Table 4 and Fig.10 As shown in (a), in the multiple measurements of 1360 mm, the average relative errors of the BM and SGBM algorithms selected in this experiment are 4.90% and 3.47%, respectively, which reflects the stability and reliability of the two algorithms in this experiment. The maximum relative errors are less than 6%, which also reflects the high precision of the two algorithms.

[0133] In order to further determine the reliability and performance of this experimental research method, the measurement time of the two algorithms was calculated respectively. The final results are shown in Table 8. It can be found from Table 8 that the matching time of the two algorithms is relatively low, which can meet the requirements of real-time ranging.

[0134]

[0135]

[0136] Table 8 Measurement time

[0137] In this test, within the measurement distance range of 1360mm, sgbm is equivalent to bm with higher accuracy and longer time. The average accuracy of the sgbm algorithm is 1.43% higher than that of the bm algorithm. In terms of calculation time, the average calculation time of the sgbm algorithm is 273ms slower than that of the bm algorithm. The errors of both algorithms are kept below 7% and the time is less than 321ms. It can be seen that the image processing speed of the BM algorithm is faster, the computational efficiency of image stereo matching is higher, and it is suitable for use in fields with high requirements for image processing speed; the depth map of the SGBM algorithm has fewer environmental noise points and holes, the depth map imaging effect is clearer and more obvious, the ranging accuracy is higher, the error is smaller, and it is suitable for use in fields with high requirements for ranging accuracy. Both algorithms meet the requirements of this test. Due to the influence of external environmental factors such as the speed of the harvesting vehicle, which method is finally adopted depends on the specific harvesting conditions in the field.

[0138] In addition, in the present invention, it is necessary to measure and analyze the material stacking height of the grain tank of the grain transport vehicle, as follows:

[0139] like Fig.11 As shown in the figure, this is a schematic diagram of measuring the stacking height of the grain box. By measuring the distance D between the bottom of the box and the binocular camera when empty and the distance d between the surface of the stacked potatoes and the binocular camera when loaded, the stacking height H at different positions when loaded can be measured by formula (3). The measurement results can provide loading information for the driver's next decision, as shown below:

[0140] H=Dd\Formula (3);

[0141] In this experiment, the average value of the distance between the bottom of the car box and the binocular camera when it is empty is selected as D, and the distance d between the surface of the piled potatoes and the binocular camera when loaded in state 1 is selected. Finally, the potato pile height H at different positions and the errors of the two algorithms are calculated as shown in Table 9.

[0142] In statistics, the error in a variable causes the function to also contain errors, which is called error propagation. In this experiment, the errors in the D and d data propagated in the calculation of H. Since the binocular camera used in this experiment is low-cost, has a small focal length, a small baseline, and low pixels, the image quality obtained is not high, making it difficult to obtain accurate distances, especially after mathematical calculations of the measurement data, the error is even greater.

[0143] In order to make the final measurement results more accurate, this experiment performed a linear regression analysis on the measured value and the true value of the potato pile height H, and obtained a calibration model. In this experiment, a total of 30 sets of true values ​​of the distance from the binocular camera to the potato pile surface and the distance to the bottom of the car box and the side measurement values ​​of the two algorithms were obtained. According to formula (3), 30 sets of potato pile height data can be obtained. In order to evaluate the linear regression equation, the potato pile heights are arranged from small to large, as follows: Fig.12 15 groups of data were evenly selected to fit the regression equation, and another 15 groups of data were used to evaluate and test the performance of the potato pile height correction model. The scatter plots, trend lines and test points are shown in Fig.13 shown.

[0144] Depend on Fig.13 The calibration model can be evaluated by 15 sets of data in the test group. The performance evaluation is shown in Table 9 and the residual graph is shown in Fig.14 shown.

[0145] From Table 9 and Fig.14 It can be seen that the correlation coefficient (r) of the correction model of the Bm algorithm and the SGBM algorithm is 0.9984, indicating that there is a very strong linear correlation between the prediction results of the two algorithms and the actual values; the determination coefficient (R 2 ) is 0.9968, and the determination coefficient (R 2 ) is 0.9963. Both are close to 1, indicating that the two algorithms fit the data very well; the MSE of the Bm algorithm correction model is 17.4618, while the MSE of the SGBM algorithm is 20.4469; the mean square error of the Bm algorithm is smaller, indicating that the difference between its prediction results and the actual values ​​is smaller, that is, the prediction accuracy of the Bm algorithm is higher; the MAE of the Bm algorithm correction model is 3.7637, and the MAE of the SGBM algorithm correction model is 3.4344. Although the Bm algorithm performs better in MSE, the SGBM algorithm correction model is slightly better in MAE, indicating that the SGBM algorithm correction model is slightly lower in the average value of absolute error; the correction model residuals of the two algorithms are randomly distributed near 0, without obvious patterns or trends, so the model fits the data reasonably.

[0146] In general, the potato stacking heights calculated by the Bm algorithm and the SGBM algorithm showed high accuracy after linear regression correction, and the correlation coefficient and determination coefficient were close to 1. Although the Bm algorithm performed better in MSE, the SGBM algorithm had an advantage in MAE. When selecting a specific algorithm correction model, you can choose an appropriate error measurement standard according to the needs of the specific application scenario. In the actual harvesting process, you can choose according to the size of the truck box and the physical properties of the material. If you are more sensitive to large errors, you can choose an algorithm with a lower MSE; if you are concerned about the average error, you can choose an algorithm with a lower MAE.

[0147] In order to verify the improvement of measurement accuracy by the correction model, the test group data was selected to compare the stacking height data before and after correction. The comparison results are as follows: Fig.15 shown.

[0148] In addition, in the present invention, Fig.15 It can be seen that when the stacking height is small, the errors before and after correction are relatively large. This is due to the definition of relative error and the nature of the value. The relative error is the ratio of the absolute error to the actual value. Therefore, when the actual value is small, the denominator is small, making the ratio (relative error) larger. At the same time, the application scenario is a grain tank transport vehicle, and the height of the tank is much greater than Fig.15 In order to better analyze the effects before and after correction, the first few groups of data in the table are discarded, and the average relative errors of the stacking heights are compared by excluding the data with stacking heights below 30 mm. The results are shown in Table 10.

[0149] Depend on Fig.15 As shown in Table 10, after calibration, both the calibration values ​​processed by the BM algorithm and the SGBM algorithm are closer to the true value, and their relative errors are greatly reduced. After calibration, the average relative error of BM is 3.70%, and the average relative error of SGBM is 3.35%. The average relative errors of the two algorithms are less than 4%. The average relative error of BM is 29.75% lower than that before calibration after calibration, and the average relative error of SGBM is 21.86% lower than that before calibration. The error reduction of the BM algorithm for smaller values ​​is more significant after calibration, and the error reduction effect of the SGBM algorithm for larger values ​​is better after calibration.

[0150] In general, the measurement results after linear regression model correction are more accurate, which greatly reduces the measurement error, thereby improving the reliability and accuracy of the data. At the same time, it can provide more accurate input for the algorithm, thereby improving the overall performance of subsequent data processing and analysis.

[0151] The present invention proposes a method and optimization study on the material stacking height measurement of the grain tank of a grain transporter based on binocular vision theory, which includes five steps: binocular camera parameter calibration, depth map calculation, binocular distance measurement, material stacking height calculation, and linear regression model optimization measurement data. The method uses Zhang Zhengyou calibration method to calibrate the binocular camera, and uses BM algorithm and SGBM algorithm to perform stereo matching and distance measurement on binocular images, and then proposes a material stacking height measurement method and linear regression model optimization. The research results show that: in multiple measurements of the bottom of the car box, the average relative errors of the BM and SGBM algorithms are 4.90% and 3.47% respectively; when measuring different distances, the errors of the two algorithms are kept below 7%, and the time is less than 321ms, which meets the basic requirements of binocular ranging; after linear regression correction, the average relative error of BM is calibrated and reduced by 29.75% compared with that before calibration, and the average relative error of SGBM is calibrated and reduced by 21.86% compared with that before calibration, indicating that the model has a very good data fitting effect; the average relative error of BM is 3.70%, and the average relative error of SGBM is 3.35%. The average relative errors of the two algorithms are less than 4%, which meets the accuracy requirements for material stacking height measurement during the combined harvesting process. The research on the measurement method and optimization of the material stacking height of the grain tank of the grain transporter based on binocular vision theory has obvious advantages over manual visual inspection in the application scenario of unloading grain in the combined harvesting, and has high reliability and accuracy.

[0152] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for measuring the material stacking height of a grain tank of a grain transport vehicle based on binocular vision, characterized in that: The steps include: S1. Build a binocular vision system, including two cameras with the same parameters, and install them on the grain transport vehicle. Take pictures of the grain box at a fixed angle. Through the parallax caused by the position of the camera and the difference in the pixel coordinate values ​​of the left and right images, calculate the spatial coordinates of the feature points through triangulation, obtain the depth information of the measured target, and locate and measure the object; S2. Calibrate and correct the binocular camera parameters, use Zhang Zhengyou calibration method to obtain the camera's internal and external parameters, and eliminate image distortion; S3. Obtain left and right images of the material in the grain box through a binocular camera, calculate the disparity map using a stereo matching algorithm, and then obtain a depth map; S4. According to the depth map and the position parameters of the binocular camera, the three-dimensional coordinates of the bottom of the grain box and the material accumulation surface are calculated to obtain the material accumulation height.

2. The method for measuring the material stacking height of a grain tank of a grain transport vehicle based on binocular vision according to claim 1 is characterized in that: The stereo matching algorithm includes a BM algorithm or a SGBM algorithm.

3. The method for measuring the material stacking height of a grain tank of a grain transport vehicle based on binocular vision according to claim 2 is characterized in that: The S1 specifically comprises the steps of: S11: Build a binocular vision system, including two cameras with the same parameters, and install them on the grain transport vehicle to shoot the grain box at a fixed angle; S12: Let XO1YZ be the world coordinates, coincident with the left camera coordinate system, O1 be the optical center of the left camera, O2 be the optical center of the right camera, the focal lengths of the two cameras be f, the distance b between the optical centers of the left and right cameras be the baseline of the binocular camera, and let the three-dimensional coordinates of point P be (x, y, z); S13: When a binocular camera is used to shoot point P, the points formed by the images of point P on the left and right camera planes are recorded as P1(x1, y1) and P2(x2, y2) respectively. The three-dimensional coordinates of point P are obtained based on the formula of similar triangle principle: Among them, d is the difference between the horizontal coordinates of points P1 and P2 imaged by the left and right cameras.

4. The binocular vision-based method for measuring the material stacking height of a grain tank of a grain transport vehicle according to claim 2 is characterized in that: The S2 specifically includes the steps of: S21: First, use a binocular camera to obtain a pair of chessboard calibration images, and then extract the corner points of each set of images. In the process of extracting corner points, the calibration chessboard is allowed to occupy 50% to 80% of the entire image, and space is reserved at the edges and corners of the image to prevent areas with severe distortion from affecting the calibration results. S22: Calculate the intrinsic and extrinsic parameters of the camera using multiple checkerboard images from different perspectives using the camera projection model and linear algebra methods; S23: After the binocular camera is calibrated, the original image acquired by the binocular camera is stereo corrected using the internal and external parameters of the camera obtained by the calibration to eliminate image distortion; S24: Calculate the depth information and the depth map through the parallax and the known internal and external parameters of the binocular camera, and calculate the distance from the object to the camera.

5. The method for measuring the material stacking height of a grain tank of a grain transport vehicle based on binocular vision according to claim 2 is characterized in that: The S3 specifically includes the steps of: S31: After obtaining the parallax of the left and right images, restore the image pixel distance according to the binocular distance measurement principle to obtain the distance information of each pixel point, and finally obtain the distance from the stacked materials in the test grain box to the binocular camera; S32: By measuring the distance D between the bottom of the vehicle box and the binocular camera when empty and the distance d between the surface of the piled potatoes and the binocular camera when loaded, the stacking height H at different positions when loaded is finally measured by a preset formula.

6. The binocular vision-based method for measuring the material stacking height of a grain tank of a grain transport vehicle according to claim 5 is characterized in that: In S32, the preset formula is: H = Dd; Where H is the stacking height, D is the measured distance between the bottom of the truck box and the binocular camera when empty, and d is the distance between the surface of the stacked grain box and the binocular camera when loaded.

7. The binocular vision-based method for measuring the material stacking height of a grain tank of a grain transport vehicle according to claim 4 is characterized in that: In S22, the intrinsic parameters of the camera are calculated including focal length, principal point, and distortion coefficient.

8. The binocular vision-based method for measuring the material stacking height of a grain tank of a grain transport vehicle according to claim 4 is characterized by: In S22, the external parameters include rotation and translation values ​​of the camera relative to the chessboard.

9. The method for measuring the material stacking height of a grain tank of a grain transport vehicle based on binocular vision according to claim 1 is characterized in that: The method of calculating the three-dimensional coordinates of the bottom of the grain box and the material accumulation surface based on the depth map and the position parameters of the binocular camera and obtaining the material accumulation height includes the following steps: S5. Perform linear regression model correction on the measurement results.

10. The binocular vision-based method for measuring the material stacking height of a grain tank of a grain transport vehicle according to claim 3, characterized in that: In S13, the similar triangle principle formula is:

Images