Real-time detection method of header height of rice and wheat harvester based on binocular vision of barcode

Abstract

A method for real-time detection of the header height of a rice and wheat harvester based on a barcode and binocular vision is provided. After the stationary feature barcode of the rotating header shaft is marked, a vehicle-mounted rapid real-time distance measurement based on binocular vision sparse feature matching is performed. The present invention combines a feature marking method based on a barcode and a vehicle-mounted real-time distance measurement method based on binocular vision sparse feature matching, which can quickly and effectively identify the rotation center axis position of the header reel and obtain the accurate header height in real time.

Description

Technical Field

[0001] The present invention relates to a technology in the field of environmental perception detection of unmanned rice and wheat harvesting equipment, in particular to a real-time detection method for the height of a cutting platform of an unmanned rice and wheat harvester based on binocular vision. Background Art

[0002] Intelligent and unmanned operation is the current research hotspot and development trend of agricultural machinery equipment. Existing full-feed harvesters often encounter crop lodging during the harvesting process. To ensure the quality of harvesting operations, unmanned intelligent harvesters need to automate the real-time adjustment of the cutting table height. In addition to perceiving the height of the crops, they are also required to be able to detect the height of the cutting table in real time, and compare the feedback information of the detected cutting table height with the crop height information input into the control system to achieve closed-loop control of the cutting table height.

[0003] The existing harvesting machinery cutting table height detection is mainly based on the angle sensing profiling device, which is used to realize the profiling tracking of uneven terrain, and cannot be used to detect the cutting table height relative to the crop height in the case of lodging; the traditional mechanical cutting table height detection method cannot meet the detection needs of the cutting table height during harvesting operations in the case of rice and wheat lodging, and the cutting table has many irregular shapes, and it is difficult to select the feature points of conventional visual detection methods in real time. For this reason, this patent invents a barcode feature position mark of the moving cutting table shaft and a real-time detection method of the cutting table height of a rice and wheat harvester based on binocular vision. Summary of the invention

[0004] In view of the problem that the prior art cannot accurately obtain the harvester header height data, the present invention proposes a real-time detection method for the header height of a rice and wheat harvester based on binocular vision using a barcode. The method combines a feature marking method based on a barcode and a vehicle-mounted real-time ranging method based on binocular vision sparse feature matching, which can quickly and effectively identify the rotation center axis position of the header wheel and obtain the accurate header height in real time.

[0005] The present invention is achieved through the following technical solutions:

[0006] The invention relates to a real-time detection method for the cutting platform height of a rice and wheat harvester based on a binocular vision with a barcode, which performs vehicle-mounted rapid real-time distance measurement based on binocular vision sparse feature matching after marking the stationary feature barcode of a rotating shaft of the cutting platform.

[0007] The method specifically comprises:

[0008] Step 1: Paste three different barcodes at different positions on the central axis of the reel of the harvester header;

[0009] The barcode is composed of parallel black and white stripes and is used to mark the characteristic mark on the rotation center axis of the reel and increase the image characteristic points of the rotation center axis.

[0010] The sticking specifically refers to sticking the barcode vertically around the rotation center axis to ensure that the marked features of the rotation center axis in the image remain unchanged during the rotation process.

[0011] The barcodes preferably use three different barcodes, which can reduce the influence of light on binocular vision recognition of barcode features.

[0012] Step 2: Car-mounted fast real-time ranging based on binocular vision sparse feature matching, specifically including:

[0013] Step 2.1: barcode mark recognition based on color and gradient fusion, specifically including:

[0014] 2.1.1. The image with color feature enhancement and the image with gradient feature enhancement in the barcode area are fused to obtain a feature enhanced image. The image fusion is as follows: e =w C ×I C +w G ×I G , where: I e Enhance the grayscale image for the fused features; I C is the color feature enhanced image, i.e., the R component image; I G is an image enhanced with gradient features, i.e., the gradient difference map in the x and y directions; w C The fusion weight of the color feature enhanced image, w G Fusion weights for gradient feature enhancement images.

[0015] The fusion weight of the color feature enhanced image is preferably 0.65.

[0016] The fusion weight of the gradient feature enhanced image is preferably 0.35.

[0017] The color feature enhanced image is obtained by extracting three-channel values ​​of the RGB color space from the image. Since the contrast between the barcode mark area and the surrounding environment in the B component image is most obvious, the B component image is extracted to achieve color feature enhancement.

[0018] The characteristic feature enhanced image is obtained by the following method: using the Sobel operator to calculate the gradient values ​​of the image in the x direction and the y direction, and then subtracting the gradients in the two directions to obtain the gradient extreme difference map in the x direction and the y direction to achieve gradient feature enhancement.

[0019] 2.1.2. Perform image morphological processing and add specific constraints on the feature enhanced image to complete the extraction of the barcode mark area in the image. The obtained barcode area is used for the distance detection method based on SURF feature barcode feature stereo matching to reduce the calculation of matching feature points that are not related to the header height.

[0020] The image morphological processing operations include: (i) performing mean filtering on the feature enhanced image to eliminate high-frequency noise; (ii) performing Otsu binarization to obtain a binary image; (iii) performing a closing operation to fill the gaps between similar white areas; (iv) performing two erosions to remove isolated small areas; (v) performing two dilations to fill the gaps between barcodes and restore the size of the barcode area; (vi) making a header mask: extracting the H component channel map in the HSV color space of the image, and then performing binarization and median filtering to obtain a mask; (vii) performing mask covering on the image after the two dilations, that is, performing an AND operation.

[0021] The specific constraints include: (i) the size constraint of the barcode mark, which assumes that the length of the barcode mark is between 25 pixels and 50 pixels, and the width is between 20 pixels and 35 pixels; (ii) the shape constraint of the barcode mark, which assumes that the ratio of the length to the width of the barcode mark should be between 1.0 and 2.5; (iii) the shape constraint of the barcode mark, which assumes that the horizontal coordinates of the two ends of the barcode mark rectangle should be between 600 pixels and 1600 pixels.

[0022] Step 2.2: Distance detection based on stereo matching of SURF feature barcode features, specifically including:

[0023] 2.2.1. Use the SURF algorithm for feature extraction. This algorithm can meet the application scenarios of real-time feature matching of binocular images of outdoor farmland, including:

[0024] a) Construct the Hessian matrix: The Hessian matrix can be used to describe the local curvature of a multivariate function. For an image I(x,y), its Hessian matrix is: Where: I refers to the image I(x,y);

[0025] The SURF algorithm determines whether it is a local feature point by calculating the value of the Hessian matrix discriminant of each pixel in the image. The Hessian matrix discriminant is: Where: I refers to the image I(x,y).

[0026] b) Constructing scale space: In order to ensure the scale invariance of the SURF algorithm, Gaussian filtering is performed before constructing the Hessian matrix to achieve the construction of the scale space. Different scales are obtained by changing the size of the filtered template. Under the scale, the Hessian matrix is: Where: L xx (x,y,σ)—I(x,y) with the second-order Gaussian template Do convolution, L xy (x,y,σ);L yy (x, y, σ) is similar;

[0027] When the SURF algorithm is actually calculated, this convolution process is approximated using a box filter to increase the calculation speed by looking up elements in the integral image.

[0028] c) Locating feature points: Compare the discriminant value of the Hessian matrix of each pixel with all the adjacent points in its two-dimensional image space and the two adjacent scale space neighborhoods. When it is greater than or less than all the adjacent points (that is, the point is an extreme point), it is initially considered to be a key point. Then filter out the key points that are less than a certain threshold, and finally locate the stable feature points.

[0029] d) Direction allocation of feature points: Direction allocation is performed based on the Haar wavelet feature. That is, within the circular neighborhood of the feature point, the sum of the horizontal and vertical Haar wavelet features of the sampling points at regular intervals within a 60° sector is counted, and the sector is rotated at intervals of 15°. After one rotation, the sector direction corresponding to the maximum value is taken as the main direction of this feature point;

[0030] e) Generate feature descriptors: Extract a square neighborhood around the feature point along the main direction of the feature point. The side length of the area is proportional to the scale. Divide this neighborhood into 16 4×4 sub-areas in total. Each sub-area counts four Haar wavelet features: the sum of the horizontal direction value, the sum of the vertical direction value, and the sum of the absolute value of the horizontal direction value. That is, each sub-area will obtain a 4-dimensional feature vector, and a SURF feature descriptor will eventually obtain a 64-dimensional vector. After the above five steps, the SURF algorithm feature extraction result image is obtained.

[0031] 2.2.2. The random kd tree algorithm in the fast nearest neighbor search package is used for feature matching. The left and right images captured by the binocular camera are used as the training set and the query set respectively, and the matching feature points in the barcode mark area obtained by the barcode mark recognition method based on color and gradient fusion are screened out.

[0032] 2.2.3. Obtain the parallax of the matching feature points in the left and right cameras (i.e., the pixel difference in the x direction) and calculate the 3D coordinates of the matching feature points And according to the camera installation external parameters, the coordinate system is transformed to obtain the height of each pair of matching points in the ground coordinate system. The coordinate system transformation is as follows: Where: c x 、c y 、f x 、f y , b l All obtained by camera calibration, O C -X C Y C Z C The coordinate system is the camera coordinate system, O G -X G Y G Z G The coordinate system is the ground coordinate system, and the relationship between the camera coordinate system and the ground coordinate system satisfies Where: R is the rotation matrix, which is calculated by the camera's built-in IMU module; t is the translation matrix, which is obtained by manual measurement.

[0033] Technical Effects

[0034] Since the frame and swing arm in the reel structure move separately during operation, the traditional method of side-mounting the tilt sensor is difficult to obtain accurate height data for the reel with two degrees of freedom. The method of using a stationary feature barcode marking method on a rotating object can effectively identify the unique rotation center axis position of the harvester reel. The barcode mark recognition method based on color and gradient fusion allows mark matching and feature point position calculation only in the marked area. Therefore, this method has accurate detection and low computational overhead. In addition, three different barcodes are pasted at different positions of the rotating axis, which can effectively reduce the influence of light on barcode recognition and enhance the robustness of the method for extracting barcode mark areas in this environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the system of the present invention;

[0036] Figure 2 It is a schematic diagram of a method for marking a stationary feature barcode on a rotating object;

[0037] Figure 3 This is the principle diagram of the real-time ranging method based on binocular vision sparse feature matching. DETAILED DESCRIPTION

[0038] This embodiment uses a Kubota 4LZ-4J (PRO988Q-Q) full-feed crawler harvester, a ZED 2 binocular camera as a visual sensor (side-by-side resolution) to obtain image information, and an NVIDIA Jetson TX2 product as an algorithm processor. Some functions in the computer vision open source library OpenCV are used for image processing, and the xfeatures2d module in the OpenCV_contrib extension package is used for feature matching; the fast nearest neighbor search package (FLANN) is used for feature matching. FLANN is a collection of algorithms for nearest neighbor search for large data sets and high-dimensional features, including random kd tree algorithm, priority search k-means tree algorithm, and hierarchical clustering tree algorithm. This method uses a random kd tree algorithm.

[0039] like Figure 1 As shown, the present embodiment relates to a real-time detection system for the header height of a rice and wheat harvester based on a barcode and binocular vision, comprising: a barcode marking module, a binocular camera acquisition module and an information processing module, wherein: the barcode marking module is located on the reel in the header structure, and is used to implement a stationary barcode marking method on a rotating moving object; the binocular camera acquisition module is located on the harvester cab, and the camera is installed according to the lifting height of the header to ensure that the header can be captured by the camera at both the highest position and the lowest position; the information processing module is connected to the binocular camera module, and is used to implement a real-time ranging method based on binocular vision sparse feature matching.

[0040] like Figure 2 As shown, the reel includes: a central rotation axis and a plurality of lateral rods parallel thereto, which are used to lift the crop and push the crop toward the cutting knife, and to support the stems during cutting. This structure is particularly important when the crop falls over, and is also the basis for adjusting the height of the cutting table. In order to use a binocular camera to accurately identify the unique rotation center axis from the multiple lateral rods, the technical details of the barcode used to mark the stationary features on the rotating moving object can be: a width of 6 cm, a black and white color, basically parallel to the width direction of the camera image, and pasted around the axis at three trisection points on the unique rotation center axis in the reel. The purpose of using three barcode marks is to reduce the influence of light on feature recognition. The purpose of pasting the barcode around the axis is to ensure that the image features captured by the camera remain unchanged during the rotation of the rotation center axis.

[0041] like Figure 3As shown, this embodiment relates to a real-time ranging method based on binocular vision sparse feature matching based on the above system, through barcode mark recognition based on color and gradient fusion and real-time ranging based on binocular vision sparse feature matching, wherein: on the one hand, the barcode mark recognition method based on color and gradient fusion first enhances the color and gradient features of the acquired image, and performs image fusion to obtain a feature enhanced image, and extracts the marked area on the feature enhanced image, and the extracted barcode mark area is used in the next step to filter out the matching feature points in other areas of the image that have no value with the height information of the rotation center axis, and directly obtain the matching feature points in the barcode mark area that have valuable information with the height information of the rotation center axis. On the other hand, the real-time ranging method based on binocular vision sparse feature matching uses the SURF algorithm for feature extraction, uses the random kd tree algorithm for feature matching, filters the obtained matching feature points, and only leaves the matching feature points in the barcode mark area for calculating the three-dimensional coordinates, and finally obtains the height of the ground coordinate system of the matching feature points through coordinate system conversion.

[0042] In actual operation, in addition to operating the lifting and lowering of the header frame, the harvester will often operate the swing arm mechanism that can be raised and lowered relative to the header frame and the reel mechanism that can rotate relative to the swing arm. It is difficult for sensing elements such as inclination sensors and laser sensors to locate this complex situation, so the problem is decomposed based on binocular vision sensors. The method of using stationary feature barcode marking on rotating objects can effectively identify the unique rotation center axis position of the header reel. The barcode mark recognition method based on color and gradient fusion allows mark matching and feature point position calculation only in the marked area. Therefore, this method has accurate detection and low calculation overhead.

[0043] The accuracy of this method was tested. The experimental platform was used to test the accuracy of this method under outdoor natural light environment. According to the working state of the header, two test tasks were designed: accuracy test at different height positions and accuracy test under the rotating state of the reel. A handheld laser rangefinder with an accuracy of 1mm was used to measure the real height of the header. First, the header of the harvester was gradually raised, and pictures of the header at 5 different positions were collected respectively. The height measured by this method was calculated, and the average height error of the 5 measurement results was 3.13cm. Then, the reel was rotated at a fixed header height, and pictures were collected for measurement. The average height error of the 5 measurement results during the rotation process was 2.45cm. According to the results of the two accuracy tests, it is believed that this method can achieve accurate measurement when the header is raised and lowered and when the header is rotated, and can meet the header height detection requirements during harvester operation.

[0044] In the harvesting fields of Jiading District, a field experiment was conducted using a modified Kubota harvester. The harvester performed harvesting operations at a speed of about 1.5m / s. The ZED 2 camera collected images in real time and the NVIDIA Jetson TX2 completed the processing. Under the experimental conditions, the recognition accuracy of the harvester barcode mark reached 93.3%. Under different lighting conditions and the influence of rice straw dust splashed during crop harvesting, it can basically ensure that there are height detection results in the two barcode mark areas.

[0045] Compared with the existing technology, this method verifies the average height error of the algorithm detection from the two aspects of header height lifting and reel rotation through precision inspection and harvesting operation practice, which is 3.13cm, and improves the measurement accuracy of the header height detection system. Under different lighting conditions and the influence of dust splashed during crop harvesting, it can basically ensure that the height detection results exist in the two barcode marking areas, which improves the adaptability of the header height detection system in various complex environments. The algorithm running time is 351ms, which can meet the needs of real-time operation of the harvester.

[0046] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A real-time detection method for the header height of a rice and wheat harvester based on barcode binocular vision, characterized in that: After the stationary feature barcode of the rotary harvesting table shaft is marked, the vehicle-mounted fast real-time distance measurement based on binocular vision sparse feature matching is carried out, including: Step 1: Paste three different barcodes at different positions on the central axis of the reel of the harvester header; Step 2: Car-mounted fast real-time ranging based on binocular vision sparse feature matching, specifically including: Step 2.1: Barcode mark recognition based on color and gradient fusion, Step 2.2, distance detection based on stereo matching of SURF feature barcode features; The barcode mark recognition based on color and gradient fusion specifically includes: 2.1.

1. The image with color feature enhancement and the image with gradient feature enhancement in the barcode area are fused to obtain a feature enhanced image. The image fusion is as follows: e =w C ×I C +w G ×I G , where: I e Enhance the grayscale image for the fused features; I C is the color feature enhanced image, i.e., the R component image; I G is an image enhanced with gradient features, i.e., the gradient difference map in the x and y directions; w C The fusion weight of the color feature enhanced image, w G Enhance the fusion weight of the image for gradient features; 2.1.

2. Perform image morphological processing and add specific constraints to the feature-enhanced image to extract the barcode mark area in the image. The obtained barcode area is used for the distance detection method based on SURF feature barcode feature stereo matching to reduce the calculation of matching feature points that are not related to the header height; The distance detection based on SURF feature barcode feature stereo matching specifically includes: 2.2.

1. Use SURF algorithm for feature extraction; 2.2.

2. The random kd tree algorithm in the fast nearest neighbor search package is used for feature matching. The left and right images captured by the binocular camera are used as the training set and the query set respectively, and the matching feature points in the barcode mark area obtained by the barcode mark recognition method based on color and gradient fusion are screened out; 2.2.

3. Obtain the parallax of the matching feature points in the left and right cameras, that is, the pixel difference in the x direction, and calculate the 3D coordinates of the matching feature points And according to the camera installation external parameters, the coordinate system is transformed to obtain the height of each pair of matching points in the ground coordinate system. The coordinate system transformation is as follows: Where: c x 、c y 、f x 、f y 、b l All obtained by camera calibration, O C -X C Y C Z C The coordinate system is the camera coordinate system, O G -X G Y G Z G The coordinate system is the ground coordinate system, and the relationship between the camera coordinate system and the ground coordinate system satisfies R is the rotation matrix, which is calculated by the camera's built-in IMU module; t is the translation matrix, which is obtained by manual measurement.

2. The method for real-time detection of header height of rice and wheat harvester based on barcode binocular vision according to claim 1 is characterized in that: The barcode is composed of parallel black and white stripes and is used to mark the characteristic mark on the rotation center axis of the reel and increase the image characteristic points of the rotation center axis.

3. The method for real-time detection of header height of rice and wheat harvester based on barcode binocular vision according to claim 1 is characterized in that: The sticking specifically refers to sticking the barcode vertically around the rotation center axis to ensure that the marked features of the rotation center axis in the image remain unchanged during the rotation process.

4. The method for real-time detection of header height of rice and wheat harvester based on barcode binocular vision according to claim 2 is characterized in that: The barcode described uses three different barcodes, which can reduce the influence of light on binocular vision recognition of barcode features.

5. The method for real-time detection of header height of rice and wheat harvester based on barcode binocular vision according to claim 1 is characterized in that: The color feature enhanced image is obtained by: extracting three-channel values ​​of the RGB color space from the image, and since the contrast between the barcode mark area and the surrounding environment in the B component image is most obvious, the B component image is extracted to achieve color feature enhancement; The gradient feature enhanced image is obtained by the following method: using the Sobel operator to calculate the gradient values ​​of the image in the x direction and the y direction, and then subtracting the gradients in the two directions to obtain the gradient extreme difference map in the x direction and the y direction to achieve gradient feature enhancement.

6. The method for real-time detection of header height of rice and wheat harvester based on barcode binocular vision according to claim 1 is characterized in that: The image morphological processing operation includes: (i) performing mean filtering on the feature enhanced image to eliminate high-frequency noise; (ii) performing Otsu binarization to obtain a binary image; (iii) performing a closing operation to fill the gaps between similar white areas; (iv) performing two erosions to remove isolated small areas; (v) performing two dilations to fill the gaps between barcodes and restore the size of the barcode area; (vi) making a header mask: extracting the H component channel map in the HSV color space of the image, and then performing binarization and median filtering to obtain a mask; (vii) performing mask covering on the image after the two dilations, that is, performing an AND operation; The specific constraints include: (i) the size constraint of the barcode mark, which assumes that the length of the barcode mark is between 25 pixels and 50 pixels, and the width is between 20 pixels and 35 pixels; (ii) the shape constraint of the barcode mark, which assumes that the ratio of the length to the width of the barcode mark should be between 1.0 and 2.5; (iii) the shape constraint of the barcode mark, which assumes that the horizontal coordinates of the two ends of the barcode mark rectangle should be between 600 pixels and 1600 pixels.

7. The method for real-time detection of header height of rice and wheat harvester based on barcode binocular vision according to claim 1 is characterized in that: The feature extraction using the SURF algorithm specifically includes: a) Construct the Hessian matrix: The Hessian matrix can be used to describe the local curvature of a multivariate function. For an image I(x,y), its Hessian matrix is: Where: I refers to the image I(x,y); the SURF algorithm determines whether it is a local feature point by calculating the value of the Hessian matrix discriminant of each pixel in the image. The Hessian matrix discriminant is: Where: I refers to the image I(x,y); b) Constructing scale space: In order to ensure the scale invariance of the SURF algorithm, Gaussian filtering is performed before constructing the Hessian matrix to achieve the construction of the scale space. Different scales are obtained by changing the template size of the filter; under the scale, the Hessian matrix is: Where: L xx (x,y,σ)-I(x,y) and the second-order Gaussian template Do convolution, L xy (x,y,σ);L yy (x, y, σ) is similar; When the SURF algorithm is actually calculated, this convolution process is approximated by a box filter, which increases the calculation speed by finding elements in the integral image. c) Locating feature points: Compare the discriminant value of the Hessian matrix of each pixel with all the adjacent points in its two-dimensional image space and the two adjacent scale space neighborhoods. When it is greater than or less than all the adjacent points (that is, the point is an extreme point), it is initially considered to be a key point; then filter out the key points that are less than a certain threshold, and finally locate the stable feature points; d) Direction allocation of feature points: Direction allocation is performed based on the Haar wavelet feature; that is, within the circular neighborhood of the feature point, the sum of the horizontal and vertical Haar wavelet features of the sampling points at regular intervals within a 60° sector is counted, and the sector is rotated at intervals of 15°. After one rotation, the sector direction corresponding to the maximum value is taken as the main direction of this feature point; e) Generate feature descriptors: extract a square neighborhood around the feature point along the main direction of the feature point, with the side length of the area proportional to the scale. Divide this neighborhood into 16 4×4 sub-areas in total. Each sub-area counts four Haar wavelet features: the sum of horizontal values, the sum of vertical values, and the sum of absolute values ​​of horizontal values. That is, each sub-area will obtain a 4-dimensional feature vector, and a SURF feature descriptor will eventually obtain a 64-dimensional vector. After the above five steps, the SURF algorithm feature extraction result image is obtained.

8. A real-time detection system for the height of a rice and wheat harvester header based on a barcode and binocular vision for implementing any of the methods described in claims 1 to 7, characterized in that: include: A barcode marking module, a binocular camera acquisition module and an information processing module, wherein: the barcode marking module is located on the reel in the header structure, and is used to implement a stationary barcode marking method on a rotating object; the binocular camera acquisition module is located on the harvester cockpit, and the camera is installed according to the lifting height of the header to ensure that the header can be captured by the camera when it is in the highest position and the lowest position; the information processing module is connected to the binocular camera module, and is used to implement a real-time ranging method based on binocular vision sparse feature matching.

Images