Binocular vision green vegetation matching and positioning method based on fusion of color features and edge features
By fusing color and edge features into a binocular vision method, the accuracy and efficiency issues of traditional binocular vision in locating green vegetation in agricultural environments have been solved, achieving low-cost and high-efficiency green vegetation location, which is suitable for mechanized agricultural operations.
Patent Information
- Application Number
- CN202111436002.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-29
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-11-29
AI Technical Summary
Traditional binocular vision stereo matching technology is greatly affected by visual and environmental factors when locating green vegetation in agricultural environments. It has high computational complexity, making it difficult to achieve real-time processing and is also costly, which cannot meet the needs of agricultural production.
A binocular vision method that integrates color and edge features is adopted. Through steps such as OpenCV calibration, image preprocessing, color space transformation, edge feature extraction, OTSU adaptive threshold segmentation, semi-global matching algorithm and disparity optimization, the rapid and accurate localization of green vegetation is achieved.
It improves the accuracy and robustness of green vegetation positioning, reduces computational complexity, is suitable for low-cost equipment, and is well-suited for mechanized agricultural operations, enabling rapid and efficient green vegetation positioning.
Smart Images

Figure CN114119718B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical fields of computer vision, digital images and agricultural intelligence, and in particular relates to a binocular vision green vegetation matching and positioning method integrating color features and edge features. Background Art
[0002] Traditional agricultural labor mainly relies on manual labor, which results in high labor intensity, low agricultural production efficiency, and poor results due to time-consuming and labor-intensive work. With the advancement of science and technology and the gradual advancement of precision agriculture, the use of agricultural machinery for precision operations has gradually become the mainstream of agricultural production. The premise of precision operations of agricultural machinery is the accurate spatial positioning of farmland vegetation. In the target space positioning method, it can be mainly divided into two categories: active sensing technology and passive sensing technology. Among them, active sensing technology mainly includes time of flight (TOF), structured light and lidar. TOF is to transmit a tempered light pulse to the target position through a sensor. After encountering an object, the receiver receives the reflected light pulse and calculates the distance to the target position based on the time difference between the light pulse being transmitted by the transmitter and received by the receiver. This method has high requirements for the transmitter and receiver, and is limited by transmitter power, transmission distance and resolution, and cannot be used on a large scale. Structured light is to emit well-tempered structured light pulses to the surface of the three-dimensional scene at the target position through the transmitter. In this process, the structured light pattern pattern will change accordingly according to the transmission distance. The depth information of the structured light pixel point can be calculated based on the change of the structured light pattern pattern using the triangulation principle. This method is easily affected by ambient light and is difficult to use in outdoor scenes. LiDAR technology is mature and has good adaptability to ambient light, transmission distance, etc., but its high cost determines that it cannot be widely used in the field of agricultural production. Binocular stereo vision is a typical passive sensing technology and an important research direction in the field of computer vision. Compared with active depth sensing technology, binocular stereo vision has the advantages of low cost, easy implementation, and no restrictions on the specific use environment. It plays an important role in target detection and positioning, medical image processing and diagnosis, industrial defect detection, three-dimensional information acquisition and three-dimensional reconstruction. The method of stereo vision is to obtain multiple image information from multiple different positions, use the triangulation principle to obtain the position deviation between image pixels, and restore the three-dimensional geometric information of the scene through parallax information. Binocular stereo matching is a hot topic in stereo vision research. It processes image information by simulating human eyes to obtain three-dimensional information of the surrounding environment. Traditional binocular vision stereo matching technology is relatively greatly affected by visual and environmental factors, has high computational complexity, is difficult to achieve real-time processing levels, the acquired depth values are inaccurate, and has poor processing of edge information. Summary of the invention
[0003] In order to make up for the gaps and shortcomings of the prior art, the present invention proposes a binocular vision green vegetation matching and positioning method that integrates color features and edge features to quickly and accurately locate farmland green vegetation. It has low requirements on acquisition equipment, computing processing platform equipment and image data quality. It improves the agricultural green vegetation positioning problem through specific digital image processing methods and can be applied to green vegetation positioning tasks in complex agricultural non-structured environments.
[0004] The technologies and processes adopted include: binocular camera parameter calibration, scene binocular image acquisition, image epipolar correction, image preprocessing to reduce the impact of illumination and enhance edge features, color space transformation and edge feature extraction, super green feature combined with OTSU adaptive threshold algorithm to segment binocular images, calculate the connected domain of green vegetation foreground target and the centroid coordinates of target image, Census change fusion color features and edge features to construct and initialize the matching cost function, calculate the matching cost of the left image and the right image according to the matching cost function, aggregate the matching cost by the semi-global matching algorithm (SGM), and the winner-takes-all (WTA) algorithm to determine the optimal disparity of each pixel in the cost matrix after aggregation and generate a disparity map, optimize the disparity map through disparity optimization methods such as left-right consistency check, median filtering, small connected domain elimination and disparity map filling, and combine the disparity map with the calibrated binocular camera parameters to generate a depth image. The spatial position coordinates of the vegetation are calculated according to the centroid coordinates of the vegetation target and the depth map to complete the spatial positioning of the green vegetation in the scene. This method can quickly and accurately locate green vegetation in outdoor unstructured complex agricultural scenes, improving the positioning accuracy and work efficiency of traditional stereo matching. It has high accuracy and robustness in complex agricultural scenes and low computational complexity, and is very suitable for tasks such as weeding and vegetable and fruit picking in mechanized precision agricultural production.
[0005] The present invention specifically adopts the following technical solutions:
[0006] A binocular vision green vegetation matching and positioning method integrating color features and edge features, characterized in that it comprises the following steps:
[0007] Step S1: Use the OpenCV binocular vision calibration algorithm to calibrate the binocular camera's intrinsic parameters, extrinsic parameters, and distortion parameters for epipolar correction and depth value measurement;
[0008] Step S2: using a binocular camera with calibrated parameters to collect field binocular image data in real time;
[0009] Step S3: performing epipolar correction on the acquired binocular image according to the calibrated binocular camera internal parameters, external parameters and distortion parameters, so that the matching pixel pairs on the binocular image are in the same row of the image, so as to reduce the search range of the matching algorithm;
[0010] Step S4: Use Gamma correction, adaptive histogram equalization and bilateral filtering to perform image preprocessing to reduce the influence of illumination and noise and enhance image edge detail information;
[0011] Step S5: converting the image processed in step S4 into a super-green feature channel image in the RGB color space to generate a super-green feature channel image; and detecting the edge information of the image using the Sobel edge detection algorithm to generate an edge feature image;
[0012] Step S6: using the OTSU adaptive threshold segmentation algorithm to calculate the binarization threshold of the super green feature channel image, and using the threshold segmentation method to segment the image to obtain foreground vegetation image information;
[0013] Step S7: using a connected domain labeling algorithm on the foreground vegetation image to calculate the foreground image connected domain, and calculating the image centroid coordinates corresponding to the foreground vegetation connected domain;
[0014] Step S8: constructing a fusion matching cost function to calculate the matching cost on the super green feature channel image, the Sobel edge feature image and the source image;
[0015] Step S9: using the semi-local matching algorithm SGM to aggregate the matching costs;
[0016] Step S10: using the winner-takes-all algorithm WTA to take the disparity value with the minimum matching cost as the optimal disparity, and generate a disparity map;
[0017] Step S11: optimizing the disparity map by a disparity optimization method, which at least includes: left-right consistency check, median filtering, small connected domain elimination and disparity map filling;
[0018] Step S12: Calculate the depth value of the corresponding pixel point pair by combining the disparity map with the internal and external parameters and distortion parameters obtained by binocular camera calibration, and generate a depth map;
[0019] Step S13: According to the vegetation centroid image coordinates, the corresponding depth value is searched on the depth map, and the corresponding vegetation centroid spatial position coordinates are calculated to complete the vegetation spatial positioning.
[0020] Furthermore, in step S1, the OpenCV binocular vision calibration algorithm function stereoCalibrate is used to calibrate the stereo camera intrinsic parameter I, extrinsic parameter and distortion parameter for epipolar correction and depth value measurement.
[0021] Furthermore, in step S2, the acquired binocular image data is transmitted to a host computer platform for processing in subsequent steps.
[0022] Further, in step S8, the matching cost based on color segmentation is obtained based on the assumption that the disparity changes smoothly within the same segmented area in the same color space;
[0023] The specific principle of constructing the cost matching function in the super green feature channel is: let I lg and I rg are the green vegetation foreground images obtained by OTSU threshold segmentation algorithm in the super green feature channel, R exg is the foreground pixel area obtained by segmentation on the reference image, p and q are a pair of pixels on the left and right images when the disparity is assumed to be d, that is, p = qd, and the minimum stage error is T r , δ is the absolute difference between the pixel values of the two matching pixel pairs. For any pair of matching points on the left and right views of the binocular image, the matching cost when the disparity is assumed to be d is:
[0024]
[0025] The Census transform is used on the source image to calculate the matching cost between the left and right views. The specific principle is: first, the left and right binocular images are converted into 8-bit grayscale images, and then a 5×5 calculation window is used to encode the grayscale value in the window as 0 and 1 according to the comparison between the grayscale value in the window and the center grayscale value. The Hamming distance is then used to calculate the matching distance in the left and right view windows as the matching cost value, and the calculation window is slid on the left and right binocular images to construct the global pixel matching cost.
[0026] Super green feature channel image C exg , edge feature image C edge and source image C census After the matching cost is calculated, the overall matching cost is constructed using the joint matching cost function. The joint matching cost function is:
[0027]
[0028] Among them, β and are coefficients respectively.
[0029] Further, in step S12, the depth value is calculated using the binocular stereo vision triangulation principle. For a point P in the space captured by the binocular camera, a pair of matching pixel points P is mapped on the left and right binocular images of the binocular camera. l and P r , calculate the spatial depth value coordinate z according to the calibrated camera internal and external parameters and the triangle similarity principle.
[0030] There are two main types of binocular vision stereo matching technology: traditional stereo matching technology and deep learning-based stereo matching technology. The deep learning-based stereo matching technology has high calculation accuracy and can achieve real-time processing with the support of GPU, but this method requires a large amount of data for model training, has poor generalization ability, and has high requirements for computing equipment, making it difficult to promote in large-scale agricultural production. Traditional stereo matching technology can be divided into three types: global matching, semi-global matching, and local matching. Global matching has high calculation accuracy and high calculation complexity, making it difficult to process in real time; the local matching method has a small amount of calculation but can only calculate the depth information of some image pixels, which cannot meet the requirements of multi-target spatial positioning of green vegetation; the semi-global matching method achieves a balance between global matching and local matching. On the basis of meeting a certain calculation accuracy, the calculation amount is small, and real-time processing can be basically achieved. It can also be deployed and used on low-computing power devices, and is currently a widely used method in production. Although the current semi-global stereo matching method can basically achieve real-time depth map generation, it is easily affected by factors such as illumination, noise, and environmental background, and is inefficient for vegetation positioning tasks in agricultural production, and cannot be applied on a large scale.
[0031] Compared with the prior art, the present invention and its preferred solution are aimed at the task of positioning green vegetation in the real environment of agricultural production, integrate the green channel color feature information of green vegetation and the edge feature information of vegetation, make full use of the feature information to enhance the positioning accuracy of the matching algorithm for green vegetation in the agricultural production environment, improve the matching efficiency of the stereo matching algorithm in this scene, only need a cheap RGB camera to collect scene images, can be deployed and used on a low-cost computing platform, and realize the rapid and accurate positioning of green vegetation. The experimental results conducted in the actual agricultural environment show that the method can achieve high-precision positioning of green vegetation in various agricultural production operation scenes. In the present method, the positioning accuracy of green vegetation is high, the positioning speed is fast, the calculation efficiency is high, the robustness is strong, and the requirements for the acquisition equipment and computing equipment are low. The overall cost of the system is relatively low, and it has strong applicability to the task of positioning green vegetation in agricultural production operation scenes, and can be used and promoted on a large scale. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 A schematic diagram of a process for locating green vegetation according to an embodiment of the present invention;
[0033] Figure 2 This is a schematic diagram of Census transformation matching cost calculation according to an embodiment of the present invention;
[0034] Figure 3 The figure is a schematic diagram of the principle of binocular stereo vision positioning according to an embodiment of the present invention. DETAILED DESCRIPTION
[0035] In order to make the features and advantages of this patent more obvious and easy to understand, the following embodiments are specifically described in detail as follows:
[0036] like Figure 1 As shown, the binocular vision green vegetation matching and positioning method proposed in this embodiment that integrates color features and edge features specifically includes the following steps:
[0037] S1. Use the OpenCV stereo vision calibration algorithm function stereoCalibrate to calibrate the stereo camera's internal parameters I, external parameters and distortion parameters for epipolar correction and depth value measurement;
[0038] Among them, the camera intrinsic parameter matrix can be expressed as:
[0039]
[0040] The camera external parameter matrix mainly includes the rotation matrix R from the left camera to the right camera rt and the translation matrix T rt , which can be specifically expressed as:
[0041] [R rt T rt ] (2)
[0042] The camera distortion parameters mainly include radial distortion and tangential distortion. The distortion parameters are specifically expressed as:
[0043] [k 1 k 2 p 1 p 2 k 3 ] (3)
[0044] S2, using a binocular camera with calibrated parameters to collect field binocular image data in real time and transmit it to the upper data processing platform;
[0045] S3, performing epipolar correction on the binocular image according to the calibrated binocular camera internal parameters, external parameters and distortion parameters, so that the matching pixel pairs on the binocular image are in the same row of the image, reducing the search range of the matching algorithm;
[0046] S4, use Gamma correction, adaptive histogram equalization and bilateral filtering to perform image preprocessing to reduce the impact of illumination and noise, while enhancing image edge detail information;
[0047] S5, converting the image processed in step S4 into a super green feature channel image in the RGB color space to generate a super green feature channel image, and detecting the edge information of the image using the Sobel edge detection algorithm to generate an edge feature image;
[0048] Since green vegetation has obvious green features in the RGB three-channel image and is greatly different from the soil background, the difference between color features can be used to effectively separate the green vegetation foreground and the soil background. In the RGB three-channel image, the image is first separated into three single-channel images of R, G, and B. The super-green feature method achieves a more effective segmentation effect by strengthening the green channel component in the color channel and enhancing the difference between the green channel feature and the other color channel features. The calculation formula of the super-green channel feature image is:
[0049]
[0050] S6, using the OTSU adaptive threshold segmentation algorithm to calculate the binarization threshold of the super green feature channel image, and using the threshold segmentation method to segment the image to obtain foreground vegetation image information;
[0051] S7, using a connected domain labeling algorithm to calculate the connected domain of the foreground vegetation image, and calculating the image centroid coordinates corresponding to the connected domain of the foreground vegetation;
[0052] S8, constructing a fusion matching cost function to calculate the matching cost on the super green feature channel image, the Sobel edge feature image and the source image;
[0053] Among them, the matching cost based on color segmentation is obtained based on the assumption that the disparity changes smoothly in the same segmentation area in the same color space. The similarity between two pixels with similar pixel values in the same color space is high. The specific principle of constructing the cost matching function in the super green feature channel is: lg and I rg are the green vegetation foreground images obtained by OTSU threshold segmentation algorithm in the super green feature channel, R exg is the foreground pixel area obtained by segmentation on the reference image, p and q are a pair of pixels on the left and right images when the disparity is assumed to be d, that is, p = qd, and the minimum stage error is T r , δ is the absolute difference between the pixel values of the two matching pixel pairs. For any pair of matching points on the left and right views of the binocular image, the matching cost when the disparity is assumed to be d is:
[0054]
[0055] Use Census transformation on the source image to calculate the matching cost between the left and right views. The principle is as follows Figure 2As shown in the figure, the specific principle is: first, the left and right binocular images are converted into 8-bit grayscale images, and then a 5×5 calculation window is used to encode the grayscale value in the window as 0 and 1 according to the comparison between the grayscale value in the window and the center grayscale value. Then, the Hamming distance is used to calculate the matching distance in the left and right view windows as the matching cost value, and the calculation window is slid on the left and right binocular images to construct the global pixel matching cost.
[0056] After the matching costs of the super green feature channel image, edge feature image, and source image are calculated, the overall matching cost is constructed using the joint matching cost function. The joint matching cost function is:
[0057]
[0058] S9, aggregate matching costs using the four-domain method in the semi-local matching algorithm (SGM);
[0059] S10, using a winner-takes-all (WTA) algorithm to take the disparity value with the minimum matching cost as the optimal disparity, and generate a disparity map;
[0060] S11, optimizing the disparity map by using disparity optimization methods such as left-right consistency check, median filtering, small connected domain elimination, and disparity map filling;
[0061] S12, calculating the depth value of the corresponding pixel point pair by combining the disparity map with the internal and external parameters and distortion parameters obtained by binocular camera calibration, and generating a depth map;
[0062] The depth value is calculated using the binocular stereo vision triangulation principle. Figure 3 As shown, for a point P in the space captured by the binocular camera, it is mapped to a pair of matching pixel points P on the left and right binocular images of the binocular camera. l and P r , according to the calibrated camera internal and external parameters and the triangle similarity principle, the spatial depth value coordinate z can be calculated. The calculation formula is:
[0063]
[0064]
[0065] S13. According to the vegetation centroid image coordinates, the corresponding depth value is searched on the depth map, and the corresponding vegetation centroid spatial position coordinates are calculated to complete the vegetation spatial positioning.
[0066] This patent is not limited to the above-mentioned optimal implementation mode. Anyone can derive other forms of binocular vision green vegetation matching and positioning methods that integrate color features and edge features under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of the present invention should be covered by this patent.
Claims
1. A binocular vision green vegetation matching and positioning method integrating color features and edge features, characterized in that: The following steps are involved: Step S1: Use the OpenCV binocular vision calibration algorithm to calibrate the binocular camera's intrinsic parameters, extrinsic parameters, and distortion parameters for epipolar correction and depth value measurement; Step S2: using a binocular camera with calibrated parameters to collect field binocular image data in real time; Step S3: performing epipolar correction on the acquired binocular image according to the calibrated binocular camera internal parameters, external parameters and distortion parameters, so that the matching pixel pairs on the binocular image are in the same row of the image; Step S4: image preprocessing using Gamma correction, adaptive histogram equalization and bilateral filtering; Step S5: converting the image processed in step S4 into a super-green feature channel image in the RGB color space to generate a super-green feature channel image; and detecting the edge information of the image using the Sobel edge detection algorithm to generate an edge feature image; Step S6: using the OTSU adaptive threshold segmentation algorithm to calculate the binarization threshold of the super green feature channel image, and using the threshold segmentation method to segment the image to obtain foreground vegetation image information; Step S7: using a connected domain labeling algorithm on the foreground vegetation image to calculate the foreground image connected domain, and calculating the image centroid coordinates corresponding to the foreground vegetation connected domain; Step S8: constructing a fusion matching cost function to calculate the matching cost on the super green feature channel image, the Sobel edge feature image and the source image; Step S9: using the semi-local matching algorithm SGM to aggregate the matching costs; Step S10: using the winner-takes-all algorithm WTA to take the disparity value with the minimum matching cost as the optimal disparity, and generate a disparity map; Step S11: optimizing the disparity map by a disparity optimization method, which at least includes: left-right consistency check, median filtering, small connected domain elimination and disparity map filling; Step S12: Calculate the depth value of the corresponding pixel point pair by combining the disparity map with the internal and external parameters and distortion parameters obtained by binocular camera calibration, and generate a depth map; Step S13: according to the vegetation centroid image coordinates, find the corresponding depth value on the depth map, calculate the corresponding vegetation centroid spatial position coordinates, and complete the vegetation spatial positioning; In step S8, the matching cost based on color segmentation is obtained based on the assumption that the disparity changes smoothly within the same segmented area in the same color space; The specific principle of constructing the cost matching function in the super green feature channel is: let I lg and I rg are the green vegetation foreground images obtained by OTSU threshold segmentation algorithm in the super green feature channel, R exg is the foreground pixel area obtained by segmentation on the reference image, p and q are a pair of pixels on the left and right images when the disparity is assumed to be d, that is, p = qd, and the minimum stage error is T r , δ is the absolute difference between the pixel values of the two matching pixel pairs. For any pair of matching points on the left and right views of the binocular image, the matching cost when the disparity is assumed to be d is: The Census transform is used on the source image to calculate the matching cost between the left and right views. The specific principle is: first, the left and right binocular images are converted into 8-bit grayscale images, and then a 5×5 calculation window is used to encode the grayscale value in the window as 0 and 1 according to the comparison between the grayscale value in the window and the center grayscale value. The Hamming distance is then used to calculate the matching distance in the left and right view windows as the matching cost value, and the calculation window is slid on the left and right binocular images to construct the global pixel matching cost. After the matching costs of the super green feature channel image, edge feature image, and source image are calculated, the overall matching cost is constructed using the joint matching cost function. The joint matching cost function is: Among them, β and are coefficients respectively.
2. The binocular vision green vegetation matching and positioning method integrating color features and edge features according to claim 1 is characterized in that: In step S1, the OpenCV stereo vision calibration algorithm function stereoCalibrate is used to calibrate the stereo camera intrinsic parameter I, extrinsic parameter and distortion parameter for epipolar correction and depth value measurement.
3. The binocular vision green vegetation matching and positioning method integrating color features and edge features according to claim 1 is characterized in that: In step S2, the acquired binocular image data is transmitted to a host computer platform for subsequent processing.
4. The binocular vision green vegetation matching and positioning method integrating color features and edge features according to claim 1 is characterized in that: In step S12, the depth value is calculated using the binocular stereo vision triangulation principle. For a point P in the space captured by the binocular camera, a pair of matching pixel points P is mapped on the left and right binocular images of the binocular camera. l and P r , calculate the spatial depth value coordinate z according to the calibrated camera internal and external parameters and the triangle similarity principle.
