A device for detecting weeds between wine grape plants based on bilateral monocular real-time detection and a method for using the same

Abstract

The application discloses a kind of interplant weeding device of wine grape based on bilateral monocular real-time detection and its use method, method includes: using camera to collect initial image in the process of moving weeding of base plate, color segmentation is carried out to initial image, the proportion of black area of grapevine is obtained, the threshold of black area of grapevine is calculated using deep learning;The black area proportion of grapevine is compared with threshold, based on the calibration result of camera, a plurality of images containing checkerboard are collected after calibration, the pretreatment image is obtained by pretreatment to image;Based on pretreatment image, the standard reflectivity of grapevine and the irradiance of grapevine surface are calculated;A comprehensive correction factor is introduced, the distance from grapevine to camera is calculated based on standard reflectivity and irradiance, and the control method of bilateral monocular real-time detection interplant weeding device of wine grape is completed.The application uses two cameras to detect simultaneously, to avoid the influence of unilateral sunlight irradiation.

Description

Technical Field

[0001] This invention relates to the field of inter-vine weeding devices, specifically to an inter-vine weeding device for wine grapes based on bilateral monocular real-time detection and its usage method. Background Technology

[0002] The vineyards on the eastern foothills of the Helan Mountains in Yinchuan have irregular planting patterns and complex terrain. If the blades are too high off the ground, the weeding rate will be reduced, and if they are too low, it will affect the chassis driving and cause damage to the blades. The existing weeding devices between vines cannot achieve the expected results in terms of terrain simulation.

[0003] Generally, methods for locating grapevines include landmarking, row and plant spacing measurement, GIS, GPS, visual positioning technology, laser scanning technology, and UAV technology. However, landmarking is time-consuming and labor-intensive; row and plant spacing measurement has too much error and can no longer meet the needs; GIS and GPS still require prior calibration; visual positioning and laser scanning technologies are easily affected by the environment, and laser scanning technology is more suitable for indoor use; UAV technology is too expensive. In summary, visual positioning technology is the most widely used and effective method for locating grapevines.

[0004] However, current visual positioning technology for grapevine location also has certain problems: First, the camera may be overexposed or underexposed due to lighting conditions, resulting in loss of highlight details, color distortion, reduced contrast, narrowed dynamic range, loss of shadow details, and increased noise, thus making it impossible to identify or locate the grapevines. For example, when a weeding device is added to the traction device, only a single blade on one side is used for weeding. Second, visual positioning technology currently generally uses deep learning to identify grapevines. Due to factors such as the grapevines in the eastern foothills of Helan Mountain in Ningxia having almost no difference in color from the soil and the complex environment of vineyards, the number of parameters in the deep learning model increases, requiring a more powerful computing system and more computation time, resulting in a lack of real-time performance. Furthermore, the complex field environment also makes the recognition accuracy insufficient to meet the detection requirements of the weeding chassis, causing the weeding equipment to fail to avoid the grapevines, avoid them in a timely manner, or avoid them incorrectly. Summary of the Invention

[0005] This invention provides a weeding device for wine grapevines based on bilateral monocular real-time detection. The device includes: an upper light-blocking plate, a chassis, a terrain-following pusher, a checkerboard target, a camera, a background plate, a cutting mechanism, and a universal travel switch.

[0006] The upper light-shielding plate is connected to the chassis;

[0007] The terrain-following push rod is installed on the inner wall of the chassis and connected to the tool mechanism, and the universal travel switch is installed at the bottom of the tool mechanism;

[0008] The background panel is equipped with cameras and checkerboard-patterned targets.

[0009] Optionally, the terrain-following push rod is installed inside the chassis and connected to the cutting tool mechanism. The terrain-following push rod moves up and down, driving the entire cutting tool mechanism to move up and down.

[0010] Optionally, the background panel may be made of red opaque acrylic sheet.

[0011] This invention also discloses a control method for a weeding device between wine grapevines based on bilateral monocular real-time detection, the method comprising:

[0012] The camera captures the initial image during the mobile weeding process of the chassis. The initial image is color segmented to obtain the proportion of black area of ​​grapevines. Deep learning is used to calculate the threshold of black area of ​​grapevines.

[0013] By comparing the proportion of black area of ​​grapevines and the threshold, and calibrating the camera based on the comparison results, several images containing checkerboard patterns are collected, and the images are preprocessed to obtain preprocessed images.

[0014] The standard reflectance of the grapevines and the irradiance of the grapevine surface were calculated based on the preprocessed images.

[0015] A comprehensive correction factor is introduced to calculate the distance from the grapevine to the camera based on standard reflectance and irradiance, thus completing the control method for a weeding device between wine grapevines with bilateral monocular real-time detection.

[0016] Optionally, the process of color segmentation of the initial image specifically includes:

[0017] The red and green colors in the initial image are converted to black, and the other colors are converted to white. The initial image after color conversion is then cropped so that the image border coincides with the background border.

[0018] Optionally, the deep learning is a feedforward neural network with three hidden layers. The first layer has 128 neurons, the second layer has 64 neurons, and the third layer has 32 neurons. Batch Normalization is added after each hidden layer, and the output layer is normalized using the sigmoid activation function.

[0019] Optionally, a comprehensive correction factor is introduced. The method for calculating the distance from the grapevine to the camera based on the standard reflectance and irradiance specifically includes:

[0020]

[0021] Where, d ref To calibrate the distance from the target plane to the camera, R refTo calibrate the standard reflectance of the target plane, R is the standard reflectance of the grapevine, ρ(θ) is the correction factor, and θ ref To calibrate the incident angle of the target plane relative to the camera's line of sight, i ref To calibrate the pixel intensity value of the target plane, i is the pixel intensity value of the grapevine in the image, and κ is the comprehensive correction factor.

[0022] Optionally, the formula for calculating the pixel intensity value of the calibration target plane is:

[0023] i ref =G·E ref ·T

[0024] Where G is the camera gain, T is the exposure time, and E is the exposure time. ref To calibrate the irradiance produced by the target plane on the camera sensor plane.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0026] This invention, by employing a dual-sided monocular camera design, solves the problems of highlight detail loss and color distortion caused by monocular vision technology under strong light conditions, ensuring high-precision detection and positioning of grapevines. Combining color segmentation, deep learning feedforward neural networks, and real-time detection and correction, it significantly improves real-time performance and recognition accuracy in complex field environments, reducing the need for high-computing equipment and ensuring the continuity and stability of weeding operations. Through the linkage design of the terrain-following push rod and the blade mechanism, the device can adaptively adjust the blade's vertical movement according to terrain undulations, adapting to complex terrain, improving weeding accuracy and blade life, and ensuring efficient operation in various field environments. By introducing a comprehensive correction factor, the standard reflectance and grapevine surface irradiance are dynamically corrected to calculate the precise distance from the grapevine to the camera, overcoming distance measurement errors caused by unstable lighting conditions in visual positioning and improving the accuracy of grapevine detection and positioning. Color segmentation and cropping preprocessing effectively distinguish the grapevine from the background, simplifying the input data for subsequent deep learning and improving the efficiency and real-time performance of image processing. Simultaneously, the use of a red opaque acrylic background enhances image contrast, further reducing visual errors. Attached Figure Description

[0027] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1This is a schematic diagram of the inter-vine weeding device based on bilateral monocular real-time detection according to an embodiment of the present invention.

[0029] Figure 2 This is a partial schematic diagram of a wine grape weeding device based on bilateral monocular real-time detection according to an embodiment of the present invention;

[0030] Figure 3 This is a flowchart illustrating the control method steps of a wine grape weeding device based on bilateral monocular real-time detection according to an embodiment of the present invention.

[0031] Figure 4 This is a comparison image of the initial image before and after color segmentation according to an embodiment of the present invention;

[0032] Figure 5 This is a schematic diagram of a deep learning model according to an embodiment of the present invention;

[0033] In the diagram, 1 is the upper light shield, 2 is the chassis, 3 is the terrain-following push rod, 4 is the checkerboard marker target, 5 is the camera, 6 is the background plate, 7 is the cutting tool mechanism, and 8 is the universal travel switch. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] Example 1

[0037] A weed control device for wine grapevines based on bilateral monocular real-time detection, such as Figure 1 , Figure 2 As shown, the device includes: an upper light shield 1, a chassis 2, a terrain-following push rod 3, a checkerboard-marked target 4, a camera 5, a background plate 6, a cutting mechanism 7, and a universal travel switch 8.

[0038] The upper light shield 1 covers the top of the entire weeding chassis 2, and the background plate 6 is installed on both sides of the chassis 2, realizing the chassis is surrounded on three sides. This creates a relatively closed environment for the vision system, greatly avoiding the influence of sunlight on the camera shooting, thereby ensuring the reliability and accuracy of visual positioning.

[0039] Camera 5 is installed in the center of the background panel 6, and the checkerboard target 4 is installed in one corner of the background panel 6. During chassis movement, camera 5 uses the checkerboard target 4 as a reference to detect and calculate the position of the grapevines, guiding the cutting tools to avoid obstacles. Using two cameras simultaneously avoids the influence of sunlight from one side, utilizing data from a camera without sunlight interference. Simultaneous detection from two directions also reduces the impact of grape leaves shading the vehicle.

[0040] The terrain-following push rod 3 is mounted on the inner wall of the chassis 2 and connected to the tool mechanism 7. By moving the push rod up and down, the tool mechanism 7 can be made to rise and fall. The universal travel switch 8 is installed on the tool mechanism 7 close to the ground. When the universal travel switch 8 touches the ground, the push rod rises; when the universal travel switch 8 does not touch the ground, the push rod continuously descends with a small stroke. This method can solve the problem of the tool following the terrain.

[0041] The terrain-following push rod 3 is installed inside the chassis and connected to the tool mechanism 7. The up-and-down movement of the terrain-following push rod 3 can drive the entire tool mechanism 7 to move up and down. A universal travel switch 8 is installed at the end of the tool near the ground. The angle between the universal travel switch and the forward direction is an acute angle, which can improve the service life of the universal travel switch. When the universal travel switch touches the ground, the terrain-following push rod 3 moves upward at full speed, driving the tool mechanism 7 to rise. When the universal travel switch does not touch the ground, the terrain-following push rod 3 descends with a small stroke, thus realizing the terrain-following function of a single tool mechanism 7.

[0042] Because the weeding chassis 2 has the characteristic of being able to be surrounded on three sides, the background plate 6 and the upper light shield 1 can create a relatively closed environment for the vision system, greatly avoiding the influence of sunlight on the camera 5, thereby ensuring the reliability and accuracy of visual positioning.

[0043] Because red and green are the easiest colors to distinguish, the background board is made of red opaque acrylic sheet.

[0044] Example 2

[0045] A control method for a weeding device between wine grape vines based on bilateral monocular real-time detection, such as... Figure 3 As shown, the method includes:

[0046] The initial images of the mobile weeding process are captured by a camera. The initial images are then segmented by color to obtain the proportion of black area of ​​the grapevines. Deep learning is then used to calculate the threshold of the black area of ​​the grapevines.

[0047] like Figure 4 As shown, Figure 4 (a) is the initial image. Figure 4(b) The initial image after color segmentation. Grapevine detection method: Color segmentation is used for recognition and detection. Since the background is red and the grapevine is brown, the easiest colors to segment, red and green, are converted to black, and all other colors are converted to white. This filters out green weeds and grape leaves. The image is then cropped so that the image border coincides with the background border. If the camera does not capture the grapevine, the image is entirely black; if the grapevine is present, a white outline of the grapevine will appear in the image. When the area of ​​the black region is less than a set threshold, it is considered that the grapevine has entered the camera's field of view. Because weeds and grapevines are significantly different in size, this method avoids interference from weeds. Moreover, objects that are always present in the image, such as droppers, will not affect the judgment result. Because there are light-blocking plates on three sides, this recognition and detection method is not easily affected by sunlight, and the recognition and detection are relatively stable. Furthermore, the entire system only needs to perform color recognition and simple calculations to obtain the location information of the grapevine. The entire system is simple and applicable, and can be easily ported to various embedded systems without requiring a large vision computing chassis.

[0048] like Figure 5 As shown, a deep learning approach is used to determine the threshold. The proportions of red and green pixels in the image segmentation are used as input parameters. To ensure real-time performance, a simple feedforward neural network with three hidden layers is employed. The first layer has 128 neurons, the second has 64 neurons, and the third has 32 neurons. Batch Normalization is added after each hidden layer to accelerate training, improve model stability, and reduce sensitivity to initial weights. Dropout layers are added, with a 20% dropout ratio in each layer, effectively preventing overfitting and improving the model's generalization ability. The output layer uses a sigmoid activation function to ensure the output is within the range of 0 to 1, and then denormalizes it to 0 to 100 during prediction to obtain the threshold. Furthermore, during sample collection, after each prediction, a human judges whether it is correct or incorrect, updating the model parameters in real time.

[0049] By comparing the proportion of black areas in the grapevines with the threshold, and calibrating the camera based on the comparison results, several images containing a checkerboard pattern are collected, and the images are preprocessed to obtain preprocessed images.

[0050] Grapevine positioning method: After the grapevine outline can be detected, the position information of the grapevine needs to be calculated. Since the tool only needs two-dimensional position information to avoid the grapevine, it is not necessary to obtain the height information of the grapevine.

[0051] First, the camera is calibrated by capturing multiple images containing a checkerboard pattern, ensuring the checkerboard appears from multiple angles and distances. The camera's intrinsic parameter matrix and distortion parameters are solved using OpenCV's `cv2.calibrateCamera` method. The quality of the calibration result is determined by minimizing the reprojection error, combined with a nonlinear optimization algorithm (Levenberg-Marquardt method). After calibration, the image is corrected using the calibrated parameters: `corrected image = cv2.undistort(original image, intrinsic parameter matrix, distortion coefficients)`. After image correction, the width and distance information of the grapevines can be calculated.

[0052] The standard reflectance of the grapevines and the irradiance of the grapevine surface were calculated based on the preprocessed images.

[0053] A comprehensive correction factor is introduced to calculate the distance from the grapevine to the camera based on the standard reflectance and irradiance, thus completing the control method for a weeding device between wine grapevines with bilateral monocular real-time detection.

[0054] The distance from the calibration target plane to the camera is known as d. ref The standard reflectance is known to be R. ref And under the same lighting conditions, a reference pixel intensity value i was obtained. ref By calibrating the target plane, we can relatively infer the distance from the grapevine to the camera without absolutely measuring the scene illumination or absolutely determining all parameters in the camera model. This relative measurement method has greater robustness and practicality under real-world conditions.

[0055] The standard reflectance R of the grapevine and the irradiance on the grapevine surface are measured as I0. The exposure parameters of the camera (camera lens diameter D, exposure time T, and camera gain G) are known. The pixel grayscale value of the white area in the image is i.

[0056] Introduce a correction factor ρ(θ) = -θ 2 +1, where θ is the angle (in radians) between the line of sight and the surface normal.

[0057] To adapt to changes in different cameras, lenses, and ambient lighting conditions, a comprehensive correction factor κ was introduced, and a better κ value was obtained through experiments.

[0058] Regarding the reference plane:

[0059] i ref =G·E ref ·T

[0060] Where G is the camera gain, T is the exposure time, and E is the exposure time. ref To calibrate the irradiance produced by the target plane on the camera sensor plane.

[0061] The effective light-gathering area of ​​the lens is:

[0062]

[0063] The irradiance from the target plane to the camera plane is:

[0064]

[0065] but:

[0066]

[0067] We obtain the following through similarity relationships:

[0068]

[0069] The distance from the grapevine to the camera is:

[0070]

[0071] Where, d ref To calibrate the distance from the target plane to the camera, R ref To calibrate the standard reflectance of the target plane, R is the standard reflectance of the grapevine, ρ(θ) is the correction factor, and θ ref To calibrate the incident angle of the target plane relative to the camera's line of sight, i ref To calibrate the pixel intensity value of the target plane, i is the pixel intensity value of the grapevine in the image, and κ is the comprehensive correction factor.

[0072] The width of the grapevine can then be calculated from the number of white pixels in the image.

[0073] This method allows us to obtain the distance from the grapevine to the camera and the width of the grapevine. Furthermore, the detection is performed by two cameras on both sides, which can avoid interference from sunlight on one side and reduce the impact of grape leaves blocking the view.

[0074] Since the distance between the grapevine and the blade is fixed when the grapevine is detected, the time for the blade to start obstacle avoidance can be calculated based on the driving speed. The width of the grapevine and the distance from the camera to the grapevine are used to guide the obstacle avoidance range of the blade.

[0075] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A control method of a wine grape inter-plant weeding device based on bilateral monocular real-time detection, the control method being used to control a wine grape inter-plant weeding device, characterized in that, The method includes: The camera captures the initial image during the mobile weeding process of the chassis. The initial image is color segmented to obtain the proportion of black area of ​​grapevines. Deep learning is used to calculate the threshold of black area of ​​grapevines. By comparing the proportion of black area of ​​grapevines and the threshold, and calibrating the camera based on the comparison results, several images containing checkerboard patterns are collected, and the images are preprocessed to obtain preprocessed images. The standard reflectance of the grapevines and the irradiance of the grapevine surface were calculated based on the preprocessed images. A comprehensive correction factor is introduced to calculate the distance from the grapevine to the camera based on standard reflectance and irradiance, thus completing a control method for a real-time, bilateral monocular detection weeding device among wine grapevines; specifically including: in, To calibrate the distance from the target plane to the camera, To calibrate the standard reflectivity of the target plane, The standard reflectance of the grapevine, As a correction factor, To calibrate the incident angle of the target plane relative to the camera's line of sight, To calibrate the pixel intensity values ​​of the target plane, i This represents the intensity value of the corresponding pixel in the image representing the grapevine. As a comprehensive correction factor; The intervine weeding device for wine grapes includes: an upper shading plate, a chassis, a terrain-following push rod, a checkerboard target, a camera, a background plate, a cutting mechanism, and a universal travel switch; the upper shading plate is connected to the chassis; the terrain-following push rod is installed on the inner wall of the chassis and connected to the cutting mechanism, and the universal travel switch is installed at the bottom of the cutting mechanism; the background plate is equipped with a camera and a checkerboard target. The terrain-following push rod is installed inside the chassis and connected to the tool mechanism. The terrain-following push rod moves up and down, which drives the entire tool mechanism to move up and down. The background panel is made of red opaque acrylic sheet.

2. The control method for the inter-vine weeding device based on bilateral monocular real-time detection of wine grapes according to claim 1, characterized in that, The process of color segmentation of the initial image specifically includes: The red and green colors in the initial image are converted to black, and the other colors are converted to white. The initial image after color conversion is then cropped so that the image border coincides with the background border.

3. The control method of the inter-row weeding device for wine grape based on bilateral monocular real-time detection according to claim 2, characterized in that, The deep learning method is a feedforward neural network with three hidden layers. The first layer has 128 neurons, the second layer has 64 neurons, and the third layer has 32 neurons. Batch Normalization is added after each hidden layer, and the output layer is normalized using the sigmoid activation function.

4. The control method of the inter-row weeding device for wine grape based on bilateral monocular real-time detection according to claim 1, characterized in that, The formula for calculating the pixel intensity value of the calibration target plane is: wherein, Gcamis the camera gain, texpis the exposure time, Ecalis the irradiance produced by the calibration target plane on the camera sensor plane.

Images