Wine grape inter-plant weeding device based on bilateral monocular real-time detection and use method of wine grape inter-plant weeding device

Abstract

The invention discloses a bilateral monocular real-time detection-based inter-plant weeding device for wine grapes and a use method thereof, and the method comprises the steps: collecting an initial image in a moving weeding process of a chassis by using a camera, carrying out the color segmentation of the initial image, obtaining the proportion of the black area of grape vines, and obtaining the proportion of the black area of the grape vines; calculating a threshold value of the black area of the grape vine by using deep learning; comparing the black area proportion of the grape vines with a threshold value, calibrating a camera based on a comparison result, then collecting a plurality of images containing checkerboards, and preprocessing the images to obtain preprocessed images; calculating the standard reflectivity of the grape vine and the irradiance of the surface of the grape vine based on the preprocessed image; a comprehensive correction factor is introduced, the distance between the grape vines and the camera is calculated based on standard reflectivity and irradiance, and the control method of the wine grape inter-plant weeding device for bilateral monocular real-time detection is completed. The two cameras are used for detection at the same time, and the influence of single-side sunlight irradiation is avoided.

Description

Technical Field

[0001] The invention relates to the technical field of grape weeding devices, and in particular to a wine grape weeding device based on bilateral monocular real-time detection and a use method thereof. Background Art

[0002] The planting pattern of wine vineyards on the eastern foot of the Helan Mountains in Yinchuan is irregular and the terrain is complex. If the cutters are too high from the ground, the weeding rate will be reduced, while if they are too low, it will affect the chassis driving and cause damage to the cutters. The terrain-mimicking effect of existing wine grape weeding devices is difficult to achieve the expected effect.

[0003] Generally speaking, there are several methods for positioning grapevines: landmark method, row and plant spacing measurement method, GIS, GPS, visual positioning technology, laser scanning technology, and UAV technology. However, the landmark method takes a long time and has high labor costs, and the row and plant spacing measurement method has too large an error and can no longer meet the needs; GIS and GPS still need to be calibrated in advance; visual positioning technology and laser scanning technology are easily affected by the environment, and laser scanning technology is more suitable for indoor use; UAV technology is too expensive. In summary, visual positioning technology is a widely used and effective grapevine positioning method.

[0004] However, there are some problems with the current visual positioning technology used for grapevine positioning: First, the camera will be overexposed or underexposed due to the influence of light, which will cause the image to lose highlight details, color distortion, reduced contrast, reduced dynamic range or dark details, increased noise and other adverse effects, resulting in the inability to identify or locate the grapevine. For example, a weeding device is installed on the traction device, and only a single cutter on one side is used for weeding. Second, visual positioning technology now generally uses deep learning to identify grapevines. Due to factors such as the fact that the grapevines on the eastern foot of the Helan Mountains in Ningxia are almost the same color as the soil and the complex environment of the grape fields, the number of model parameters for deep learning will increase, requiring a stronger computing system and more computing time to adapt, and the real-time performance is relatively lacking. In addition, the complex field environment also makes the recognition accuracy unable to meet the detection requirements of the weeding chassis, resulting in the weeding machine not avoiding the grapevines, avoiding them untimely or avoiding them incorrectly. Summary of the invention

[0005] The present invention provides a wine grape weeding device based on bilateral monocular real-time detection, the device comprising: an upper shading plate, a chassis, a terrain simulation, a chessboard calibration target, a camera, a background plate, a tool mechanism and a universal travel switch;

[0006] The upper sunshade is connected to the chassis;

[0007] The terrain-mimicking push rod is installed on the inner wall of the chassis and connected to the tool mechanism, and the universal travel switch is installed at the bottom of the tool mechanism;

[0008] The background plate is provided with a camera and a chessboard calibration target.

[0009] Optionally, the terrain-simulating push rod is installed on the inner side of the chassis and connected to the tool mechanism. The terrain-simulating push rod moves up and down, driving the entire tool mechanism to move up and down.

[0010] Optionally, the background plate uses a red opaque acrylic plate.

[0011] The present invention also discloses a control method for a wine grape weeding device based on bilateral monocular real-time detection, the method comprising:

[0012] Use a camera to capture the initial image of the chassis during mobile weeding, perform color segmentation on the initial image, obtain the percentage of black area of ​​the grape vine, and use deep learning to calculate the threshold of the black area of ​​the grape vine;

[0013] Compare the black area ratio and threshold of the grape vine, calibrate the camera based on the comparison result, collect several images containing checkerboards, and preprocess the images to obtain preprocessed images;

[0014] Calculate the standard reflectance of the grapevine and the irradiance of the grapevine surface based on the preprocessed images;

[0015] A comprehensive correction factor is introduced to calculate the distance from the grapevine to the camera based on standard reflectivity and irradiance, thus completing the control method of the wine grape weeding device with bilateral monocular real-time detection.

[0016] Optionally, the process of performing color segmentation on the initial image specifically includes:

[0017] The red and green colors in the initial image are converted to black, and the other colors are converted to white. The initial image after the color conversion is cropped so that the image border overlaps with the background border.

[0018] Optionally, the deep learning is a feedforward neural network comprising three hidden layers, the first layer has 128 neurons, the second layer has 64 neurons, and the third layer has 32 neurons. Batch Normalization is added after each hidden layer, and the output layer is standardized using a sigmoid activation function.

[0019] Optionally, a comprehensive correction factor is introduced, and the method for calculating the distance from the grapevine to the camera based on the standard reflectivity and irradiance specifically includes:

[0020]

[0021] Among them, d ref To calibrate the distance from the target plane to the camera, R refis the standard reflectivity of the calibration target plane, R is the standard reflectivity of the grapevine, ρ(θ) is the correction factor, θ ref is the incident angle of the calibration target plane relative to the camera line of sight, i ref is the pixel intensity value for calibrating the target plane, i is the corresponding pixel intensity value of the grapevine in the image, and κ is the comprehensive correction factor.

[0022] Optionally, the calculation formula for the calibration target plane pixel intensity value is:

[0023] i ref =G·E ref ·T

[0024] Among them, G is the camera gain, T is the exposure time, and E ref It is the irradiance produced by the calibration target plane on the camera sensor plane.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] The present invention solves the problems of loss of highlight details and color distortion caused by monocular vision technology under strong light conditions by adopting a bilateral monocular camera design, and ensures high-precision detection and positioning of grapevines; combined with color segmentation, deep learning feedforward neural network, real-time detection and correction, the real-time performance and recognition accuracy are significantly improved in complex field environments, reducing the demand for high-computing power equipment and ensuring the continuity and stability of weeding operations; through the linkage design of the terrain-mimicking push rod and the tool mechanism, the device can realize the up and down adaptive adjustment of the tool according to the terrain undulations, can adapt to complex terrain, improve the weeding accuracy and tool life, and ensure that the device works efficiently in a variety of field environments. By introducing a comprehensive correction factor, the standard reflectivity and the irradiance of the grapevine surface are dynamically corrected, and the precise distance from the grapevine to the camera is calculated, which overcomes the distance measurement error caused by the unstable lighting conditions in visual positioning and improves the accuracy of grapevine detection and positioning; through color segmentation and cropping preprocessing, the grapevine is effectively distinguished from the background, which simplifies the input data of subsequent deep learning and improves the efficiency and real-time performance of image processing. At the same time, the red opaque acrylic background plate is used to enhance the image contrast, further reducing the visual error. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0028] Figure 1It is an overall schematic diagram of a wine grape weeding device based on bilateral monocular real-time detection according to an embodiment of the present invention;

[0029] Figure 2 It is a partial schematic diagram of a wine grape weeding device based on bilateral monocular real-time detection according to an embodiment of the present invention;

[0030] Figure 3 This is a step diagram of a control method of a wine grape weeding device based on bilateral monocular real-time detection according to an embodiment of the present invention;

[0031] Figure 4 This is a comparison diagram of the initial image before and after color segmentation is performed in an embodiment of the present invention;

[0032] Figure 5 A schematic diagram of a deep learning model according to an embodiment of the present invention;

[0033] In the figure, there are an upper shading plate 1, a chassis 2, a terrain-simulating push rod 3, a chessboard calibration target 4, a camera 5, a background plate 6, a tool mechanism 7, and a universal travel switch 8. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] Embodiment 1

[0037] A wine grape weeding device based on bilateral monocular real-time detection, such as Figure 1 , Figure 2 As shown, the device includes: an upper shading plate 1, a chassis 2, a terrain-simulating push rod 3, a chessboard calibration target 4, a camera 5, a background plate 6, a tool mechanism 7, and a universal travel switch 8.

[0038] The upper sunshade 1 covers the top of the entire weeding chassis 2, and the background plate 6 is installed on both sides of the chassis 2 to achieve three-sided enclosure of the chassis, which can create a relatively closed environment for the visual system and greatly avoid the influence of sunlight on the camera shooting, thereby ensuring the reliability and accuracy of visual positioning.

[0039] The camera 5 is installed in the middle of the background plate 6, and the checkerboard calibration target 4 is installed in a corner of the background plate 6. When the chassis is driving, the camera 5 uses the checkerboard calibration target 4 as a reference to detect and calculate the position information of the grapevine to guide the tool to avoid obstacles. And using two cameras to detect at the same time can avoid the influence of sunlight on one side and use camera data without sunlight interference. The method of detecting in two directions at the same time can also reduce the occlusion effect of grape leaves.

[0040] The terrain-simulating push rod 3 is installed on the inner wall of the chassis 2 and connected to the tool mechanism 7. The push rod moves up and down, which can drive the tool mechanism 7 to rise and fall. The universal travel switch 8 is installed at the position of the tool mechanism 7 close to the ground. When the universal travel switch 8 touches the ground, the push rod rises; when the universal travel switch 8 does not touch the ground, the push rod continuously descends with a small stroke. This method can solve the problem of tool terrain simulation.

[0041] A terrain-simulating push rod 3 is installed on the inner side of the chassis and connected to the tool mechanism 7. The up and down movement of the terrain-simulating push rod 3 can drive the entire tool mechanism 7 to move up and down; a universal travel switch 8 is installed at the end of the tool close to the ground. The angle between the universal travel switch and the forward direction is an acute angle, which can increase the service life of the universal travel switch. When the universal travel switch touches the ground, the terrain-simulating push rod 3 moves upward at full speed to drive the tool mechanism 7 to rise. When the universal travel switch does not touch the ground, the terrain-simulating push rod 3 descends with a small stroke, thereby realizing the terrain-simulating function of a single tool mechanism 7.

[0042] Since the weeding chassis 2 has the characteristic of being enclosed on three sides, the background plate 6 and the upper sunshade plate 1 can create a relatively closed environment for the visual system, greatly preventing the influence of sunlight on the shooting of the camera 5, thereby ensuring the reliability and accuracy of visual positioning.

[0043] Because red and green are the easiest colors to separate, the background board is made of red opaque acrylic board.

[0044] Embodiment 2

[0045] A control method for a wine grape weeding device based on bilateral monocular real-time detection, such as Figure 3 As shown, the method includes:

[0046] Use a camera to capture the initial image of the chassis during mobile weeding, perform color segmentation on the initial image, obtain the proportion of the black area of ​​the grape vine, and use deep learning to calculate the threshold of the black area of ​​the grape vine.

[0047] like Figure 4 As shown, Figure 4 (a) is the initial image, Figure 4(b) is the initial image after color segmentation. Grapevine detection method: The recognition detection adopts the form of color segmentation. Because the background board is red and the grapevine is brown, the red and green backgrounds that are easiest to segment are turned into black, and the rest of the colors are turned into white. In this way, the green weeds and grape leaves are filtered out, and then the image is cropped so that the image border coincides with the background border. If there is no camera and no grapevine is captured, the picture will be all black. If there is a grapevine, a white grapevine outline will appear in the image. When the area of ​​the black area is smaller than the set threshold, it is considered that the grapevine has entered the camera's field of view. Because the volume of weeds and grapevines is far different, this method can avoid the interference of weeds, and objects such as droppers that have always existed in the image will not affect the judgment result. Because there are sunshades on three sides, this recognition detection is not easily affected by sunlight, and the recognition detection is relatively stable. Moreover, the entire system only needs to perform color recognition and simple calculations to obtain the location information of the grapevine. The entire system is simple and applicable, and can be easily transplanted into various embedded systems without the need for a large visual computing chassis.

[0048] like Figure 5 As shown, the threshold is determined by deep learning, and the red and green proportions of the image segmentation are used as input parameters. To ensure real-time performance, a simple feedforward neural network is used, which contains three hidden layers. The first layer has 128 neurons, the second layer has 64 neurons, and the third layer has 32 neurons. BatchNormalization is added after each hidden layer, which can accelerate training and improve the stability of the model and reduce sensitivity to initial weights. Dropout layers are added, and the ratio of Dropout in each layer is 20%, which can effectively prevent overfitting of the model and improve the generalization ability of the model. The output layer uses a sigmoid activation function to ensure that the output is in the range of 0 to 1, and then it is denormalized to 0 to 100 during prediction, that is, the threshold is obtained. And in the process of sample collection, each prediction is made, and then the right and wrong are manually judged, and the model parameters are updated in real time.

[0049] The black area ratio and threshold of the grape vines are compared, and the camera is calibrated based on the comparison results, and then several images containing checkerboards are collected, and the images are preprocessed to obtain preprocessed images.

[0050] Grapevine positioning method: After the grapevine outline is detected, the position information of the grapevine needs to be calculated. Since the tool only needs two-dimensional position information to avoid the grapevine, there is no need to obtain the grapevine height information.

[0051] First, calibrate the camera and take multiple images containing checkerboards to ensure that the checkerboards appear from multiple angles and distances. Use OpenCV's cv2.calibrateCamera method to solve the camera's intrinsic matrix and distortion parameters. By minimizing the reprojection error and combining the nonlinear optimization algorithm (Levenberg-Marquardt method), the quality of the calibration result can be measured by the reprojection error. After the calibration is completed, use the calibrated parameters to correct the image. Corrected image = cv2.undistort (original image, intrinsic matrix, distortion coefficient). After the image is corrected, the vine width and distance information can be calculated.

[0052] The standard reflectance of the grapevine and the irradiance of the grapevine surface were calculated based on the preprocessed images.

[0053] A comprehensive correction factor is introduced to calculate the distance from the grapevine to the camera based on the standard reflectivity and irradiance, thereby completing the control method of the wine grape inter-vine weeding device with bilateral monocular real-time detection.

[0054] The distance from the calibration target plane to the camera is known to be d ref , the standard reflectivity is known as R ref , and shoot under the same lighting conditions to obtain the reference pixel intensity value i ref ,By calibrating the target plane, we can relatively infer the distance from the vine to the camera, without absolutely measuring the scene illumination or absolutely determining all parameters in the camera model.,This relative measurement method can be more robust and practical under real conditions.

[0055] The standard reflectivity R of the grapevine and the irradiance on the surface of the grapevine are measured to be I0. The exposure parameters of the camera (camera lens diameter D, exposure time T, camera gain G) are all known, and the pixel grayscale value of the white area in the image is i.

[0056] Introduce a correction factor ρ(θ) = -θ 2 +1, where θ is the angle (radians) between the line of sight and the surface normal.

[0057] In order to adapt to the changes of different cameras, lenses and ambient lighting conditions, a comprehensive correction factor κ is introduced, and a better κ value is obtained through experiments.

[0058] For reference surfaces:

[0059] i ref =G·E ref ·T

[0060] Among them, G is the camera gain, T is the exposure time, and E ref It is the irradiance produced by the calibration target plane on the camera sensor plane.

[0061] The effective luminous flux collection area of ​​the lens is:

[0062]

[0063] The irradiance from the calibration target plane to the camera plane is:

[0064]

[0065] but:

[0066]

[0067] Through similarity relationship we get:

[0068]

[0069] The distance from the vine to the camera is:

[0070]

[0071] Among them, d ref To calibrate the distance from the target plane to the camera, R ref is the standard reflectivity of the calibration target plane, R is the standard reflectivity of the grapevine, ρ(θ) is the correction factor, θ ref is the incident angle of the calibration target plane relative to the camera line of sight, i ref is the pixel intensity value for calibrating the target plane, i is the corresponding pixel intensity value of the grapevine in the image, and κ is the comprehensive correction factor.

[0072] The width of the vine can then be calculated from the number of white width pixels in the image.

[0073] This method can be used to obtain the distance from the vine to the camera and the width of the vine, and the two bilateral cameras can detect them together to avoid unilateral sunlight interference and reduce the impact of grape leaf occlusion.

[0074] Since the distance between the grapevine and the tool is fixed when the grapevine is detected, the time when the tool starts to avoid obstacles can be calculated based on the driving speed, and the width of the grapevine and the distance from the camera to the grapevine are used to guide the obstacle avoidance range of the tool.

[0075] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A wine grape weeding device based on bilateral monocular real-time detection, characterized in that: The device comprises: an upper shading plate, a chassis, a terrain simulation push, a chessboard calibration target, a camera, a background plate, a tool mechanism and a universal travel switch; The upper sunshade is connected to the chassis; The terrain-mimicking push rod is installed on the inner wall of the chassis and connected to the tool mechanism, and the universal travel switch is installed at the bottom of the tool mechanism; The background plate is provided with a camera and a chessboard calibration target.

2. The wine grape weeding device based on bilateral monocular real-time detection according to claim 1, characterized in that: The terrain-simulating push rod is installed on the inner side of the chassis and connected to the tool mechanism. The terrain-simulating push rod moves up and down to drive the entire tool mechanism to move up and down.

3. The wine grape weeding device based on bilateral monocular real-time detection according to claim 1, characterized in that: The background plate uses a red opaque acrylic plate.

4. A control method for a wine grape weeding device based on bilateral monocular real-time detection, the control method being used to control the wine grape weeding device according to any one of claims 1 to 3, characterized in that: The method comprises: Use a camera to capture the initial image of the chassis during mobile weeding, perform color segmentation on the initial image, obtain the percentage of black area of ​​the grape vine, and use deep learning to calculate the threshold of the black area of ​​the grape vine; Compare the black area ratio and threshold of the grape vine, calibrate the camera based on the comparison result, collect several images containing checkerboards, and preprocess the images to obtain preprocessed images; Calculate the standard reflectance of the grapevine and the irradiance of the grapevine surface based on the preprocessed images; A comprehensive correction factor is introduced to calculate the distance from the grapevine to the camera based on standard reflectivity and irradiance, thus completing the control method of the wine grape weeding device with bilateral monocular real-time detection.

5. The control method of the wine grape weeding device based on bilateral monocular real-time detection according to claim 4 is characterized in that: The process of color segmenting the initial image specifically includes: The red and green colors in the initial image are converted to black, and the other colors are converted to white. The initial image after the color conversion is cropped so that the image border overlaps with the background border.

6. The control method of the wine grape weeding device based on bilateral monocular real-time detection according to claim 5 is characterized in that: The deep learning is a feed-forward neural network with three hidden layers, the first layer has 128 neurons, the second layer has 64 neurons, and the third layer has 32 neurons. Batch Normalization is added after each hidden layer, and the output layer is normalized using the sigmoid activation function.

7. The control method of the wine grape weeding device based on bilateral monocular real-time detection according to claim 4 is characterized in that: A comprehensive correction factor is introduced, and the method for calculating the distance from the grapevine to the camera based on the standard reflectivity and irradiance specifically includes: Among them, d ref To calibrate the distance from the target plane to the camera, R ref is the standard reflectivity of the calibration target plane, R is the standard reflectivity of the grapevine, ρ(θ) is the correction factor, θ ref is the incident angle of the calibration target plane relative to the camera line of sight, i ref is used to calibrate the pixel intensity value of the target plane, i is the intensity value of the corresponding pixel of the grapevine in the image, and κ is the comprehensive correction factor.

8. The control method of the wine grape weeding device based on bilateral monocular real-time detection according to claim 7 is characterized in that: The calculation formula for the calibration target plane pixel intensity value is: i ref =G·E ref ·T Among them, G is the camera gain, T is the exposure time, and E ref It is the irradiance produced by the calibration target plane on the camera sensor plane.

Images