Grape picking method based on deep learning and deep information fusion

Abstract

The application discloses a grape picking method based on deep learning and deep information fusion, which comprises collecting pictures of grapes. In order to reduce the interference of background on image processing and make the image processing result more accurate in the process of obtaining grape picking points, the method introduces depth information in the process of contour detection, marks grape target points by comparing the depth information, and then converts the value of the marked points into a binary graph, thereby reducing the interference of similar colors and background information in the environment and improving the accuracy of contour detection. Since the ROI region of the grape contains grapes and a little background information, the method uses a depth calculation method in the grape contour to replace the method of using the average value of all depth values in the ROI region to predict the depth interval of the fruit stem, so that the obtained depth interval is smaller, the background information that can be excluded is more, the environmental interference is less, the result is more accurate, and only the grape needs to be detected, without detecting the fruit stem, thereby reducing the hardware requirement.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of intelligent fruit picking, and particularly relates to a grape picking method based on deep learning and depth information fusion. BACKGROUND

[0002] The fruit and vegetable market is huge, but the degree of mechanization is not high, among which the comprehensive mechanization level of vegetable production is only 20% to 30%, and the mechanization technology of many links is still in the laboratory and experimental demonstration stage and has not been applied in practice. The fruit mechanization level is even lower, especially the fruits planted in hilly areas, which are basically planted by farmers manually. With China entering an aging society, the labor force is decreasing, and the aging of agricultural labor force will directly affect the labor productivity.

[0003] For grapes, due to the complex growing environment and non-destructive picking requirements, grape picking still adopts manual picking. With the shortage of agricultural labor force and the increasing picking cost, the use of grape picking robots is of great significance and has broad market prospects

[0004] Accurate identification of grapes and stems is crucial for picking robots. So far, grape identification mainly falls into two categories: image processing methods and deep learning methods. Image processing methods such as Otsu algorithm, K-means algorithm, support vector machines, etc. These methods are based on the color, shape and texture of grapes. When grapes are subjected to uneven lighting and there is occlusion between grapes, the recognition accuracy will be significantly reduced. Deep learning methods such as two-stage RCNN, FasterRCNN, single-stage YOLOv3, BPNN (back propagation neural network) mainly train neural networks to obtain recognition models for detecting grapes and stems. This kind of method has high real-time performance, but has the disadvantage of high requirement for hardware. SUMMARY

[0005] The purpose of the present application is to solve the problems raised in the background art, and to provide a grape picking method based on deep learning and depth information fusion.

[0006] To achieve the above purpose, the technical solution adopted by the present application is as follows:

[0007] The grape picking method based on deep learning and depth information fusion proposed by the present application comprises collecting pictures of grapes, obtaining RGB images and depth images, and inputting the superimposed pixel points of the two into a trained grape recognition model to obtain the ROI region of the harvestable grapes.

[0008] The grape outline in the ROI region is obtained by using a Canny algorithm based on depth information, the ROI region of the pedicel is obtained by using a coefficient matrix and the ROI region of the grape, and then the depth interval of the pedicel is obtained by using the depth information in the grape outline.

[0009] The depth map in the ROI region of the pedicel is binarized by combining the depth interval of the pedicel, and a binary image of the pedicel is obtained.

[0010] The binary image of the pedicel is superimposed with the ROI region image mask of the pedicel to obtain an RGB image of the pedicel with the background removed, and then grayscale conversion, bilateral filtering and image thinning processing are performed to obtain a skeleton image of the pedicel.

[0011] The edges and skeleton lines with a large included angle with the vertical direction in the skeleton image of the pedicel are removed, and the midpoint of the longest skeleton line is calculated as the corresponding two-dimensional picking point.

[0012] The n*n range is selected with the two-dimensional picking point as the center, and each point in the range is traversed, the points with depth values not in the depth interval of the pedicel are removed, and the average value of the depth values of the remaining points is taken as the depth value of the picking point.

[0013] The two-dimensional picking point and the depth value of the picking point are combined to obtain a three-dimensional picking point, which is combined with the actuator of the mechanical arm for picking grapes to complete the picking of the grapes until all the grapes are picked.

[0014] Preferably, during training of the grape recognition model, a plurality of grape images are collected according to different angles, different weather conditions and different time periods, the region where the grape is located in each image is labeled, and the images are divided into training set images and verification set images, each training set image is identified and trained by using a deep learning model to obtain a grape recognition model, and the verification set images are used to verify the obtained grape recognition model.

[0015] Preferably, the grape recognition model is YOLOv5s.

[0016] Preferably, verifying the obtained grape recognition model with the verification set images includes: calculating the precision and recall rate of the grape recognition model, drawing a P-R curve of the precision and recall rate, and further calculating the average precision, and the larger the average precision value, the more reliable the grape recognition model.

[0017] and,

[0018]

[0019]

[0020] Wherein, Precision represents the precision of the grape recognition model, Recall represents the recall rate of the grape recognition model, TP represents the number of true positive samples, FP represents the number of false negative samples, and FN represents the number of true negative samples.

[0021] Preferably, the grape contour in the ROI region is obtained by using the canny algorithm based on depth information, comprising:

[0022] The average depth in the ROI region of the grape is used to eliminate part of the background, and all points in the ROI region of the grape are traversed from left to right and from top to bottom, the target grape point is marked according to the depth value of the target grape point and the depth value of the surrounding points, and the image is binarized according to the value of the marked point, and then input into the canny algorithm to detect the grape contour, and

[0023]

[0024] M(x,y)=f(x,y)*Sobel x (x,y)+f(x,y)*Sobel y (x,y).

[0025]

[0026] Wherein, d(x,y) is the depth value at point (x,y), δ is the depth difference threshold, f(x,y) is the marking value at point (x,y), M(x,y) is the gradient at point (x,y), θ M is the gradient direction at point (x,y), Sobel x (x,y), Sobel y (x,y) are third-order Sobel operators in horizontal and vertical directions respectively.

[0027] The ROI region of the fruit stem is obtained by using the coefficient matrix and the ROI region of the grape, comprising:

[0028] According to the position and size of the ROI region of the grape, a coefficient matrix is introduced, and the ROI region of the fruit stem is calculated and obtained as follows by using the ROI region of the grape, and represented in YOLO format:

[0029]

[0030]

[0031] Wherein, x1, y1, w1, h1 represent the ROI region of the grape in YOLO format, x stem , y stem , w stem , hstem The ROI region of the fruit stem in the YOLO format is represented, and p, s, t, is represented as a value between 0 and 1.

[0032] The depth interval of the fruit stem is obtained by using the depth information in the grape contour, including:

[0033] The depth interval of the fruit stem is obtained by using the depth information of the grape contour combined with the correction coefficient, as follows:

[0034] d min ≤ d ≤ d min + Δw.

[0035] And,

[0036]

[0037] wherein d represents the depth value of any point on the fruit stem, d min represents the minimum depth value in the grape contour, (x1, y1) represents the leftmost point in the grape contour, d(x1, y1) represents the depth value at point (x1, y1), (x2, y2) represents the rightmost point in the grape contour, d(x2, y2) represents the depth value at point (x2, y2), Δw represents the pixel width of the ROI region frame image of the fruit stem, and a and β represent correction coefficients.

[0038] Preferably, the depth map in the ROI region of the fruit stem is binarized by combining the depth interval of the fruit stem, to obtain a binary map of the fruit stem, including:

[0039] The value of the pixel point in the ROI region of the fruit stem that meets the depth value in the depth interval is set to 255, and the value of the pixel point that does not meet the depth value is set to 0, to obtain:

[0040]

[0041] wherein d(x, y) represents the depth value at point (x, y), and τ(x, y) represents the binarized value at point (x, y).

[0042] Preferably, the binary map of the fruit stem is superimposed with the ROI region mask of the fruit stem to obtain an RGB map of the fruit stem with the background removed, and then grayscale conversion, bilateral filtering, and image thinning processing are performed to obtain a skeleton map of the fruit stem, including:

[0043] Grayscale conversion:

[0044]

[0045] Bilateral filtering:

[0046]

[0047] Wherein, Gray(x, y) represents the gray value at point (x, y), R, G, B represent the numerical value of three channels in each pixel, a, b, c represent the coefficients on R, G, B channels in turn, G(i, j) represents the output point of point (i, j) after bilateral filter processing, w(i, j, x, y) represents the coefficient of bilateral filter.

[0048] Preferably, taking the two-dimensional picking point as the center, an n*n range is selected and each point in the range is traversed, the points with depth values not in the stem depth interval are removed, and the average value of the depth values of the remaining points is taken as the depth value of the picking point, and the formula is as follows:

[0049]

[0050]

[0051] And,

[0052]

[0053] Wherein, represents the depth value of the picking point, n represents the side length of the selected n*n range, w0, w1 represent the left and right boundary points on the stem binary graph respectively, h represents the height of the stem binary graph, and i is an integer with a value of 0-n 2 , m represents the number of points that meet the condition, [d min , d min +Δw] represents the depth interval of the stem.

[0054] Compared with the prior art, the beneficial effects of the present application are:

[0055] 1. In order to reduce the interference of the background on the image processing in the process of obtaining the grape picking point, the image processing result is more accurate, the depth information is introduced in the process of contour detection, the grape target point is marked by comparing the depth information, then the value of the marked point is converted into a binary graph, and the interference of the similar color and background information in the environment is reduced by combining the canny algorithm, so that the accuracy of the contour detection is improved.

[0056] 2. Since the ROI region of the grape contains grape and a little background information, the depth calculation method in the grape contour is used instead of the method of using the average value of all depth values in the ROI region to estimate the depth interval of the stem, so that the obtained depth interval is smaller, the background information that can be excluded is more, the environmental interference is less, the result is more accurate, and only the grape needs to be detected, without detecting the stem, so that the requirement for hardware is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1A flowchart of a grape picking method based on deep learning and deep information fusion. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present application.

[0059] It should be noted that when a component is referred to as being "connected" with another component, it can be directly connected with the other component or there can be a component in between. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application.

[0060] As shown in Figure 1 A grape picking method based on deep learning and deep information fusion, comprising:

[0061] S1, according to different angles, different weather conditions and different time periods, a plurality of pictures of grapes are collected, the area where the grapes are located in each picture is labeled, and the pictures are divided into training set pictures and validation set pictures, each training set picture is identified and trained by using a deep learning model to obtain a grape identification model, and the validation set pictures are used to verify the obtained grape identification model.

[0062] Specifically, 10000 pictures of grapes are collected by using Astra Pro camera (the model of the camera is not limited) according to different angles of backlight, front light and shadow, different weather conditions of sunny and cloudy days, and different time periods from morning to afternoon, the area where the grapes are located in each picture is labeled by using labelImg software, and the pictures are divided into training set pictures and validation set pictures according to a ratio of 9:1. The number of pictures of grapes and the ratio between the training set pictures and the validation set pictures can be set according to actual conditions and are not limited.

[0063] It should be noted that the grape identification model is YOLOv5s. Before training, the yaml file parameters are configured, nc is modified to 1, the number of training rounds is set to 300 rounds, batch-size is set to 64, and rectangular training is used. After training, the grape identification model is obtained. The number of training rounds can be set according to actual needs and is not limited. Only the grapes need to be detected, and the fruit stem does not need to be detected, which reduces the requirement for hardware.

[0064] The verification of the obtained grape recognition model by the verification set picture includes: calculating the precision and recall of the grape recognition model, drawing a P-R curve of the precision and the recall, and further calculating an average precision (AP, Average Precision), wherein the greater the average precision value is, the more reliable the grape recognition model is.

[0065] And,

[0066]

[0067]

[0068] wherein the Precision represents the precision of the grape recognition model, the Recall represents the recall of the grape recognition model, the TP represents the number of true positive samples, the FP represents the number of false negative samples, and the FN represents the number of true negative samples.

[0069] S2, collecting pictures of grapes, obtaining RGB images and depth images, and inputting the corresponding pixel points of the two to the trained grape recognition model to obtain the ROI region of the harvestable grapes.

[0070] Specifically, when obtaining the RGB image and the depth image, an AstraPro camera is used, and the specific model of the camera is not limited. The ROI (region of interest) region is an area of interest. In machine vision and image processing, the area to be processed is outlined in a box, a circle, an ellipse, an irregular polygon, etc., which is called an area of interest.

[0071] S3, obtaining the grape contour in the ROI region by using the canny algorithm based on depth information, obtaining the ROI region of the fruit stem by using the coefficient matrix and the ROI region of the grape, and then obtaining the depth interval of the fruit stem by using the depth information in the grape contour.

[0072] It should be noted that in addition to the target grape, there is a small amount of background, which can cause a large error in calculating the depth interval of the fruit stem. Therefore, by using the large difference between the depth value of the grape and the depth value of the background, the contour line of the grape in the ROI region is obtained by using the canny algorithm based on depth information.

[0073] The canny algorithm based on depth information is used to obtain the grape contour in the ROI region, which includes:

[0074] First, the average depth in the ROI region of the grape is used to eliminate part of the background, and then all points in the ROI region of the grape are traversed from left to right and top to bottom, and the target grape point is marked according to the depth value of the target grape point and the depth value of the surrounding points, the target grape point is marked as 1, and the rest of the points are marked as 0. After the image is binarized according to the value of the marked point, it is input into the canny algorithm to detect the grape contour, and

[0075]

[0076] M(x,y)=|f(x,y)*Sobel x (x,y)|+|f(x,y)*Sobel y (x,y)|;

[0077]

[0078] Wherein, d(x,y) is the depth value at point (x,y), δ is the depth difference threshold, i.e. the preset value, f(x,y) is the marked value at point (x,y), M(x,y) is the gradient at point (x,y), θ M is the gradient direction at point (x,y), Sobel x (x,y), Sobel y (x,y) are third-order Sobel operators in horizontal and vertical directions respectively.

[0079] The ROI region of the fruit stem is obtained by using the coefficient matrix and the ROI region of the grape, which comprises:

[0080] Based on the planting conditions and growth patterns of grapes, it can be obtained that the fruit stem is vertically grown and located above the grape. According to the position and size of the ROI region of the grape, a coefficient matrix is introduced, and the ROI region of the fruit stem is calculated and obtained through the ROI region of the grape as follows, and represented in YOLO format:

[0081]

[0082]

[0083] Wherein, x1, y1, w1, h1 represent the ROI region of the grape in YOLO format, x stem , y stem , w stem , h stem represent the ROI region of the fruit stem in YOLO format, ρ, σ, τ, represent values between 0 and 1;

[0084] The depth interval of the pedicel is obtained by using the depth information within the grape outline, and comprises:

[0085] The depth interval of the pedicel is obtained by using the depth information within the grape outline in combination with the correction coefficient, and comprises:

[0086] d min ≤d≤d min +Δw;

[0087] And,

[0088]

[0089] wherein d represents a depth value of any point on the pedicel, d min represents a minimum depth value within the grape outline, (x1, y1) represents a leftmost point in the grape outline, d(x1, y1) represents a depth value of the point (x1, y1), (x2, y2) represents a rightmost point in the grape outline, d(x2, y2) represents a depth value of the point (x2, y2), Δw represents a pixel width of the ROI region frame image of the pedicel, and α and β represent correction coefficients (the correction coefficient refers to a coefficient added to a calculation formula in order to make the formula as real as possible when there is a deviation between ideal and reality, reality and investigation, and the like).

[0090] S4, binarizing the depth map within the ROI region of the pedicel in combination with the depth interval of the pedicel to obtain a binary map of the pedicel.

[0091] The value of a pixel point within the ROI region of the pedicel that meets the depth value in the depth interval is set to 255, and the value of a pixel point that does not meet the depth value is set to 0, to obtain:

[0092]

[0093] wherein d(x, y) represents a depth value at a point (x, y), and τ(x, y) represents a value after binarization at the point (x, y).

[0094] S5, superimposing the binary map of the pedicel and the ROI region mask of the pedicel to obtain an RGB image of the pedicel with the background removed, and then performing grayscale conversion, bilateral filtering, and image thinning to obtain a skeleton map of the pedicel.

[0095] Grayscale conversion:

[0096]

[0097] Bilateral filtering:

[0098]

[0099] Wherein, Gray(x, y) represents the gray value at the point (x, y), R, G, B represent the numerical value of three channels in each pixel, a, b, c represent the coefficients on R, G, B channels in turn, G(i, j) represents the output point of the point (i, j) after bilateral filtering processing, and w(i, j, x, y) represents the coefficient of the bilateral filter.

[0100] The binary graph of the fruit stem is superimposed on the ROI area graph mask of the fruit stem: the binary graph of the fruit stem and the ROI area graph of the fruit stem are ANDed, the points with 1 on the binary graph of the fruit stem are retained as the original values of the corresponding points on the ROI area graph of the fruit stem, and the points with 0 on the binary graph of the fruit stem are also changed to 0.

[0101] S6, the edge in the skeleton graph of the fruit stem and the skeleton line with a larger included angle with the vertical direction are removed, and the midpoint of the longest skeleton line is calculated as the corresponding two-dimensional picking point.

[0102] S7, taking the two-dimensional picking point as the center, selecting an n*n range and traversing each point in the range, removing the points with depth values not in the depth interval of the fruit stem, and taking the average value of the depth values of the remaining points as the depth value of the picking point.

[0103] The formula is as follows:

[0104]

[0105]

[0106] And,

[0107]

[0108] Wherein, represents the depth value of the picking point, i.e. the distance from the camera to the picking point, n represents the side length of the selected n*n range, w0 and w1 represent the left and right boundary points on the fruit stem binary graph respectively, h represents the height of the fruit stem binary graph, and i takes an integer value of 0-n 2 , m represents the number of points that meet the condition, and [d min ,d min +Δw] represents the depth interval of the fruit stem.

[0109] Step S8, combining the two-dimensional picking point and the depth value of the picking point to obtain a three-dimensional picking point, and cooperating with the actuator of the mechanical arm for picking grapes to complete the picking of the grapes until all the grapes are picked.

[0110] Specifically, after completing the picking of one ROI region of the grapes, repeating S2-S8 to continue picking the next ROI region of the grapes until all the grapes are picked.

[0111] Any combination of the technical features in the above-described embodiments can be made, and for the sake of brevity, not all possible combinations are described, however, as long as there is no conflict, any combination of the technical features should be considered within the scope of the present disclosure.

[0112] The above-described embodiments are merely representative and detailed embodiments described in the present application, but should not be construed as limiting the scope of the patent application. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of the present application. Therefore, the scope of the patent of the present application should be subject to the appended claims.

Claims

1. A grape harvesting method based on deep learning and deep information fusion, characterized in that: The grape harvesting method based on deep learning and deep information fusion includes: Collect images of grapes, obtain RGB images and depth images, and then overlay the corresponding pixels of the two images and input them into a trained grape recognition model to obtain the ROI region of the grapes that can be picked. The grape contour within the ROI region is obtained using the depth-based Canny algorithm. The ROI region of the fruit stem is obtained using the coefficient matrix and the ROI region of the grape. Then, the depth range of the fruit stem is obtained using the depth information within the grape contour. The depth map within the ROI region of the fruit stalk is binarized by combining the depth range of the fruit stalk to obtain a binary map of the fruit stalk. The binary image of the fruit stalk is superimposed with the ROI region image of the fruit stalk to obtain the RGB image of the fruit stalk after removing the background. Then, grayscale conversion, bilateral filtering and image thinning are performed to obtain the skeleton image of the fruit stalk. Remove the edges of the fruit stalk in the skeletal diagram and the skeletal lines with large angles to the vertical direction, and select the midpoint of the longest skeletal line to be the corresponding two-dimensional picking point. Centered on the two-dimensional picking point, select an n×n range and traverse each point within the range, remove points whose depth values ​​are not in the fruit stem depth range, and take the average depth value of the remaining points as the depth value of the picking point. By combining the two-dimensional picking point and the depth value of the picking point, a three-dimensional picking point is obtained. Then, in conjunction with the actuator of the robotic arm for picking grapes, the grapes are picked until all grapes have been picked. The step of obtaining the grape contour within the ROI region using the depth-based Canny algorithm includes: The background is partially removed using the average depth within the grape's Region of Interest (ROI). All points within the ROI are traversed from left to right and top to bottom. Based on the depth values ​​of the target grape point and its surrounding points, the target grape point is marked. The image is then binarized using the marked points and fed into the Canny algorithm to detect the grape contour. M(x,y)=|f(x,y)*Sobel x (x,y)|+|f(x,y)*Sobel y (x,y)|; Where d(x,y) is the depth value at point (x,y), δ is the depth difference threshold, f(x,y) is the label value at point (x,y), M(x,y) is the gradient at point (x,y), and θ M Let Sobel be the gradient direction at point (x,y). x (x,y), Sobel y (x,y) are the third-order Sobel operators in the horizontal and vertical directions, respectively; The method of obtaining the ROI region of the fruit stalk using the coefficient matrix and the ROI region of the grape includes: Based on the location and size of the ROI region of the grape, a coefficient matrix is ​​introduced. The ROI region of the fruit stalk was calculated using the ROI region of the grape, and is shown below in YOLO format: Where x1, y1, w1, and h1 represent the Regions of Interest (ROIs) of the grape in YOLO format, x stem y stem w stem h stem Represented as the ROI region of the fruit stalk in YOLO format, ρ, σ, τ, Values ​​represented as 0-1; The step of then using depth information within the grape outline to obtain the depth range of the fruit stalk includes: By combining the depth information of the grape outline with a correction coefficient, the depth range of the fruit stalk is obtained as follows: d min ≤d≤d min +Δw; and, Where d represents the depth value at any point on the fruit stalk, d min d(x1,y1) represents the minimum depth value within the grape outline, (x1,y1) represents the leftmost point in the grape outline, d(x1,y1) represents the depth value at point (x1,y1), (x2,y2) represents the rightmost point in the grape outline, d(x2,y2) represents the depth value at point (x2,y2), Δw represents the pixel width of the ROI region of the fruit stem in the frame image, and α and β represent correction coefficients.

2. The grape harvesting method based on deep learning and deep information fusion as described in claim 1, characterized in that: During the training of the grape recognition model, several images of grapes are collected from different angles, under different weather conditions, and at different time periods. The area where the grapes are located in each image is labeled and divided into training set images and validation set images. Each training set image is used to train a deep learning model to obtain the grape recognition model, and the obtained grape recognition model is verified using the validation set images.

3. The grape harvesting method based on deep learning and deep information fusion as described in claim 1, characterized in that: The grape recognition model is YOLOv5s.

4. The grape harvesting method based on deep learning and deep information fusion as described in claim 2, characterized in that: The verification of the obtained grape recognition model using validation set images includes: calculating the precision and recall of the grape recognition model, plotting the precision and recall as PR curves, and further calculating the average precision. The higher the average precision value, the more reliable the grape recognition model is. and, Where Precision represents the accuracy of the grape recognition model, Recall represents the recall rate of the grape recognition model, TP represents the number of true positive samples, FP represents the number of false negative samples, and FN represents the number of true negative samples.

5. The grape harvesting method based on deep learning and deep information fusion as described in claim 1, characterized in that: The method of binarizing the depth map within the ROI region of the fruit stalk by combining the depth range of the fruit stalk to obtain a binary map of the fruit stalk includes: Within the ROI region of the fruit stalk, pixels whose depth values ​​fall within the specified depth range are set to 255, and pixels whose depth values ​​do not fall within the specified depth range are set to 0. This yields: Where d(x,y) represents the depth value at point (x,y), and τ(x,y) represents the binarized value at point (x,y).

6. The grape harvesting method based on deep learning and deep information fusion as described in claim 5, characterized in that: The process involves overlaying a binary image of the fruit stalk with a mask of its region of interest (ROI) to obtain an RGB image of the fruit stalk after background removal. This is followed by grayscale conversion, bilateral filtering, and image thinning to obtain the skeletal image of the fruit stalk. Grayscale conversion: Bilateral filtering: Where Gray(x,y) represents the gray value at point (x,y), R, G, and B represent the values ​​of the three channels in each pixel, a, b, and c represent the coefficients on the R, G, and B channels respectively, G(i,j) represents the output point after bilateral filtering of point (i,j), and w(i,j,x,y) represents the coefficients of the bilateral filter.

7. The grape harvesting method based on deep learning and deep information fusion as described in claim 1, characterized in that: The process involves selecting an n×n range centered on the two-dimensional picking point, traversing each point within that range, removing points whose depth values ​​are outside the fruit stem depth range, and taking the average depth value of the remaining points as the picking point depth value. The formula is as follows: and, in, The depth of the picking point is represented by , n represents the side length of the selected n×n range, w0 and w1 represent the left and right boundary points on the binary graph of the fruit stalk, respectively, h represents the height of the binary graph of the fruit stalk, and i takes values ​​from 0 to n. 2 An integer, m represents the number of points that match, [d min ,d min +Δw] represents the depth range of the fruit stalk.

Images