Fruit picking method and system based on rgb-d sensor and computer

Abstract

The application provides a fruit picking method and system based on an RGB-D sensor and a computer, the method comprising: collecting color-depth images of target crops and target fruits, and performing fruit clustering analysis on the calibrated color-depth images under the limitation of color and depth to obtain a picking area interest image; performing point cloud processing on the picking area interest image to obtain a point cloud image of double-channel color and double-channel depth and a new picking area interest image of double-channel color and double-channel depth; using a symmetric structure coding-decoding deep neural network to detect pickable fruits from the point cloud image of double-channel color and double-channel depth and the new picking area interest image of double-channel color and double-channel depth to obtain a detection result corresponding to the pickable fruits, and picking the fruits according to the detection result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a fruit picking method and system based on an RGB-D sensor and a computer. BACKGROUND

[0002] Fruit picking robots usually need to use a visual system to identify and locate the target, and after a series of algorithm processing, the operation information is transmitted to the actuator, and the fruit is grabbed by the mechanical hand to complete the final picking task. In recent years, with the wide application of deep learning technology in the field of image processing, deep learning algorithm shows great superiority in fruit detection. In the prior art, two methods are usually used, 1) using YOLOX-Dense-CT network to detect cherry tomatoes. 2) Tomato string picking point identification and positioning method based on GB-D information fusion and target detection: first, the YOLOv4 target detection algorithm is used to quickly identify the current region (Region of Interest, ROI) of the tomato string and the pickable fruit stem, and then the depth information and color features in the RGB-D image are fused to accurately identify the picking point. The fruit stem image is extracted by a depth segmentation algorithm, morphological operation, K-means clustering algorithm and thinning algorithm.

[0003] The first method greatly deviates from the cherry tomato in the actual picking scene when the center point of the prediction box is equal to 0.5, which will result in a low success rate of the robot picking, and the second method first obtains the approximate target region on the RGB image by the deep learning algorithm, and then combines the depth information to make accurate positioning, so the depth information does not play a role in the deep learning method. SUMMARY

[0004] Therefore, the present application aims to provide a fruit picking method and system based on an RGB-D sensor and a computer to at least solve the above-mentioned deficiencies in the related art.

[0005] The present application provides a fruit picking method based on an RGB-D sensor, which comprises:

[0006] Color-depth images of target crops and target fruits are collected, and fruit clustering analysis is performed on the calibrated color-depth images based on the color and depth under the limited conditions to obtain picking region interest images;

[0007] Point cloud processing is performed on the picking region interest images to obtain point cloud images of double-channel color and double-channel depth and new picking region interest images of double-channel color and double-channel depth, respectively;

[0008] The symmetrical structure coding-decoding deep neural network is used for detecting the pickable fruits in the double-channel color and double-channel depth point cloud images and the double-channel color and double-channel depth picking region of interest new images, to obtain the detection results corresponding to the pickable fruits, and the fruits are picked according to the detection results.

[0009] Further, the fruit clustering analysis of the calibrated color-depth image under the color and depth limitation conditions is performed to obtain the picking region of interest image, and the steps include:

[0010] The calibrated color-depth image is sequentially subjected to filtering and denoising and alignment processing to obtain a preliminary processing image;

[0011] The preliminary processing image is subjected to color space transformation based on the RGB image, and the H channel is subjected to threshold segmentation to obtain a color segmentation result;

[0012] The depth channel in the preliminary processing image is subjected to threshold segmentation to obtain a depth segmentation result, and the color segmentation result and the depth segmentation result are respectively subjected to morphological filtering to fuse the color segmentation result and the depth image segmentation result to form a composite image;

[0013] The clustering parameters are calculated according to the mask image size, the output region of interest size and the minimum radius of the target fruit, and the mask image is clustered by using the clustering parameters to obtain a clustering output result;

[0014] According to the clustering output result, a plurality of clustering interest images are segmented from the composite image to obtain a picking region of interest image.

[0015] Further, the picking region of interest image is subjected to point cloud processing to obtain a double-channel color and double-channel depth point cloud image and a double-channel color and double-channel depth picking region of interest new image, and the steps include:

[0016] The depth image in the picking region of interest image is converted into point cloud data, and the normal vector of each point in the point cloud data is calculated;

[0017] The included angle between each point normal vector and the Z axis is calculated, and the picking region of interest image is converted into a double-channel color and double-channel depth point cloud image and a double-channel color and double-channel depth picking region of interest new image based on the included angle.

[0018] Further, the symmetrical structure encoding-decoding deep neural network is used for detecting the pickable fruits from the double-channel color and double-channel depth point cloud images and the double-channel color and double-channel depth picking region of interest new images, so as to obtain the detection results corresponding to the pickable fruits, and the fruit picking step according to the detection results includes:

[0019] The double-channel color and double-channel depth point cloud images and the double-channel color and double-channel depth picking region of interest new images are compared with the picking region of interest image to rebalance the color and depth information;

[0020] The color information and the depth information are fused by the decoder of the symmetrical structure encoding-decoding deep neural network to obtain the three-dimensional boundary vertex coordinates of the pickable fruits, the three-dimensional mass center of the pickable fruits and the picking probability of the pickable fruits;

[0021] The pickable fruits are picked according to the three-dimensional boundary vertex coordinates, the three-dimensional mass center and the picking probability.

[0022] The application further provides a fruit picking system based on an RGB-D sensor, which includes:

[0023] A clustering analysis module is configured to collect color-depth images of target crops and target fruits, and perform fruit clustering analysis on the calibrated color-depth images based on color and depth under a limited condition to obtain a picking region of interest image.

[0024] A point cloud processing module is configured to perform point cloud processing on the picking region of interest image to obtain double-channel color and double-channel depth point cloud images and double-channel color and double-channel depth picking region of interest new images.

[0025] A fruit picking module is configured to use a symmetrical structure encoding-decoding deep neural network to detect pickable fruits from the double-channel color and double-channel depth point cloud images and the double-channel color and double-channel depth picking region of interest new images, so as to obtain detection results corresponding to the pickable fruits, and pick the pickable fruits according to the detection results.

[0026] Further, the clustering analysis module includes:

[0027] A preliminary processing unit is configured to sequentially perform filtering and denoising and alignment processing on the calibrated color-depth images to obtain a preliminary processing image.

[0028] a threshold segmentation unit, configured to perform color space transformation on the preliminary processing image based on an RGB image, and perform threshold segmentation on an H channel to obtain a color segmentation result;

[0029] a data fusion unit, configured to perform threshold segmentation on a depth channel in the preliminary processing image to obtain a depth segmentation result, and perform morphological filtering on the color segmentation result and the depth segmentation result respectively to fuse the color segmentation result and the depth image segmentation result to form a composite image;

[0030] an image clustering unit, configured to calculate clustering parameters according to a mask image size, an output region of interest size and a minimum radius of the target fruit, and perform clustering on the mask image by using the clustering parameters to obtain a clustering output result;

[0031] a clustering analysis unit, configured to segment a plurality of clustered region of interest images from the composite image according to the clustering output result to obtain a picking region of interest image.

[0032] Further, the point cloud processing module comprises:

[0033] a point cloud processing unit, configured to convert a depth image in the picking region of interest image into point cloud data, and calculate a normal vector of each point in the point cloud data;

[0034] an image conversion unit, configured to calculate an included angle between each normal vector and a Z axis, and convert the picking region of interest image into a point cloud image of double-channel color and double-channel depth and a new picking region of interest image of double-channel color and double-channel depth based on the included angle.

[0035] Further, the fruit picking module comprises:

[0036] an image comparison unit, configured to perform image comparison between the point cloud image of double-channel color and double-channel depth and the new picking region of interest image of double-channel color and double-channel depth and the picking region of interest image to rebalance color and depth information;

[0037] a fusion processing unit, configured to perform fusion processing on color information and depth information by using a decoder of the symmetric structure encoding-decoding deep neural network to obtain a three-dimensional boundary vertex coordinate of the pickable fruit, a three-dimensional mass center of the pickable fruit and a picking probability of the pickable fruit;

[0038] a fruit picking unit, configured to perform fruit picking on the pickable fruit according to the three-dimensional boundary vertex coordinate, the three-dimensional mass center and the picking probability.

[0039] The application further provides a computer comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the fruit picking method based on the RGB-D sensor when executing the computer program.

[0040] Compared with the prior art, the application has the beneficial effects that: after the color-depth image is preliminarily segmented, the fruit picking interest image is obtained through density-based image clustering under the limitation conditions of neighborhood radius, maximum cluster number and neighborhood density threshold, the color space transformation and point cloud processing are performed on the color-depth image of the fruit picking interest image to obtain the normal vector angle, and the fruit picking interest image is converted into double-channel color and double-channel 3D information; the color and 3D information are balanced and fully fused by using the symmetric Y-shaped coding-decoding deep neural network suitable for the four-channel color-3D image, so that the fruit picking is performed on the pickable fruit according to the obtained detection result. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 a flowchart of the fruit picking method based on the RGB-D sensor in the first embodiment of the application;

[0042] Figure 2 a detailed flowchart of step S102 in the first embodiment of the application; Figure 1

[0043] Figure 3 a detailed flowchart of step S103 in the first embodiment of the application; Figure 1

[0044] Figure 4 a detailed flowchart of step S104 in the first embodiment of the application; Figure 1

[0045] a structure diagram of the Y-shaped coding-decoding deep neural network in the first embodiment of the application; Figure 5

[0046] a structure block diagram of the fruit picking system based on the RGB-D sensor in the second embodiment of the application; Figure 6

[0047] a structure block diagram of the computer in the third embodiment of the application. Figure 7 The following specific embodiments will further illustrate the application in combination with the above drawings.

[0048] DETAILED DESCRIPTION

[0049] ​​​For the purpose of promoting the understanding of the present application, reference will be made to a more full description of the application which follows, read in light of the accompanying drawings. The application is illustrated by a number of embodiments in the drawings. It should be understood, however, that the application can be practiced in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the application to those skilled in the art.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0051] Embodiment one

[0052] Referring to Figure 1 , a fruit picking method based on an RGB-D sensor in a first embodiment of the present application is shown, and the method specifically includes steps S101 to S103:

[0053] S101, color-depth images of target crops and target fruits are collected, and fruit clustering analysis is performed on the calibrated color-depth images based on color and depth under the defined conditions to obtain picking area interest images;

[0054] Referring to Figure 2 , the step S101 specifically includes steps S1011-S1015:

[0055] S1011, the calibrated color-depth images are sequentially subjected to filtering and denoising and alignment processing to obtain preliminary processing images;

[0056] S1012, the preliminary processing images are subjected to color space transformation based on RGB images, and threshold segmentation is performed on the H channel to obtain color segmentation results;

[0057] S1013, threshold segmentation is performed on the depth channel of the preliminary processing images to obtain depth segmentation results, and morphological filtering is performed on the color segmentation results and the depth segmentation results respectively to fuse the color segmentation results and the depth image segmentation results to form a composite image;

[0058] S1014, clustering parameters are calculated according to the size of the mask image, the output interest region size and the minimum radius of the target fruit, and the mask image is clustered using the clustering parameters to obtain clustering output results;

[0059] S1015, a plurality of clustering interest images are segmented from the composite image according to the clustering output results to obtain picking area interest images.

[0060] In a specific implementation, a picking robot with a color-depth (RGB-D) sensor mounted on an end effector is calibrated for camera and hand-eye, and after calibration, the picking robot is used to collect color-depth images of target crops and target fruits for subsequent image processing operations.

[0061] Further, the color (RGB) and depth (D) images are corrected and aligned according to the camera intrinsic and extrinsic parameters obtained from the camera calibration, and the calibrated color-depth images are sequentially filtered and denoised and aligned to obtain a preliminary processed image (I RGB , I D ), using RGB image color space transformation, for example: I HSV = T(I RGB ); the color image is color space transformed and segmented and processed, i.e. the H channel is threshold segmented cMask = H_low < I H < H_high, to obtain the color segmentation result cMask, and the depth image is segmented and processed, i.e. the D channel I D is threshold segmented dMask = D_low < I D < D_high, to obtain the depth segmentation result dMask, and the color segmentation result cMask and the depth segmentation result dMask are respectively morphologically filtered cMask_f = MorphFilter(cMask) and dMask_f = MorphFiter(dMask) to fuse the color segmentation result and the depth image segmentation result to form a composite image Mask(x, y, 1) = cMask_f && dMask_f, Mask(x, y, 2) = I D (x, y) ＊ dMask_f; according to the mask image size (M, N), the output region of interest size (P, Q) and the minimum radius R min of the target fruit, the clustering parameters are calculated: 0 < neighbor threshold radius e < min(P, Q) / 2, maximum cluster number C < [M / P] * [N / Q], D min < neighbor threshold density threshold D < max(P, Q) / R min , and the mask image is clustered using the above clustering parameters to obtain a clustering output result, and a plurality of clustered region of interest images are segmented from the composite image according to the clustering output result, and the information of each clustered region of interest image is output: starting point, original size and corresponding mixed image (u, v, W, H, ROI c The condition limitation based on fruit picking can provide the efficiency and stability of the clustering method and the result.

[0062] S102, performing point cloud processing on the picking area interest image to obtain a point cloud image of double-channel color and double-channel depth and a new picking area interest image of double-channel color and double-channel depth respectively;

[0063] Referring to Figure 3 , the step S102 specifically comprises steps S1021-S1022:

[0064] S1021, converting the depth image in the picking area interest image into point cloud data and calculating a normal vector of each point in the point cloud data;

[0065] S1022, calculating an included angle between each normal vector and the Z axis and converting the picking area interest image into a point cloud image of double-channel color and double-channel depth and a new picking area interest image of double-channel color and double-channel depth based on the included angle.

[0066] In a specific implementation, the RGB color space has a large amount of redundant information, for example, after the RGB color space is transformed according to the following formula, R * +G * +B * =1, so that all color information can be fully retained by using only two channels:

[0067]

[0068] Further, the information (u, v, W, H, ROI c ) of the obtained clustering interest image is subjected to RGB-D camera calibration to convert the depth image in the information of the clustering interest image into point cloud PC(x, y, z); the normal vector n(x, y, z) of each point in the point cloud PC(x, y, z) is calculated, and for the depth image, in order to more fully utilize 3D information, the depth image can be converted into a point cloud image according to the depth camera internal parameter, and the included angle A between the normal vector of each point in the point cloud image and the Z axis, i.e. Ang(x, y, z) is calculated:

[0069]

[0070] In the formula, n zx , n zy , n zz are respectively the x, y, z direction components of the normal vector n z .

[0071] Specifically, the clustering interest image is converted into a new composite image ROI_Input0:

[0072] ROI_Input0(x, y, 1) = ROI C (x, y, 1) color R ＊Channel, R ＊ = R / (R + G + B);

[0073] ROI_Input0(x, y, 2) = ROI c G in (x, y, 2) color ＊ Channel, G * = G / (R + G + B);

[0074] ROI_Input0(x, y, 3) = ROI c (xy, z) depth value;

[0075] ROI_InputO(x, y, 4) = ROI c (x, y, Ang(x, y, z)) normal vector and Z axis included angle;

[0076] Boundary filling is performed on the new composite image ROI_Input0 to obtain ROI_Input, which has a size of (P, Q, 4).

[0077] S103, using a symmetric structure encoding-decoding deep neural network to respectively perform pickable fruit detection on the double-channel color and double-channel depth point cloud image and the double-channel color and double-channel depth picking region of interest new image, to obtain a detection result corresponding to the pickable fruit, and performing fruit picking on the pickable fruit according to the detection result.

[0078] Please refer to Figure 4 , the step S103 specifically includes steps S1031-S1033:

[0079] S1031, image comparison is performed on the double-channel color and double-channel depth point cloud image and the double-channel color and double-channel depth picking region of interest new image and the picking region of interest image to rebalance the color and depth information;

[0080] S1032, the color information and the depth information are fused by a decoder of the symmetric structure encoding-decoding deep neural network to obtain three-dimensional boundary vertex coordinates of the pickable fruit, a three-dimensional mass center of the pickable fruit, and a picking probability of the pickable fruit;

[0081] S1033, fruit picking is performed on the pickable fruit according to the three-dimensional boundary vertex coordinates, the three-dimensional mass center, and the picking probability.

[0082] In the specific implementation, the 2 color channels and 2 3D channels of the fruit picking region ROI image and the original RGB-D image are compared, and the color and depth information is rebalanced to a certain extent. The depth neural network (ED-DNN, Encoder-Decoder Deep Neural Network) of the encoding-decoding structure is commonly used for target detection, and on the basis of the image balancing, the application proposes a Y-shaped structure ED-DNN ( Figure 5 ). The input of the two encoder branches of the Y-shaped structure ED-DNN is the symmetrical 2-channel color and 2-channel 3D information, the decoder part fuses the color and 3D information, and the output result is the 3D boundary vertex coordinates (X1, Y1, Z1, X2, Y2, Z2) of the detected fruit, the 3D mass center (X c , Y c , Z c ) of the fruit, and the pickable probability P of the fruit. The network needs to be trained by a large number of fruit sample images in a labeled picking scene to obtain the detection result, and the pickable fruit is picked according to the detection result.

[0083] In summary, the fruit picking method based on the RGB-D sensor in the above embodiment of the application performs preliminary segmentation processing on the color-depth image, performs density-based image clustering under the limitation conditions of neighborhood radius, maximum cluster number and neighborhood density threshold to obtain a fruit picking interest image, performs color space transformation and point cloud processing on the color-depth image of the fruit picking interest image to obtain a normal vector angle, and converts the fruit picking interest image into double-channel color and double-channel 3D information; the color and 3D information are balanced and fully fused by using the symmetrical Y-shaped structure encoding-decoding deep neural network suitable for the above four-channel color-3D image, so that the pickable fruit is picked according to the obtained detection result.

[0084] Embodiment two

[0085] Another aspect of the application also provides a fruit picking system based on an RGB-D sensor, please refer to Figure 6 , which is a fruit picking system based on an RGB-D sensor in the second embodiment of the application, comprising:

[0086] The clustering analysis module 11 is used for collecting color-depth images of target crops and target fruits, and performing fruit clustering analysis on the labeled color-depth images based on the limitation conditions of color and depth to obtain a picking region interest image;

[0087] Further, the clustering analysis module 11 comprises:

[0088] A preliminary processing unit is configured to sequentially perform filtering and denoising and alignment processing on the calibrated color-depth image to obtain a preliminary processing image;

[0089] A threshold segmentation unit is configured to perform color space transformation on the preliminary processing image based on an RGB image, and perform threshold segmentation on an H channel to obtain a color segmentation result;

[0090] A data fusion unit is configured to perform threshold segmentation on a depth channel in the preliminary processing image to obtain a depth segmentation result, and perform morphological filtering on the color segmentation result and the depth segmentation result respectively to fuse the color segmentation result and the depth image segmentation result to form a composite image;

[0091] An image clustering unit is configured to calculate clustering parameters according to a mask image size, an output region of interest size, and a minimum radius of the target fruit, and perform clustering on the mask image by using the clustering parameters to obtain a clustering output result;

[0092] A clustering analysis unit is configured to segment a plurality of clustered regions of interest images from the composite image according to the clustering output result to obtain a picking region of interest image.

[0093] A point cloud processing module 12 is configured to perform point cloud processing on the picking region of interest image to obtain a point cloud image of double-channel color and double-channel depth and a new picking region of interest image of double-channel color and double-channel depth respectively;

[0094] Further, the point cloud processing module 12 comprises:

[0095] A point cloud processing unit is configured to convert a depth image in the picking region of interest image into point cloud data, and calculate a normal vector of each point in the point cloud data;

[0096] An image conversion unit is configured to calculate an included angle between each normal vector and a Z axis, and convert the picking region of interest image into the point cloud image of double-channel color and double-channel depth and the new picking region of interest image of double-channel color and double-channel depth based on the included angle.

[0097] A fruit picking module 13 is configured to perform pickable fruit detection on the point cloud image of double-channel color and double-channel depth and the new picking region of interest image of double-channel color and double-channel depth by using a symmetric structure encoding-decoding deep neural network to obtain a detection result corresponding to the pickable fruit, and perform fruit picking on the pickable fruit according to the detection result.

[0098] Further, the fruit picking module 13 comprises:

[0099] An image comparison unit is configured to compare the dual-channel color and dual-channel depth point cloud image and the dual-channel color and dual-channel depth picked region of interest new image with the picked region of interest image to rebalance the color and depth information.

[0100] A fusion processing unit is configured to fuse the color information and the depth information by a decoder of the symmetric structure encoding-decoding deep neural network to obtain a three-dimensional boundary vertex coordinate of the pickable fruit, a three-dimensional mass center of the pickable fruit and a picking probability of the pickable fruit.

[0101] A fruit picking unit is configured to pick the pickable fruit according to the three-dimensional boundary vertex coordinate, the three-dimensional mass center and the picking probability.

[0102] The functions or operation steps realized when the above modules and units are executed are substantially the same as those of the above method embodiments, and thus will not be described herein.

[0103] The fruit picking system based on the RGB-D sensor provided in the embodiments of the present application has the same implementation principle and technical effects as the above method embodiments, and for brevity of description, the parts not mentioned in the system embodiments can be referred to the corresponding contents in the above method embodiments.

[0104] Embodiment Three

[0105] The present application also provides a computer, please refer to Figure 7 , which is a computer in the third embodiment of the present application, comprising a memory 10, a processor 20 and a computer program 30 stored in the memory 10 and executable on the processor 20, and the processor 20 implements the above-mentioned fruit picking method based on the RGB-D sensor when executing the computer program 30.

[0106] The memory 10 includes at least one type of storage medium, including flash memory, hard disk, multimedia card, card memory (such as SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. The memory 10 can be an internal storage unit of the computer in some embodiments, such as the hard disk of the computer. The memory 10 can also be an external storage device in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 10 can include both the internal storage unit and the external storage device of the computer. The memory 10 can be used not only to store application software and various data installed on the computer, but also to temporarily store data that has been output or will be output.

[0107] The processor 20 may, in some embodiments, be an electronic control unit (ECU), a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip, for running program codes stored in the memory 10 or processing data, such as executing the access restriction program.

[0108] It should be noted that, Figure 7 The illustrated structure does not constitute a limitation on the computer, which may, in other embodiments, include fewer or more components than shown, or combine certain components, or arrange the components differently.

[0109] The embodiment of the present application also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the fruit picking method based on an RGB-D sensor as described above.

[0110] Those skilled in the art can understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a list of executable instructions for implementing the logic function, which can be specifically implemented in any computer storage medium for use by or in conjunction with an instruction execution system, device or apparatus, such as a computer-based system, a system including a processor or other system that can fetch and execute instructions from the instruction execution system, device or apparatus. For the present specification, the "computer storage medium" can be any device that can contain, store, communicate, propagate or transport programs for use by or in conjunction with the instruction execution system, device or apparatus, or in conjunction with these instruction execution systems, devices or apparatus.

[0111] More specific examples (a non-exhaustive list) of the computer storage medium include the following: an electrical connection having one or more wires (electrical devices), a portable computer diskette (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer storage medium can even be paper or other suitable storage medium on which the program can be printed, because the program can be electronically obtained, for example, by optical scanning of the paper or other storage medium, followed by editing, interpreting or otherwise processing, if necessary, in other suitable ways, to be electronically obtained and then stored in the computer memory.

[0112] It should be understood that portions of the present application can be implemented with hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, known in the art, or a combination thereof, can be used: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0113] The technical features of the above-described embodiments can be combined in any manner. For the sake of brevity, not all possible combinations of the technical features in the above-described embodiments are described, however, as long as the combination of the technical features does not result in a contradiction, it should be considered within the scope of the present disclosure.

[0114] The above-described embodiments only express several implementation manners of the present application, and the description is specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of the patent of the present application should be subject to the appended claims.

Claims

1. A fruit picking method based on an RGB-D sensor, characterized in that, The method comprises the following steps: Collecting color-depth images of target crops and target fruits, and performing fruit clustering analysis on the calibrated color-depth images based on color and depth under defined conditions to obtain picking area interest images; Performing point cloud processing on the picking area interest images to obtain point cloud images of double-channel color and double-channel depth and new picking area interest images of double-channel color and double-channel depth, respectively; Using a symmetric structure encoding-decoding deep neural network to detect pickable fruits from the point cloud images of double-channel color and double-channel depth and the new picking area interest images of double-channel color and double-channel depth to obtain detection results corresponding to the pickable fruits, and picking the fruits according to the detection results; Wherein, using a symmetric structure encoding-decoding deep neural network to detect pickable fruits from the point cloud images of double-channel color and double-channel depth and the new picking area interest images of double-channel color and double-channel depth to obtain detection results corresponding to the pickable fruits, and picking the fruits according to the detection results, the step comprises: Comparing the point cloud images of double-channel color and double-channel depth and the new picking area interest images of double-channel color and double-channel depth with the picking area interest images to rebalance the color and depth information; Fusing the color information and the depth information through the decoder of the symmetric structure encoding-decoding deep neural network to obtain three-dimensional boundary vertex coordinates of the pickable fruits, three-dimensional mass centers of the pickable fruits, and picking probabilities of the pickable fruits; Picking the fruits according to the three-dimensional boundary vertex coordinates, the three-dimensional mass centers, and the picking probabilities. 2.The RGB-D sensor-based fruit picking method according to claim 1, wherein, The step of performing fruit clustering analysis on the calibrated color-depth images based on color and depth under defined conditions to obtain picking area interest images comprises: Performing filtering denoising and alignment processing on the calibrated color-depth images in sequence to obtain a preliminary processed image; Performing color space transformation on the preliminary processed image based on an RGB image, and performing threshold segmentation on an H channel to obtain a color segmentation result; Performing threshold segmentation on a depth channel in the preliminary processed image to obtain a depth segmentation result, and performing morphological filtering on the color segmentation result and the depth segmentation result, respectively, to fuse the color segmentation result and the depth image segmentation result to form a composite image; Calculating clustering parameters according to the size of a mask image, the size of an output interest region, and the minimum radius of the target fruit, and clustering the mask image using the clustering parameters to obtain a clustering output result; Segmenting a plurality of clustering interest images from the composite image according to the clustering output result to obtain picking area interest images. 3.The RGB-D sensor-based fruit picking method according to claim 1, wherein, The step of performing point cloud processing on the picking area interest images to obtain point cloud images of double-channel color and double-channel depth and new picking area interest images of double-channel color and double-channel depth comprises: convert the depth image in the picking region interest image into point cloud data, and calculate a normal vector of each point in the point cloud data; calculate an included angle between each of the normal vectors and a Z axis, and convert the picking region interest image into a point cloud image of double-channel color and double-channel depth and a new picking region interest image of double-channel color and double-channel depth based on the included angle.

4. A fruit picking system based on RGB-D sensor, characterized in that, Comprise: a clustering analysis module, configured to collect a color-depth image of a target crop and a target fruit, and perform fruit clustering analysis on the calibrated color-depth image based on a color and depth defined condition to obtain a picking region interest image; a point cloud processing module, configured to perform point cloud processing on the picking region interest image to obtain a point cloud image of double-channel color and double-channel depth and a new picking region interest image of double-channel color and double-channel depth, respectively; a fruit picking module, configured to perform pickable fruit detection on the point cloud image of double-channel color and double-channel depth and the new picking region interest image of double-channel color and double-channel depth by using a symmetric structure encoding-decoding deep neural network, to obtain a detection result corresponding to the pickable fruit, and perform fruit picking on the pickable fruit according to the detection result; wherein the fruit picking module comprises: an image comparison unit, configured to compare the point cloud image of double-channel color and double-channel depth and the new picking region interest image of double-channel color and double-channel depth with the picking region interest image to rebalance color and depth information; a fusion processing unit, configured to perform fusion processing on color information and depth information by using a decoder of the symmetric structure encoding-decoding deep neural network to obtain a three-dimensional boundary vertex coordinate of the pickable fruit, a three-dimensional mass center of the pickable fruit, and a picking probability of the pickable fruit; a fruit picking unit, configured to perform fruit picking on the pickable fruit according to the three-dimensional boundary vertex coordinate, the three-dimensional mass center, and the picking probability.

5. The RGB-D sensor based fruit picking system as claimed in claim 4, wherein, The clustering analysis module comprises: a preliminary processing unit, configured to sequentially perform filtering denoising and alignment processing on the calibrated color-depth image to obtain a preliminary processing image; a threshold segmentation unit, configured to perform color space transformation on the preliminary processing image based on an RGB image, and perform threshold segmentation on an H channel to obtain a color segmentation result; a data fusion unit, configured to perform threshold segmentation on a depth channel in the preliminary processing image to obtain a depth segmentation result, and perform morphological filtering on the color segmentation result and the depth segmentation result, respectively, to fuse the color segmentation result and the depth image segmentation result to form a composite image; an image clustering unit, configured to calculate clustering parameters according to a mask image size, an output interest region size, and a minimum radius of the target fruit, and perform clustering on the mask image by using the clustering parameters to obtain a clustering output result; a clustering analysis unit, configured to segment a plurality of clustering interest images from the composite image according to the clustering output result to obtain a picking region interest image.

6. The RGB-D sensor based fruit picking system as claimed in claim 4, wherein, The point cloud processing module comprises: a point cloud processing unit, configured to convert a depth image in the picking area of interest image into point cloud data, and calculate a normal vector of each point in the point cloud data; an image conversion unit, configured to calculate an angle between each normal vector and a Z axis, and convert the picking area of interest image into a point cloud image with two channels of color and two channels of depth and a new picking area of interest image with two channels of color and two channels of depth based on the angle.

7. A computer comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the fruit picking method based on the RGB-D sensor according to any one of claims 1 to 3 when executing the computer program.

Images