A high-speed tomato fruit collection system, method and tomato fruit picking machine

Abstract

The present invention provides a high-speed tomato fruit collection system, method and tomato fruit picking machine, which includes a tomato picking robot, a collection mechanism, a lidar, a first image acquisition device, a second image acquisition device and a control unit; the present invention fuses the tomato fruit image and the point cloud data of the radar to obtain the three-dimensional coordinates of the key points for tomato fruit picking, drives the tomato picking robot to grab the fruit, rotates above the collection mechanism and releases it, uses visual guidance to identify the size of the tomato and locate the falling pose of the tomato, drives the cable to collect the tomato fruit through a cylindrical fruit collector, and the control unit controls the narrowing or widening of the mouth of the cylindrical fruit collector through an air pump. Compared with the traditional fruit collection device, the present invention has simple operation, precise and efficient collection, reduces the tomato collection damage rate, completes the sorting of the fruits, and realizes the integration of tomato fruit picking and collection.

Description

Technical Field

[0001] The present invention belongs to the technical field of fruit and vegetable picking equipment, and particularly relates to a high-speed tomato fruit collection system and method, and a tomato fruit picking machine. Background Art

[0002] At present, the mechanization level of fruit and vegetable picking and collection in China is not high, and the collection of fruits of many fruits and vegetables still relies on a large amount of manual picking and collection. At the present stage, manual picking is still the first choice of the vast majority of fruit farmers. When carrying out manual picking, the fruits are easily dropped on the ground and thus damaged, and this method not only requires a large amount of manpower but also has very low efficiency. For the existing picking robots, the configured fruit collection device installs a pipeline under the robotic arm, and after the tomatoes are picked, they slide down along the pipeline, so the tomatoes will be damaged when they fall below the collection device, or after the picking robot grabs the tomatoes, the robotic arm is controlled to continue to execute and place the tomatoes into the collection device below, and the tomatoes are damaged when they fall, and the picking efficiency is low. Summary of the Invention

[0003] In view of the above technical problems, the present invention provides a high-speed tomato fruit collection system and method. The present invention fuses the tomato fruit image and the point cloud data of the lidar to obtain the three-dimensional coordinates of the key points for tomato fruit picking, drives the tomato picking robot to grab the fruit, rotates above the collection mechanism and releases it, identifies the size of the tomato and locates the dropping pose of the tomato through visual guidance, drives the cable to collect the tomato fruit through the cylindrical fruit collector, and the control unit controls the narrowing or widening of the barrel mouth of the cylindrical fruit collector through an air pump. Compared with the traditional fruit collection device, the present invention is simple to operate, collects accurately and efficiently, reduces the tomato collection damage rate, completes the sorting of the fruits, and realizes the integration of tomato fruit picking and collection.

[0004] The present invention also provides a tomato fruit picking machine including the above high-speed tomato fruit collection system, which is used for picking tomatoes, has a stable operation, improves the picking efficiency, and reduces the tomato breakage rate.

[0005] The present invention is realized through the following technical solutions:

[0006] A high-speed tomato fruit collection system includes a tomato picking robot, a collection mechanism, a lidar, a first image acquisition device, a second image acquisition device, and a control unit;

[0007] The tomato picking robot is equipped with a first image acquisition device and a lidar. The end of the tomato picking robot is provided with an end effector of the robotic arm. The first image acquisition device is used to collect RGB images of tomato fruits and stalks and transmit them to the control unit. The lidar is used to collect point cloud data of tomato fruits and stalks and transmit them to the control unit. The control unit fuses the RGB images of tomato fruits and stalks with the point cloud data to obtain the three-dimensional coordinates of tomato fruits and stalks, and controls the end effector of the robotic arm to pick tomatoes.

[0008] The collection mechanism includes a mobile platform, an end effector of the collection device, and a fruit collection device. The fruit collection device is located inside the mobile platform, and the end effector of the collection device is located above the fruit collection device. The end effector of the collection device is provided with a cylindrical fruit collector with upper and lower openings. The cylindrical fruit collector is connected to an air pump, and the air pump is connected to the control unit. The second image acquisition device is used to collect RGB images of the tomato fruits falling from the end effector of the robotic arm and transmit them to the control unit, calculate the relative pose between the tomato fruits and the mobile platform, control the end effector of the collection device to move the cylindrical fruit collector to the position where the tomato fruits fall, and the control unit controls the air pump to narrow and expand the fruit collector to collect the falling tomato fruits.

[0009] In the above solution, the control unit includes an RGB image processing module, a depth image processing module, a tomato picking control module, a vision guidance and air pump control module.

[0010] The RGB image processing module is used to collect and process the RGB images of tomato fruits obtained by the first image acquisition device to obtain the two-dimensional coordinates of tomato fruits and stalks.

[0011] The depth image processing module is used to preprocess the point cloud data of tomato fruits and stalks collected by the lidar to obtain a dense depth map, combine the RGB images of tomato fruits and stalks, extract the depths of tomato fruits and stalks, and obtain the three-dimensional coordinates of tomato fruits and stalks.

[0012] The tomato picking control module is used to control the tomato picking robot to pick according to the three-dimensional coordinates of tomato fruits and stalks.

[0013] The vision guidance module is used to calculate the relative pose between the tomato fruits falling from the end effector of the robotic arm collected by the second image acquisition device and the mobile platform of the collection device, and transmit it to the control unit. The control unit controls the end effector of the collection device to move the cylindrical fruit collector to the position where the tomato fruits fall.

[0014] The air pump control module is used to control the fruit collector to first narrow to catch the tomato fruits, and then expand to make the tomato fruits fall into the fruit collection device.

[0015] In the above solution, the collection mechanism further includes a plurality of cables. One end of each cable is connected to the mobile platform, and the other end is connected to the end effector of the collection device. The control unit controls the extension and retraction of the cables through the end effector of the collection device, so that the cylindrical fruit collector moves to the position where the tomato fruits fall.

[0016] Furthermore, the mobile platform includes a plurality of mobile bases, which are arranged around the fruit collection device and connected to the fruit collection device.

[0017] In the above solution, the cylindrical fruit collector includes an inner inflatable layer and an outer inflatable layer; the inner inflatable layer is located inside the ring of the outer inflatable layer, the height of the inner inflatable layer is lower than that of the outer inflatable layer, and the inner inflatable layer is connected to an air pump. The control unit controls the inner inflatable layer to inflate first to catch the tomato fruits, and then deflate to make the tomato fruits fall into the fruit collection device.

[0018] A tomato fruit picking machine includes the above-mentioned tomato fruit high-speed collection system.

[0019] In the above solution, the tomato fruit picking machine further includes a mobile trolley; the tomato picking robot and the collection mechanism are arranged on the mobile trolley; the control unit is connected to the control system of the mobile trolley.

[0020] A control method according to the above-mentioned tomato fruit high-speed collection system includes the following steps:

[0021] Tomato picking: The first image acquisition device acquires the RGB images of the tomato fruits and the fruit stalks and transmits them to the control unit. The lidar acquires the point cloud data of the tomato fruits and the fruit stalks and transmits them to the control unit. The control unit fuses the images of the tomato fruits and the fruit stalks with the point cloud data to obtain the three-dimensional coordinates of the tomato fruits and the fruit stalks, and controls the end effector of the robotic arm to pick the tomatoes.

[0022] Vision-guided tomato collection: The second image acquisition device acquires the images of the tomato fruits falling from the end effector of the robotic arm and transmits them to the control unit. The control unit calculates the relative pose between the tomato fruits and the mobile platform, controls the end effector of the collection device to move to the position where the tomato fruits fall, and controls the air pump to inflate when picking up the tomatoes so that the mouth of the cylindrical fruit collector narrows to catch the tomato fruits. When the tomato fruits are picked up, the control unit controls the air pump to suck air to expand the mouth, so that the tomatoes fall into the fruit collection device below.

[0023] In the above solution, it further includes the identification of the three-dimensional coordinates of the key points for tomato fruit picking, which are specifically obtained through the following steps:

[0024] Tomato fruit depth map processing: The point cloud data of tomato fruits and fruit stalks collected by lidar is used to generate a sparse depth image by an upsampling method in the form of bilateral filtering. The lidar tomato fruit point cloud data is projected onto the 2D RGB image plane of the tomato fruit and fruit stalk, and it is aligned with the image through coordinate transformation using a transformation matrix to obtain the coordinate values on the RGB image and the coordinates on the depth image. The obtained sparse depth image is again subjected to an upsampling method to supplement the details of the depth map;

[0025] Tomato fruit picking key point positioning: The RGB image of tomato fruits is collected by the first image acquisition device. Multiple picking key points of the tomato fruit area and the fruit stalk are identified through a CNN neural network. Depth map processing is performed on the lidar point cloud data and the RGB image, the point clouds of the tomato fruit and the fruit stalk are extracted and separated, the depth of the tomato fruit picking key points is obtained, and a filter is used to eliminate the depth noise of the picking key points to obtain the three-dimensional coordinates of the tomato picking key points.

[0026] In the above solution, the specific steps of the vision-guided tomato collection are as follows:

[0027] The second image acquisition device collects the point cloud data of the tomato fruits falling at the end effector of the robotic arm and sends it to the control unit. The control unit performs target recognition and pose estimation on the tomato fruits according to the point cloud data, and calculates the relative pose between the tomato fruits and the moving platform of the collection device. The control unit performs hand-eye calibration on the end effector of the collection device and the second image acquisition device, and calculates the static balance condition between the end effector of the collection device and the tomato fruits. The tilting condition of the end effector of the collection device is calculated by the hyperplane shifting method, and a collection instruction is issued to control the elongation or contraction of the driving cable of the end effector of the collection device to move the cylindrical fruit collector to the position where the tomato falls. When the tomato falls, the air pump is controlled to inflate, and the cylindrical fruit collector narrows, so that the fruit falls into the cylindrical fruit collector. Then, the air pump is controlled to inhale, and the cylindrical fruit collector expands, and the fruit falls into the bottom fruit collection device.

[0028] Compared with the prior art, the beneficial effects of the present invention are:

[0029] The present invention fuses the tomato fruit image and the lidar point cloud data to obtain the three-dimensional coordinates of the tomato fruit picking key points, drives the tomato picking robot to grab the fruit, rotates it above the collection mechanism and releases it. Through vision guidance, the size of the tomato is recognized and the falling pose of the tomato is located. The driving cable is used to collect the tomato fruits through the cylindrical fruit collector. The control unit controls the narrowing or expansion of the mouth of the cylindrical fruit collector through the air pump. Compared with the traditional fruit collection device, the present invention is simple to operate, accurate and efficient in collection, reduces the tomato collection damage rate, completes the sorting of fruits, and realizes the integration of tomato fruit picking and collection. Description of the Drawings

[0030] Figure 1 is a schematic structural diagram of an embodiment of the present invention;

[0031] Figure 2 is a visual guidance architecture diagram of an embodiment of the present invention;

[0032] Figure 3 is a flowchart of target recognition and pose estimation of an embodiment of the present invention;

[0033] Figure 4 is a schematic diagram of the positions of key tomato picking points of an embodiment of the present invention;

[0034] Figure 5 is a schematic structural diagram of a cylindrical fruit collector of an embodiment of the present invention.

[0035] In the figure: 1 - mobile platform, 2 - cable, 3 - end effector of the collection device, 4 - cylindrical fruit collector, 401 - inner inflation layer, 402 - outer inflation layer, 5 - depth camera, 6 - spring blade, 7 - end effector of the robotic arm, 8 - industrial camera, 9 - upper arm, 10 - middle arm, 11 - lower arm, 12 - base, 13 - fruit collection device, 14 - key picking points. Detailed Embodiment

[0036] The embodiments of the present invention will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present invention, but should not be construed as limiting the present invention.

[0037] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "front", "rear", "left", "right", "upper", "lower", "axial", "radial", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as limiting the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, "a plurality" means two or more unless otherwise specifically defined.

[0038] In the present invention, unless otherwise clearly specified and defined, terms such as "install", "connect", "join", "fix", etc. shall be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or an integral connection; it may be a mechanical connection or an electrical connection; it may be a direct connection or an indirect connection through an intermediate medium, and it may be the communication inside two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0039] Embodiment 1

[0040] Figure 1 Shown is a preferred embodiment of the tomato fruit high-speed collection system. The tomato fruit high-speed collection system includes a tomato picking robot, a collection mechanism, a lidar, a first image acquisition device, a second image acquisition device, and a control unit.

[0041] The tomato picking robot is provided with a first image acquisition device and a lidar. At the end of the tomato picking robot, there is an end effector 7 of the robotic arm. The first image acquisition device is used to acquire the RGB images of the tomato fruit and the fruit stalk and transmit them to the control unit. The lidar is used to acquire the point cloud data of the tomato fruit and the fruit stalk and transmit them to the control unit. The control unit fuses the RGB images of the tomato fruit and the fruit stalk with the point cloud data to obtain the three-dimensional coordinates of the tomato fruit and the fruit stalk, and controls the end effector 7 of the robotic arm to pick the tomato.

[0042] The collection mechanism includes a mobile platform 1, an end effector 3 of the collection device, and a fruit collection device 13. The fruit collection device 13 is located inside the mobile platform 1. The end effector 3 of the collection device is located above the fruit collection device 13. The end effector 3 of the collection device is provided with a fruit collector 4 in the shape of a cylindrical tube with upper and lower openings. The cylindrical fruit collector 4 is connected to an air pump, and the air pump is connected to the control unit. The second image acquisition device is used to acquire the RGB image of the tomato fruit falling from the end effector 7 of the robotic arm and transmit it to the control unit, calculate the relative pose between the tomato fruit and the mobile platform 1, control the end effector 3 of the collection device to move the cylindrical fruit collector 4 to the position where the tomato fruit drops, and the control unit controls the air pump to narrow and expand the fruit collector 4 to collect the dropped tomato fruit.

[0043] The control unit includes an RGB image processing module, a depth image processing module, a tomato picking control module, a vision guidance module, and an air pump control module. The RGB image processing module is used to collect and process the RGB images of tomato fruits and stalks obtained by the first image acquisition device to obtain the two-dimensional coordinates of the tomato fruits and stalks. The depth image processing module is used to preprocess the point cloud data of tomato fruits and stalks collected by the lidar to obtain a dense depth map, and combine it with the RGB images of tomato fruits and stalks to extract the depths of the tomato fruits and stalks to obtain the three-dimensional coordinates of the tomato fruits and stalks. The tomato picking control module is used to control the tomato picking robot to pick according to the three-dimensional coordinates of the tomato fruits and stalks. The vision guidance module is used to calculate the relative pose between the falling tomato fruit image of the end effector 7 of the robotic arm collected by the second image acquisition device and the moving platform 1 of the collection device, and transmit it to the control unit. The control unit controls the end effector 3 of the collection device to move the cylindrical fruit collector 4 to the position where the tomato fruit falls. The air pump control module is used to control the fruit collector 4 to first narrow to catch the tomato fruit, and then expand to make the tomato fruit fall into the fruit collection device 13.

[0044] The collection mechanism further includes multiple cables 2. One end of the cable 2 is connected to the moving platform 1, and the other end is connected to the end effector 3 of the collection device. The control unit controls the telescopic movement of the cable 2 through the end effector 3 of the collection device to move the cylindrical fruit collector 4 to the position where the tomato fruit falls. The cable 2 moves quickly, and the fruit collector 4 can accurately and quickly reach the fruit falling position. Moreover, the cylindrical fruit collector 4 is made of flexible material, and the control unit controls the air pump to adjust the size of the mouth of the cylindrical fruit collector 4 according to the size of the tomato to reduce the damage rate during tomato collection.

[0045] According to this embodiment, preferably, the moving platform 1 includes multiple moving bases, which are arranged around the fruit collection device 13 and connected to the fruit collection device 13.

[0046] As Figure 5 shown, according to this embodiment, preferably, the cylindrical fruit collector 4 is made of flexible material. The cylindrical fruit collector 4 includes an inner inflatable layer 401 and an outer inflatable layer 402. The inner inflatable layer 401 is located inside the ring of the outer inflatable layer 402. The height of the inner inflatable layer 401 is lower than that of the outer inflatable layer 402. The inner inflatable layer 401 is connected to the air pump. The control unit controls the inner inflatable layer 401 to first inflate to catch the tomato fruit, and then deflate to make the tomato fruit fall into the fruit collection device 13.

[0047] According to this embodiment, preferably, the tomato picking robot includes a base 12, a robotic arm, and an end effector 7 of the robotic arm. The rotating joints of the robotic arm are rotatably installed on the base 12. The robotic arm includes a large arm 11, a middle arm 10, and a small arm 9 that are connected in sequence. The rotating joint of the end effector 7 of the robotic arm is installed on the small arm 9 at the end of the robotic arm.

[0048] The base 12 of the robotic arm contains a second driving mechanism, which controls the slewing motion of the robotic arm through this second driving mechanism. Third driving mechanisms are respectively built in the joints where the large arm 11 is connected to the middle arm 10 and the middle arm 10 is connected to the small arm 9 to control the rotation of the relative rotating joints. A fourth driving mechanism is built in the base between the small arm 9 and the end effector 7 of the robotic arm to control the slewing motion of the end effector 7. A control method according to the tomato fruit high-speed collection system includes the following steps:

[0049] Tomato picking: The first image acquisition device acquires the RGB images of the tomato fruit and the fruit stalk and transmits them to the control unit. The lidar acquires the point cloud data of the tomato fruit and the fruit stalk and transmits them to the control unit. The control unit fuses the images of the tomato fruit and the fruit stalk with the point cloud data to obtain the three-dimensional coordinates of the tomato fruit and the fruit stalk, and controls the end effector 7 of the robotic arm to pick the tomato.

[0050] Vision-guided tomato collection: The second image acquisition device acquires the image of the tomato fruit falling from the end effector 7 of the robotic arm and transmits it to the control unit. The control unit calculates the relative pose between the tomato fruit and the mobile platform 1, and controls the end effector 3 of the collection device to move to the position where the tomato fruit drops. When picking up the tomato, the control unit controls the air pump to inflate to narrow the opening of the cylindrical fruit collector 4 to catch the tomato fruit. When the tomato fruit is picked up, the control unit controls the air pump to inhale to expand the opening, so that the tomato drops into the fruit collection device 13 below.

[0051] As Figure 4 shown, according to this embodiment, preferably, the three-dimensional coordinates of the key points for picking tomato fruits are specifically obtained through the following steps:

[0052] Tomato fruit depth map processing: The point cloud data of the tomato fruit and the fruit stalk collected by the lidar is used to generate a sparse depth image by using an upsampling method in the form of bilateral filtering. The lidar tomato fruit point cloud data is projected onto the 2D RGB image plane of the tomato fruit and the fruit stalk, and it is aligned with the image through coordinate transformation using a transformation matrix to obtain the coordinate values on the RGB image and the coordinates on the depth image. The obtained sparse depth image is again subjected to an upsampling method to supplement the details of the depth map.

[0053] Key point positioning for tomato fruit picking: Collect the RGB image of the tomato fruit through the first image acquisition device, identify multiple picking key points 14 of the tomato fruit area and the fruit stalk through the deep learning method of the CNN neural network, perform depth map processing on the lidar point cloud data and the RGB image, extract and separate the point cloud of the tomato fruit and the fruit stalk, obtain the depth of the picking key points of the tomato fruit, and use a filter to eliminate the depth noise of the picking key points to obtain the three-dimensional coordinates of the tomato picking key points.

[0054] According to this embodiment, preferably, the specific steps of the vision-guided tomato collection are as follows:

[0055] The second image acquisition device collects the point cloud data of the tomato fruit falling at the end effector 7 of the robotic arm and sends it to the control unit. The control unit performs target recognition and pose estimation on the tomato fruit according to the point cloud data, calculates the relative pose between the tomato fruit and the moving platform 1 of the collection device, calibrates the hand-eye relationship between the end effector 3 of the collection device and the second image acquisition device, and calculates the static balance condition between the end effector 3 of the collection device and the tomato fruit. Calculate the tilting condition of the end effector 3 of the collection device by the hyperplane shift method, issue a collection command, control the elongation or contraction of the cable 2 driven by the end effector 3 of the collection device to move the cylindrical fruit collector 4 to the position where the tomato falls. When the tomato falls, control the air pump to inflate, and the cylindrical fruit collector 4 narrows, so that the fruit falls into the cylindrical fruit collector 4. When it is determined that the tomato fruit is collected on the cylindrical collector 4, control the air pump to inhale, the cylindrical fruit collector 4 expands, and the fruit falls into the bottom fruit collection device 13 along the trend.

[0056] According to this embodiment, preferably, the first image acquisition device is an industrial camera 8, and the second image acquisition device is a depth camera 5.

[0057] The working principle of the present invention is as follows:

[0058] Data acquisition: Collect information on the target tomato through the lidar and the industrial camera 8. The industrial camera 8 collects the RGB image of the tomato fruit and transmits it to the control unit. The lidar collects the point cloud data of the tomato fruit and transmits it to the control unit;

[0059] Data preprocessing: The data received by the control unit is often affected by environmental noise and interference and cannot be directly used. Filter and denoise the data through filtering, map the point cloud data of the lidar to the RGB image plane, and obtain a depth map aligned with the camera image through depth sampling technology;

[0060] The collection mechanism described in this invention features a mobile platform 1 at its base, consisting of four motor-mounted trolleys fixedly connected together. This platform occupies a small footprint and offers flexible mobility. A cylindrical fruit collector 4 is positioned within the collection device's end effector 3 to collect fallen tomatoes. This cylindrical fruit collector 4 is made of flexible material and has openings at both ends, with an air pump controlling the size of the opening. When collecting tomatoes, the air pump inflates the opening, narrowing it. Once the tomatoes are collected, the air pump inflates the opening, expanding it and allowing the tomatoes to fall into the fruit collection device 13 at the bottom. This design boasts a sophisticated structure and simple control principles, ensuring the integrity of the tomatoes during collection. A cable 2 is connected to and controlled by the collection device's end effector 3. A depth camera 5 is positioned adjacent to the device for visual guidance. Once the depth camera 5 locates the coordinates of the fallen tomatoes, a control unit controls the vertical movement of the cable 2 to facilitate flexible collection by the collection device's end effector 3, achieving both efficiency and speed.

[0061] The present invention designs an easy-to-build collection mechanism. Multiple cables 2 are connected to a mobile platform 1 and a collection device end effector 3. The collection device end effector 3 is controlled by a motor and can extend or retract the cables 2. The collection device end effector 3 can be equipped with various accessories, including hooks, cameras, and robotic grippers. The collection device end effector 3 is equipped with a cylindrical fruit collector 4 made of flexible material for collecting tomato fruits. A depth camera 5 serves as visual hardware to calibrate the tomato fruits. The purpose of calibration is to obtain the relationship between the image pixel coordinates and the world coordinate system.

[0062] Vision Guidance: The point cloud information captured by the depth camera 5 is transmitted to the control unit for depth processing. This is primarily accomplished using the PCL point cloud library. The control unit performs tomato fruit calibration, which involves both depth camera calibration and hand-eye calibration. Depth camera 5 calibration determines the relationship between the image coordinate system and the camera coordinate system, while hand-eye calibration determines the relationship between the camera coordinate system and the collection mechanism coordinate system.

[0063] Combine Figure 2 and 3 As shown, the visual guidance principle of the method of the present invention is explained:

[0064] Visual guidance can locate the position and orientation of tomatoes, and estimate the pose and speed of the falling tomatoes to control the end effector 3 of the collection device to collect the falling tomato fruits. The depth camera 5 is responsible for collecting visual images. The machine vision algorithm is embedded in the control unit. The depth camera 5 can collect the RGB image of the tomato fruit and the corresponding depth image in the current scene. Using the object detection algorithm based on deep learning, the tomato fruit is recognized, and the central position x, y, height h, and width w of the target tomato are obtained, presenting an accurate two-dimensional prediction box. The point cloud information collected by the depth camera 5 is transmitted to the control unit for in-depth processing of the point cloud information. The main method is to use the PCL point cloud library to process the collected point cloud data, extract the feature points of the tomato fruit and calculate its feature descriptors, and pair them with the tomato point cloud in the PCL point cloud library, so as to identify the target detection object and provide its three-dimensional coordinates. The calibration of the control unit for tomato fruits is divided into depth camera calibration and hand-eye calibration. Since the depth camera 5 obtains pixel coordinates and the end effector 3 of the collection device is in space coordinates, hand-eye calibration is to obtain the coordinate transformation between the pixel coordinate system and the space coordinate system of the end effector 3 of the collection device. In actual control, after the depth camera 5 detects the pixel position of the target in the image, the pixel coordinates of the camera are transformed to the space coordinate system of the end effector 3 of the collection device through the calibrated coordinate transformation matrix, and then how each cable 2 should move is calculated according to the coordinate system of the end effector 3 of the collection device, so as to control the end effector 3 of the collection device to reach the specified position. This process involves image calibration, image processing, forward and inverse kinematics, hand-eye calibration, etc. The control system consists of an upper computer and underlying hardware devices. The point cloud data collected by the depth camera is transmitted to the upper computer for point cloud preprocessing, target recognition and pose estimation are performed through the fruit recognition and positioning algorithm, and an access instruction is issued after obtaining the three-dimensional coordinates of the tomato fruit. The underlying hardware device controls the cable drive device to collect the fruit.

[0065] According to this embodiment, preferably, the specific steps of processing the depth map of tomato fruits are as follows:

[0066] Step M1, point cloud data processing: Generate a depth image from the lidar point cloud data by using the upsampling method in the form of bilateral filtering;

[0067] Step M2, data fusion: Project the lidar point cloud data onto the RGB image plane of the tomato fruit;

[0068] Step M3, coordinate generation: Use the transformation matrix to align it with the image through coordinate transformation to obtain the coordinate values on the RGB image and the coordinates on the depth image;

[0069] Step M4, depth map processing: Take the upsampling method again for the obtained sparse depth map to supplement the details of the depth map.

[0070] According to this embodiment, preferably, the specific steps for positioning the key points of tomato fruit picking are as follows:

[0071] Step N1, Image acquisition: Collect RGB images of tomato fruits through an industrial camera;

[0072] Step N2, Fruit recognition and key point detection: Solve the tomato fruit region and the N key points of the fruit through deep learning methods;

[0073] Step N3, Depth map processing: Perform depth map processing on the lidar point cloud data and the RGB image;

[0074] Step N4, Extraction and separation of tomato fruit point cloud;

[0075] Step N5, Extract the depth of the key points for tomato fruit picking;

[0076] Step N6, Use a filter to eliminate the depth noise of the picking key points;

[0077] Step N7, Obtain the three-dimensional coordinates of the picking key points.

[0078] According to this embodiment, preferably, the specific steps for the manipulator to drive and realize tomato fruit picking are as follows:

[0079] Step D1, Manipulator drive: After obtaining the three-dimensional coordinates of the precise tomato fruit picking point, the control unit controls the drive joint drive mechanism to move the end effector 7 of the manipulator to the corresponding position;

[0080] Step D2, End effector drive: Drive the end effector 7 of the manipulator to grasp the tomato fruit;

[0081] Step D3, Spring blade drive: Pop out the spring blade to cut the stem of the tomato fruit;

[0082] Step D4, Base drive: Drive the base of the picking robot to drive, so that the end effector 7 of the manipulator is placed above the tomato fruit collection device;

[0083] Step D5, End effector drive: Release the end effector 7 of the manipulator and the tomato falls.

[0084] According to this embodiment, preferably, the specific steps for visual guidance are as follows:

[0085] Step K1, Data acquisition: The depth camera 5 collects the point cloud data of tomato fruits;

[0086] Step K2, Data processing: The control unit preprocesses the point cloud data;

[0087] Step K3, Target recognition: The control unit performs target recognition and pose estimation on the tomato fruit;

[0088] Step K4, Depth camera calibration: The control unit calculates the relative pose between the tomato fruit and the moving platform of the collection device;

[0089] Step K5, Hand-eye calibration: The control unit performs hand-eye calibration on the end effector 3 of the collection device and the depth camera 5;

[0090] Step K6, Communication: Send a collection instruction to drive the end effector 3 of the collection device to collect the fallen fruit.

[0091] According to this embodiment, preferably, the specific operation steps of the tomato fruit cable are as follows:

[0092] Step S1, The end effector 3 of the collection device drives the elongation and contraction of the cable 2 to control the movement of the cylindrical fruit collector 4 so that it reaches the position where the tomato falls;

[0093] Step S2, Calculate the static balance condition between the end effector 3 of the collection device and the moving platform 1;

[0094] Step S3, Calculate the tilting condition of the end effector 3 of the collection device by the hyperplane shift method;

[0095] Step S4, When the tomato falls, the control unit controls the air pump to inflate according to the size of the tomato, the mouth of the cylinder becomes smaller, and the fruit falls on the cylindrical fruit collector 4;

[0096] Step S5, When it is determined that the tomato fruit is collected on the cylindrical collector, control the air pump to inhale, the mouth of the cylinder becomes larger, and the fruit falls into the bottom fruit collection device 13.

[0097] According to this embodiment, preferably, the process of target tomato recognition is as follows:

[0098] Step F1, Collect more tomato fruit samples to obtain 3D data;

[0099] Step F2, Collect the point cloud data of the tomato fruit samples;

[0100] Step F3, Extract the feature points of the tomato fruit samples and calculate the feature descriptors;

[0101] Step F4, Import the object model and the feature descriptors into the tomato database.

[0102] According to this embodiment, preferably, the process of the pose estimation is as follows:

[0103] Step Q1, The depth camera 5 obtains the point cloud data of the tomato fruit;

[0104] Step Q2, Preprocess the point cloud data;

[0105] Step Q3: Extract the feature points of the tomato fruit sample and calculate the feature descriptors;

[0106] Step Q4: Register the tomato fruit point cloud;

[0107] Step Q5: Combine with the object database for target recognition and output the recognition result.

[0108] The present invention uses a method that combines lidar and an industrial camera to locate the three-dimensional coordinates of tomato fruits, drives a tomato picking robot to grasp the fruits, rotates to above the fruit collection device 13 and releases them, uses visual guidance to locate the falling pose of the tomatoes, drives a cable to collect tomato fruits through a cylindrical collector made of a flexible material, and the cylindrical collector is controlled by an internal air pump. Compared with traditional fruit collection devices, it avoids fruit damage caused by tomato extrusion, improves the efficiency of fruit collection, completes the sorting of fruits, and realizes the integration of tomato fruit picking and collection.

[0109] Embodiment 2

[0110] A tomato fruit picking machine includes the tomato fruit high-speed collection system described in Embodiment 1, and thus has the beneficial effects of Embodiment 1, which will not be elaborated here. The tomato fruit picking machine is characterized in that it further includes a mobile trolley; the tomato picking robot and the collection mechanism are arranged on the mobile trolley; the control unit is connected to the control system of the mobile trolley.

[0111] It should be understood that although this specification is described according to each embodiment, not every embodiment only contains an independent technical solution. This narrative way of the specification is only for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0112] The series of detailed descriptions listed above are only specific descriptions of the feasible embodiments of the present invention, and they are not used to limit the protection scope of the present invention. Any equivalent embodiments or changes made without departing from the technical spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A high-speed tomato fruit collection system, characterized in that, It includes a tomato picking robot, a collection mechanism, a lidar, a first image acquisition device, a second image acquisition device, and a control unit; The first image acquisition device and the lidar are provided on the tomato picking robot, and an end effector (7) of a robotic arm is provided at the end of the tomato picking robot; the first image acquisition device is used to acquire RGB images of tomato fruits and fruit stalks and transmit them to the control unit, and the lidar is used to acquire point cloud data of tomato fruits and fruit stalks and transmit them to the control unit. The control unit fuses the RGB images of tomato fruits and fruit stalks with the point cloud data to obtain the three-dimensional coordinates of tomato fruits and fruit stalks, and controls the end effector (7) of the robotic arm to pick tomatoes; The collection mechanism includes a mobile platform (1), an end effector (3) of the collection device, and a fruit collection device (13). The fruit collection device (13) is located inside the mobile platform (1), and the end effector (3) of the collection device is located above the fruit collection device (13). The end effector (3) of the collection device is provided with a cylindrical fruit collector (4) with upper and lower openings. The cylindrical fruit collector (4) is connected to an air pump, and the air pump is connected to the control unit; the second image acquisition device is used to acquire RGB images of the tomato fruits falling from the end effector (7) of the robotic arm and transmit them to the control unit, calculate the relative pose between the tomato fruits and the mobile platform (1), control the end effector (3) of the collection device to move the cylindrical fruit collector (4) to the position where the tomato fruits fall, and the control unit controls the air pump to narrow and expand the fruit collector (4) to collect the falling tomato fruits; The control unit includes an RGB image processing module, a depth image processing module, a tomato picking control module, a visual guidance module, and an air pump control module; The RGB image processing module is used to acquire and process the RGB images of tomato fruits and fruit stalks obtained by the first image acquisition device to obtain the two-dimensional coordinates of tomato fruits and fruit stalks; The depth image processing module is used to preprocess the point cloud data of tomato fruits and fruit stalks collected by the lidar to obtain a dense depth map, combine it with the RGB images of tomato fruits and fruit stalks, extract the depth of tomato fruits and fruit stalks, and obtain the three-dimensional coordinates of tomato fruits and fruit stalks; The tomato picking control module is used to control the tomato picking robot to pick according to the three-dimensional coordinates of tomato fruits and fruit stalks; The visual guidance module is used to calculate the relative pose between the tomato fruits falling from the end effector (7) of the robotic arm collected by the second image acquisition device and the mobile platform (1) of the collection device, and transmit it to the control unit. The control unit controls the end effector (3) of the collection device to move the cylindrical fruit collector (4) to the position where the tomato fruits fall; The air pump control module is used to control the fruit collector (4) to first narrow to catch the tomato fruits, and then expand to make the tomato fruits fall into the fruit collection device (13); The cylindrical fruit collector (4) includes an inner inflatable layer (401) and an outer inflatable layer (402); the inner inflatable layer (401) is located inside the ring of the outer inflatable layer (402), the height of the inner inflatable layer (401) is lower than that of the outer inflatable layer (402), and the inner inflatable layer (401) is connected to an air pump; The control unit controls the inner inflatable layer (401) to inflate first to catch the tomato fruits, and then deflate to make the tomato fruits fall into the fruit collection device (13); The control unit controls the air pump to adjust the size of the opening of the cylindrical fruit collector (4) according to the size of the tomatoes.

2. The high-speed tomato fruit collection system according to claim 1, wherein The collection mechanism further includes a plurality of cables (2), one end of the cable (2) is connected to the mobile platform (1), and the other end is connected to the end effector (3) of the collection device. The control unit controls the telescopic movement of the cable (2) through the end effector (3) of the collection device, so that the cylindrical fruit collector (4) moves to the position where the tomato fruits fall.

3. The high-speed tomato fruit collection system according to claim 2, characterized in that The mobile platform (1) includes a plurality of mobile bases, and the mobile bases are arranged around the fruit collection device (13) and connected to the fruit collection device (13).

4. A tomato fruit picking machine, characterized in that, It includes the high-speed tomato fruit collection system according to any one of claims 1-3.

5. The tomato fruit picking machine according to claim 4, characterized in that, It further includes a mobile cart; the tomato picking robot and the collection mechanism are arranged on the mobile cart; the control unit is connected to the control system of the mobile cart.

6. A control method for the high-speed tomato fruit collection system according to any one of claims 1-3, characterized in that, It includes the following steps: Tomato picking: The first image acquisition device acquires the RGB images of the tomato fruits and the fruit stalks and transmits them to the control unit. The lidar acquires the point cloud data of the tomato fruits and the fruit stalks and transmits them to the control unit. The control unit fuses the images of the tomato fruits and the fruit stalks with the point cloud data to obtain the three-dimensional coordinates of the tomato fruits and the fruit stalks, and controls the end effector (7) of the robotic arm to pick the tomatoes; Vision-guided tomato collection: The second image acquisition device acquires the image of the tomato fruits falling from the end effector (7) of the robotic arm and transmits it to the control unit. The control unit calculates the relative pose between the tomato fruits and the mobile platform (1), controls the end effector (3) of the collection device to move to the position where the tomato fruits fall, and controls the air pump to inflate when picking up the tomatoes so that the opening of the cylindrical fruit collector (4) narrows to catch the tomato fruits. When the tomato fruits are picked up, the control unit controls the air pump to inhale to expand the opening, so that the tomatoes fall into the fruit collection device (13) below.

7. The control method of the high-speed tomato fruit collection system according to claim 6, characterized in that, It further includes the identification of the three-dimensional coordinates of the key points for tomato fruit picking, which is specifically obtained through the following steps: Tomato fruit depth map processing: The point cloud data of the tomato fruits and the fruit stalks collected by the lidar is used to generate a sparse depth image by the upsampling method in the form of bilateral filtering. The lidar tomato fruit point cloud data is projected onto the 2D RGB image plane of the tomato fruits and the fruit stalks, and is aligned with the image through coordinate transformation using a transformation matrix to obtain the coordinate values on the RGB image and the coordinates on the depth image. The obtained sparse depth image is again subjected to the upsampling method to supplement the details of the depth map; Tomato fruit picking key point positioning: Collect the RGB image of the tomato fruit through the first image acquisition device, identify multiple picking key points (14) of the tomato fruit area and the fruit stalk through the CNN neural network, perform depth map processing on the lidar point cloud data and the RGB image, extract and separate the point clouds of the tomato fruit and the fruit stalk, obtain the depth of the tomato fruit picking key points, and use a filter to eliminate the depth noise of the picking key points to obtain the three-dimensional coordinates of the tomato picking key points.

8. The control method of the high-speed tomato fruit collection system according to claim 6, characterized in that The specific steps of the vision-guided tomato collection are as follows: The second image acquisition device collects the point cloud data of the tomato fruit falling at the end effector (7) of the robotic arm and sends it to the control unit. The control unit performs target recognition and pose estimation on the tomato fruit based on the point cloud data, and calculates the relative pose between the tomato fruit and the moving platform (1) of the collection device. The control unit calibrates the hand-eye relationship between the end effector (3) of the collection device and the second image acquisition device, and calculates the static balance condition between the end effector (3) of the collection device and the tomato fruit. The tilting condition of the end effector (3) of the collection device is calculated by the hyperplane shift method, and a collection instruction is issued to control the elongation or contraction of the cable (2) driven by the end effector (3) of the collection device to move the cylindrical fruit collector (4) to the position where the tomato falls. When the tomato falls, the air pump is controlled to inflate, and the cylindrical fruit collector (4) narrows, so that the fruit falls into the cylindrical fruit collector (4). Then, the air pump is controlled to inhale, and the cylindrical fruit collector (4) expands, and the fruit falls into the bottom fruit collection device (13).

Images