Intelligent picking and collecting system and control method thereof

Abstract

The application relates to the technical field of agricultural picking, in particular to an intelligent picking and collecting system and a control method thereof. The intelligent picking and collecting system comprises a collecting mechanism, a controller, an executing mechanism and a clamping mechanism in communication connection with the controller, wherein the controller is used for controlling the executing mechanism, the clamping mechanism and the collecting mechanism; the executing mechanism is used for adjusting the pose of the clamping mechanism under the control of the controller; the clamping mechanism is used for picking crops under the control of the controller; and the collecting device is used for collecting the crops picked by the clamping mechanism. The depth camera arranged on the clamping mechanism is used for realizing the identification and classification of the objects to be picked; and the object bowl added to the clamping mechanism is used for conveniently and quickly storing the picked objects.

Description

Technical Field

[0001] This invention relates to the field of agricultural harvesting technology, and in particular to an intelligent harvesting and collection system and its control method. Background Technology

[0002] my country is a major agricultural country with a large area dedicated to the cultivation of fruits and vegetables. Traditional agriculture relies heavily on manual harvesting; however, in large-scale agricultural production, crop harvesting is a massive undertaking, and timely harvesting is crucial to ensuring crop quality, making it the most arduous part of the entire operation. Furthermore, the quality of harvested crops directly impacts subsequent processing, significantly affecting economic benefits. To address these issues, high-value crop harvesting robots have become a key development area in agricultural technology, with the design of their harvesting and collection systems being paramount. However, with rising labor costs year after year, labor shortages, harvesting difficulties, and increased costs have become major obstacles to the development of the high-value crop industry. Summary of the Invention

[0003] This invention addresses the major problems hindering the development of high-economic-value agricultural industries, namely, rising labor costs, labor shortages, harvesting difficulties, and increased costs. It provides an intelligent harvesting and collection system and its control method. The system utilizes a depth camera in the gripping mechanism to identify and classify the harvested items, and adds a container to the gripping mechanism for convenient and rapid collection of the harvested goods.

[0004] The intelligent harvesting and collection system proposed in this invention includes a collection mechanism, a controller, and an execution mechanism and a gripping mechanism that communicate with the controller. The execution mechanism is used to adjust the position of the gripping mechanism under the control of the controller, the gripping mechanism is used to harvest crops under the control of the controller, and the collection device is used to collect the crops harvested by the gripping mechanism.

[0005] The controller includes a depth camera, an image processing module, and a control module. The depth camera is used to acquire color and depth images of the crop within the same scene. The image processing module includes an image acquisition unit, an image recognition unit, an image segmentation unit, an image fusion unit, a point cloud data calculation unit, a pose calculation unit, and a data conversion unit.

[0006] The image acquisition unit is used to receive and store color images and depth images;

[0007] The image recognition unit is used to identify crops in color images;

[0008] The image segmentation unit is used to segment the specific contours of the crop based on the recognition results of the image recognition unit to obtain the contours to be extracted;

[0009] The image fusion unit is used to register and fuse color images and depth images to obtain a registered depth image;

[0010] The point cloud data computing unit is used to process the registered depth image based on the contour to be extracted, and to calculate the point cloud dataset of the crop by combining the intrinsic and extrinsic parameters of the depth camera.

[0011] The pose calculation unit is used to fit the point cloud dataset to obtain the pose data of the crop in the depth camera coordinate system;

[0012] The data conversion unit is used to convert the crop's pose data in the depth camera coordinate system to pose data in the actuator's coordinate system;

[0013] The control module is used to control the actuator and gripping mechanism based on the crop's pose data in the actuator's coordinate system.

[0014] Preferably, the gripping mechanism includes an electrically controlled gripper assembly for picking, a drive rod, a driven rod, a loading bowl, a first synchronous belt gear, a second synchronous belt gear, a synchronous belt, a mounting bracket mounted on the electrically controlled gripper assembly, a motor mounted on the mounting bracket, a depth camera, and a micro servo motor;

[0015] One end of the drive rod is fixedly mounted on the output shaft of the motor, and the other end is hinged to the container bowl. The drive rod is used to drive the container bowl to rotate around the output shaft of the motor.

[0016] The container bowl is fixedly connected to the second synchronous belt gear near the driven rod end. The container bowl is used to hold the items picked up by the electronically controlled gripper assembly. The initial position of the container bowl is located below the electronically controlled gripper assembly.

[0017] The synchronous belt is tensioned between the first synchronous belt gear and the second synchronous belt gear;

[0018] The first synchronous belt gear is fixedly mounted on the output shaft of the micro servo motor, and drives the cargo bowl to rotate through the synchronous belt and the second synchronous belt gear.

[0019] One end of the driven rod is hinged to the end face of the first synchronous belt gear, and the other end is hinged to the end face of the second synchronous belt gear. The driven rod is used to support the container without interfering with the flipping of the container.

[0020] The optical axis of the depth camera is parallel to the central axis of the electronically controlled gripper assembly; the central axis of the motor's output shaft, the central axis of the first synchronous belt gear, and the central axis of the micro servo motor coincide; the lever arm of the active lever when rotating and the lever arm of the driven lever when rotating are parallel and perpendicular to the central axis of the motor's output shaft.

[0021] The gripping mechanism includes at least two sets of finger assemblies, a base, a pneumatic tube, and an air source. The base is fixedly mounted on the actuator, and the finger assemblies are mounted opposite each other on the base. The finger assemblies are provided with air inlets. One end of the pneumatic tube is connected to the finger assembly through the pneumatic inlet, and the other end of the pneumatic tube is connected to the air source.

[0022] Preferably, the electrically controlled gripper assembly is equipped with a flange for fixing and connecting external components.

[0023] Preferably, the connection section between the container and the drive rod is a straight rod structure, and the connection section between the container and the second synchronous belt gear is also a straight rod structure, with their central axes coinciding; the middle section of the container is a bowl-shaped structure.

[0024] Preferably, the picking end of the electronically controlled gripper assembly is provided with a plastic pad for flexible picking.

[0025] Preferably, the actuator is a harvesting robotic arm with three to seven degrees of freedom.

[0026] The control method for the intelligent harvesting and collection system provided by this invention, implemented using the intelligent harvesting and collection system, specifically includes the following steps:

[0027] S1: Establish an intelligent harvesting and collection system;

[0028] S2: Use a depth camera to acquire images of crops against the same background, obtaining depth and color images;

[0029] S3: The controller processes the depth and color images through the image processing module to obtain the crop's pose data in the coordinate system of the actuator;

[0030] S4: The control module adjusts the gripping mechanism to the target pose based on the crop's pose data in the coordinate system of the actuator, and then controls the gripping mechanism to harvest the crop.

[0031] S5: The control module controls the actuator to transport the crops picked by the clamping mechanism to the collection mechanism, thus completing the picking and collection of the crops.

[0032] Preferably, in step S4, the control module sends an opening or closing command to the gripping mechanism to harvest the crop.

[0033] Preferably, the controller determines the position state of the gripping mechanism by receiving the current position feedback from the actuator in real time.

[0034] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0035] 1. This invention enables the identification and classification of crops by setting a depth camera in the gripping mechanism.

[0036] 2. This invention provides a convenient and quick way to store harvested crops by adding a container to the clamping mechanism. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the intelligent harvesting and collection system provided according to an embodiment of the present invention;

[0038] Figure 2 This is a schematic diagram of an actuator mounted on a traveling mechanism according to an embodiment of the present invention;

[0039] Figure 3 This is a schematic diagram of the clamping mechanism provided in an embodiment of the present invention;

[0040] Figure 4 This is a flowchart illustrating the control method of the intelligent harvesting and collection system provided in an embodiment of the present invention.

[0041] Reference numerals: 1. Gripping mechanism; 2. Controller; 3. Actuator; 4. Collecting mechanism; 5. Walking mechanism; 1-1. Finger assembly; 1-2. Base; 1-3. Flange; 1-4. Electrically controlled gripper assembly; 1-5. Driving rod; 1-6. Loading bowl; 1-7. Micro servo motor; 1-8. First synchronous belt gear; 1-9. Fixture; 1-10. Synchronous belt; 1-11. Second synchronous belt gear; 1-12. Motor; 1-13. Driven rod; and 1-14. Depth camera. Detailed Implementation

[0042] In the following description, embodiments of the invention will be described with reference to the accompanying drawings. In the description below, the same modules are denoted by the same reference numerals. Where the same reference numerals are used, their names and functions are also the same. Therefore, their detailed description will not be repeated.

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.

[0044] Figure 1 The structure of an intelligent harvesting and collection system provided according to an embodiment of the present invention is shown.

[0045] like Figure 1 As shown, the intelligent harvesting and collection system proposed in this embodiment of the invention includes a collection mechanism 4, a controller 2, and an execution mechanism 3 and a clamping mechanism 1 that communicate with the controller 2. The execution mechanism 3 is used to adjust the position of the clamping mechanism 1 under the control of the controller 2, the clamping mechanism 1 is used to harvest crops under the control of the controller 2, and the collection device is used to collect the crops harvested by the clamping mechanism 1.

[0046] Controller 2 includes depth cameras 1-14, an image processing module, and a control module. Depth cameras 1-14 are used to acquire color and depth images of crops in the same scene. The image processing module includes an image acquisition unit, an image recognition unit, an image segmentation unit, an image fusion unit, a point cloud data calculation unit, a pose calculation unit, and a data conversion unit.

[0047] The image acquisition unit is used to receive and store color images and depth images;

[0048] The image recognition unit is used to identify crops in color images;

[0049] The image segmentation unit is used to segment the specific contours of the crop based on the recognition results of the image recognition unit to obtain the contours to be extracted;

[0050] The image fusion unit is used to register and fuse color images and depth images to obtain a registered depth image;

[0051] The point cloud data calculation unit is used to process the registered depth image according to the contour to be extracted, and calculate the point cloud dataset of the crop by combining the intrinsic and extrinsic parameters of depth cameras 1-14.

[0052] The pose calculation unit is used to fit the point cloud dataset to obtain the pose data of the crop in the 1-14 coordinate system of the depth camera.

[0053] The data conversion unit is used to convert the crop's pose data in the coordinate system of the depth camera 1-14 to the pose data in the coordinate system of the actuator 3;

[0054] The control module is used to control the actuator 3 and the gripping mechanism 1 based on the crop's pose data in the coordinate system of the actuator 3.

[0055] The intelligent harvesting and collection system proposed in this embodiment of the invention can be configured for use on the walking mechanism 5, such as... Figure 2 As shown, the walking mechanism 5 is assembled with the actuator 3 and the walking mechanism 5 is connected to the controller 2 for communication, which can realize the harvesting of crops in the area. Since the walking mechanism 5 is a mature technology and this invention only considers the harvesting and collection of crops, the walking mechanism 5 will not be described in detail below.

[0056] Combination Figures 2-3 It can be seen that the clamping mechanism 11 can adopt the following... Figure 2The conventional pneumatic gripper shown can also be replaced by the gripping mechanism 1 with a loading bowl 1-6 designed by this invention. The loading bowl 1-6 can receive and collect the items picked up by the electronically controlled gripper assembly 1-4. When the loading bowl 1-6 is full, it is then collected by the collection mechanism 4. Compared with the collection mechanism 4 collecting the items picked up by the electronically controlled gripper assembly 1-4 one by one, the gripping mechanism 1 with a loading bowl 1-6 proposed by this invention plays a buffering role. The two types of gripping mechanisms 1 will be described in detail below.

[0057] The gripping mechanism 1 includes an electrically controlled gripper assembly 1-4 for picking, an active rod 1-5, a driven rod 1-13, a container 1-6, a first synchronous belt gear 1-8, a second synchronous belt gear 1-11, a synchronous belt 1-10, a fixed frame 1-9 mounted on the electrically controlled gripper assembly 1-4, a motor 1-12 mounted on the fixed frame 1-9, a depth camera 1-14, and a micro servo motor 1-7;

[0058] One end of the active rod 1-5 is fixedly mounted on the output shaft of the motor 1-12, and the other end is hinged to the container 1-6. The active rod 1-5 is used to drive the container 1-6 to rotate around the output shaft of the motor 1-12.

[0059] The container bowl 1-6 is fixedly connected to the end near the driven rod 1-13 and the second synchronous belt gear 1-11. The container bowl 1-6 is used to hold the items picked up by the electronically controlled gripper assembly 1-4. The initial position of the container bowl 1-6 is located below the electronically controlled gripper assembly 1-4.

[0060] Synchronous belt 1-10 is tensioned between the first synchronous belt gear 1-8 and the second synchronous belt gear 1-11;

[0061] The first synchronous belt gear 1-8 is fixedly installed on the output shaft of the micro servo motor 1-7, and drives the container 1-6 to rotate through the synchronous belt 1-10 and the second synchronous belt gear 1-11.

[0062] One end of the driven rod 1-13 is hinged to the end face of the first synchronous belt gear 1-8, and the other end is hinged to the end face of the second synchronous belt gear 1-10. The driven rod 1-13 is used to support the load bowl 1-6 without interfering with the flipping of the load bowl 1-6.

[0063] The optical axis of the depth camera 1-14 is parallel to the central axis of the electronically controlled gripper assembly 1-4; the central axis of the output shaft of the motor 1-12, the central axis of the first synchronous belt gear 1-8, and the central axis of the micro servo motor 1-7 coincide; the lever arm of the active lever 1-5 when rotating and the lever arm of the driven lever 1-13 when rotating are parallel and perpendicular to the central axis of the output shaft of the motor 1-12.

[0064] The gripping mechanism 1 includes at least two sets of finger assemblies 1-1, a base 1-2, a pneumatic tube, and an air source. The base 1-2 is fixedly mounted on the actuator 3, and the finger assemblies 1-1 are mounted opposite to the base 1-2. The finger assemblies 1-1 are provided with an air inlet. One end of the pneumatic tube is connected to the finger assemblies 1-1 through the pneumatic inlet, and the other end of the pneumatic tube is connected to the air source.

[0065] The electrically controlled gripper assembly 1-4 is equipped with a flange 1-3 for fixing and connecting external components.

[0066] The connection between the cargo bowl 1-6 and the drive rod 1-5 is a straight rod structure, and the connection between the cargo bowl 1-6 and the second synchronous belt 1-10 gear is also a straight rod structure, with their central axes coinciding; the middle section of the cargo bowl 1-6 is a bowl-shaped structure.

[0067] The picking end of the electrically controlled gripper assembly 1-4 is equipped with a plastic pad for flexible picking.

[0068] The actuator 3 employs a harvesting robotic arm with three to seven degrees of freedom.

[0069] The following describes in detail the structure of the intelligent harvesting and collection system proposed in the embodiments of the present invention, and the flow of the control method of the intelligent harvesting and collection system.

[0070] Figure 4 The flowchart of the control method for the intelligent harvesting and collection system provided according to an embodiment of the present invention is shown.

[0071] like Figure 4 As shown in the figure, the control method of the intelligent harvesting and collection system proposed in this embodiment of the invention specifically includes the following steps:

[0072] S1: Build an intelligent harvesting and collection system.

[0073] S2: Use depth cameras 1-14 to acquire images of crops against the same background, obtaining depth and color images.

[0074] S3: Controller 2 processes the depth image and color image through the image processing module to obtain the crop's pose data in the coordinate system of actuator 3.

[0075] The controller 2 determines the position and orientation of the gripping mechanism 1 by receiving the current position and orientation feedback from the actuator 3 in real time.

[0076] S4: After adjusting the gripping mechanism 1 to the target pose based on the crop's pose data in the coordinate system of the actuator 3, the control module controls the gripping mechanism 1 to harvest the crop.

[0077] In step S4, the control module sends an opening or closing command to the gripping mechanism 1 to harvest the crop.

[0078] S5: The control module controls the actuator 3 to transport the crop picked by the clamping mechanism 1 to the collection mechanism 4, thus completing the picking and collection of the crop.

[0079] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0080] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An intelligent picking and collecting system, characterized in that, The system comprises a collecting mechanism, a controller, and an executing mechanism and a clamping mechanism in communication with the controller, wherein the executing mechanism is used to adjust the pose of the clamping mechanism under the control of the controller, the clamping mechanism is used to pick the crops under the control of the controller, and the collecting mechanism is used to collect the crops picked by the clamping mechanism; The controller comprises a depth camera, an image processing module, and a control module, wherein the depth camera is used to collect color images and depth images of the crops in the same scene; the image processing module comprises an image acquisition unit, an image recognition unit, an image segmentation unit, an image fusion unit, a point cloud data calculation unit, a pose calculation unit, and a data conversion unit, wherein the image acquisition unit is used to receive and store the color images and the depth images; the image recognition unit is used to recognize the crops in the color images; the image segmentation unit is used to segment the specific contour of the crops according to the recognition result of the image recognition unit to obtain a contour to be extracted; the image fusion unit is used to register and fuse the color images and the depth images to obtain registered depth images; the point cloud data calculation unit is used to process the registered depth images according to the contour to be extracted and calculate the point cloud dataset of the crops in combination with the intrinsic and extrinsic parameters of the depth camera; the pose calculation unit is used to fit the point cloud dataset to obtain the pose data of the crops in the depth camera coordinate system; the data conversion unit is used to convert the pose data of the crops in the depth camera coordinate system to pose data in the coordinate system of the executing mechanism; the control module is used to control the executing mechanism and the clamping mechanism according to the pose data of the crops in the coordinate system of the executing mechanism; the clamping mechanism comprises an electrically controlled clamping jaw assembly for picking, a driving rod, a driven rod, a carrier bowl, a first synchronous belt gear, a second synchronous belt gear, a synchronous belt, a fixing frame mounted on the electrically controlled clamping jaw assembly, a motor mounted on the fixing frame, a depth camera, and a micro servo; one end of the driving rod is fixedly mounted on the output shaft of the motor, and the other end is hingedly connected with the carrier bowl, and the driving rod is used to drive the carrier bowl to rotate around the output shaft of the motor; the carrier bowl is fixedly connected with the second synchronous belt gear near the end of the driven rod, and is used to accommodate the picked objects by the electrically controlled clamping jaw assembly; the initial position of the movement of the carrier bowl is located below the electrically controlled clamping jaw assembly; the synchronous belt is tensioned between the first synchronous belt gear and the second synchronous belt gear; the first synchronous belt gear is fixedly mounted on the output shaft of the micro servo, and the carrier bowl is turned over through the synchronous belt and the second synchronous belt gear; one end of the driven rod is hingedly connected with the end face of the first synchronous belt gear, and the other end is hingedly connected with the end face of the second synchronous belt gear, and the driven rod is used to carry the carrier bowl without interfering with the turning over of the carrier bowl. The optical axis of the depth camera is parallel to the central axis of the electrically controlled gripper assembly; the central axis of the output shaft of the motor, the central axis of the first synchronous belt gear and the central axis of the micro steering engine coincide; the force arm when the driving rod rotates and the force arm when the driven rod rotates are parallel and perpendicular to the central axis of the output shaft of the motor; A flange for fixedly connecting an external component is mounted on the electrically controlled gripper assembly.

2. The smart picking and collecting system according to claim 1, characterized in that, The connecting section of the object bowl and the driving rod is a straight rod structure, and the connecting section of the object bowl and the second synchronous belt gear is also a straight rod structure, and the central axes of the two coincide; the middle section of the object bowl is a bowl structure.

3. The smart picking and collecting system according to claim 2, wherein, The picking end of the electrically controlled gripper assembly is provided with a plastic gasket for flexible picking.

4. The smart picking and collecting system according to claim 1, wherein, The actuator adopts a picking mechanical arm with three to seven degrees of freedom.

5. A control method of an intelligent picking and collecting system, implemented by the intelligent picking and collecting system according to any one of claims 1-4, characterized in that, Specifically comprises the following steps: S1: building the intelligent picking and collecting system; S2: using the depth camera to collect images of crops in the same background to obtain depth images and color images; S3: the controller processes the depth images and the color images through the image processing module to obtain the pose data of the crops in the coordinate system of the actuator; S4: the control module adjusts the gripping mechanism to the target pose according to the pose data of the crops in the coordinate system of the actuator, and then controls the gripping mechanism to pick the crops; S5: the control module controls the actuator to transport the crops picked by the gripping mechanism to the collecting mechanism, thereby completing the picking and collecting of the crops. 6.The control method of the smart picking and collecting system according to claim 5, wherein, In the step S4, the control module sends the gripping mechanism an opening instruction or a closing instruction to realize picking of the crops. 7.The control method of the smart picking and collecting system according to claim 5, wherein, The controller determines the pose state of the gripping mechanism by receiving the current pose feedback of the actuator in real time.

Images