Control methods and systems for fruit picking robots; fruit picking robots

Abstract

This invention discloses a control method and system for a fruit-picking robot, and the fruit-picking robot itself. The method includes: the fruit-picking robot determining a picking task and moving to a target picking location according to the task; performing information collection and picking queue construction operations on all picking targets at the target picking location to obtain first picking information; performing multi-level information analysis and processing operations on all picking targets based on the first picking information to obtain second picking information; performing a third picking processing operation based on the second picking information; and determining whether the picking task at the target picking location has been completed. If so, updating the picking task and moving to the next picking location. The third picking processing operation includes at least one of information collection, information processing, picking force calculation, and fruit picking operation. Therefore, implementing this invention improves the accuracy of detecting fruit location, the accuracy of judging fruit picking, and the precision and reliability of fruit picking.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a control method and device for a fruit-picking robot. Background Technology

[0002] Fruit-picking robots are a new type of robotic equipment developed to replace manual fruit harvesting. These robots typically employ mechatronics technology, automatically identifying fruit locations and using robotic arms, grippers, and bladed tools to pick the fruit. Furthermore, they possess automatic navigation, obstacle avoidance, and positioning capabilities to aid their movement within orchards, and can be intelligently controlled and managed through machine vision and artificial intelligence technologies. Fruit-picking robots not only improve the quality of fruit harvesting but also reduce labor costs and intensity, alleviating the problem of agricultural labor shortages.

[0003] However, achieving a fully autonomous fruit-picking robot still requires overcoming some technical challenges, including: how to improve the accuracy of detecting fruit location when the fruit-picking robot is performing fruit-picking operations, how to improve the accuracy of judging fruit picking (accurately picking ripe fruit), how to determine the appropriate gripping force, and the picking method, etc. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a control method and device for a fruit picking robot, which can improve the accuracy of detecting the location of fruit, improve the accuracy of judging fruit picking, reduce the situation of picking unripe and / or poor quality fruit, thereby improving the accuracy and reliability of fruit picking.

[0005] To address the aforementioned technical problems, the first aspect of this invention discloses a method for controlling a fruit-picking robot, the method comprising:

[0006] The fruit picking robot determines the current picking task to be executed, the picking task includes multiple picking sites and the location information corresponding to each picking site; and moves to the target picking site according to the location information of the target picking site, all of which include the target picking site;

[0007] The fruit picking robot performs a first picking processing operation matching the first picking item on all picking targets corresponding to the target picking site, and obtains the first picking information corresponding to all picking targets. The first picking item is the execution item for the target picking site in the picking task. The first picking processing operation includes information collection operation and picking queue construction operation.

[0008] The fruit picking robot performs a second picking process operation matching the second picking task on all the picking targets based on the first picking information, thereby obtaining second picking information corresponding to all the picking targets; and performs a third picking process operation on all the picking targets based on the second picking information.

[0009] The fruit picking robot determines whether the picking task for the target picking site has been completed. When it determines that the picking task for the target picking site has been completed, it updates the completion progress of the picking task and moves to the next picking site corresponding to the target picking site to perform the picking task for the next picking site.

[0010] The second picking process includes multi-level information analysis and processing; the third picking process includes at least one of information collection, information processing, picking force calculation, and fruit picking.

[0011] As an optional implementation, in the first aspect of the present invention, the fruit picking robot performs a first picking processing operation matching the target picking task on all picking targets corresponding to the target picking site, to obtain first picking information corresponding to all the picking targets, including:

[0012] The fruit picking robot generates information collection instructions for all picking targets corresponding to the target picking site, and collects first images corresponding to all picking targets based on the information collection instructions and the first sensor configured on the fruit picking robot.

[0013] The fruit picking robot analyzes all the first acquired images to obtain the fruit ripening information of each picking target; at the same time, it constructs a target picking queue corresponding to all the picking targets based on all the first acquired images, and the target picking queue includes the picking order and picking position corresponding to each picking target.

[0014] The fruit picking robot determines the fruit ripeness information and the target picking queue as the first picking information corresponding to all the picking targets.

[0015] As an optional implementation, in a first aspect of the present invention, the fruit-picking robot performs a second picking processing operation matching the second picking task on all the picking targets based on the first picking information, to obtain second picking information corresponding to all the picking targets, including:

[0016] The fruit picking robot performs a first analysis and processing on the fruit ripening information to obtain first ripening information corresponding to the fruit ripening information. The first ripening information includes a second acquired image, and the ripening degree of the fruit corresponding to the picking target in the second acquired image is greater than the target ripening threshold.

[0017] The fruit-picking robot performs a second analysis on the first maturity information according to preset maturity analysis parameters for the picking target, to obtain second maturity information corresponding to the first maturity information, and updates the target picking queue according to the second maturity information, the position information of the end effector, and preset calibration parameters. The end effector is configured on the fruit-picking robot for performing fruit picking. The calibration parameters include sensor parameters corresponding to the first sensor and picking type parameters corresponding to the picking target. The first sensor is externally mounted on the fruit-picking robot.

[0018] The fruit picking robot determines the second ripeness information and the updated target picking queue as the second picking information corresponding to all the picking targets.

[0019] As an optional implementation, in a first aspect of the invention, the fruit-picking robot performs a third picking process on all the picking targets based on the second picking information, including:

[0020] The fruit picking robot generates a movement path for the end effector based on the target picking queue and the actuator coordinates of the end effector. The movement path includes the picking coordinates of each picking target.

[0021] The fruit picking robot controls the end effector to move according to the queue order of the target picking queue and the picking coordinates of each target picking according to the movement path;

[0022] For any of the picking targets, when the end effector is determined to have moved to the picking coordinates of the picking target, the fruit picking robot controls the end effector to collect the third picking information corresponding to the picking target, and controls the end effector to perform the third picking processing operation corresponding to the third picking information on the picking target;

[0023] For any of the picking targets, after determining that the third processing operation for the picking target has been completed, the fruit picking robot updates the target picking queue and determines whether the completion progress of the target picking queue has reached a preset progress value. When it is determined that the completion progress of the target picking queue has reached the preset progress value, the determination of whether the picking task for the target picking site has been completed is triggered.

[0024] When it is determined that the completion progress of the target picking queue has not reached the preset progress value, the fruit picking robot controls the end effector to move to the next picking target corresponding to the picking target, so as to perform the third picking processing operation on the next picking target.

[0025] As an optional implementation, in a first aspect of the invention, the fruit-picking robot controls the end effector to collect third picking information corresponding to the picking target, including:

[0026] The fruit picking robot controls the gripper corresponding to the end effector to contact the picking target; and controls the second sensor to collect the contact information of the picking target. The contact information includes embedded feature point offset information and spectral analysis information of the picking target. The embedded feature point offset information is the information corresponding to the force offset of the second sensor embedded in the gripper after the gripper contacts the picking target.

[0027] The fruit-picking robot calculates the actual spatial coordinates corresponding to the contact information based on preset multi-camera parameters; and calculates the offset between the actual spatial coordinates and the preset standard spatial coordinates.

[0028] The fruit-picking robot converts the offset into a target three-dimensional force, which is the force corresponding to the gripper gripping the picking target.

[0029] The fruit-picking robot determines the target's three-dimensional force and the spectral analysis information as the third picking information corresponding to the picking target.

[0030] As an optional implementation, in a first aspect of the present invention, the spectral analysis information is used to determine whether the harvested target meets the conditions of no damage and maturity; the information type of the spectral analysis information includes a first type, a second type, or a third type, wherein the first type indicates that the harvested target simultaneously meets the conditions of no damage and maturity; the second type indicates that the harvested target does not meet the conditions of no damage but meets the conditions of maturity; and the third type indicates that the harvested target does not simultaneously meet the conditions of no damage and maturity.

[0031] The fruit-picking robot controls the end effector to perform a third picking processing operation on the picking target corresponding to the third picking information, including:

[0032] When the spectral analysis information is of the first type or the second type, the fruit picking robot controls the gripper of the end effector to grip the picking target based on the target three-dimensional force, and places the gripped picking target into the placement container that matches the information type of the spectral analysis information;

[0033] When the spectral analysis information is of the third type, the fruit picking robot generates a discard flag based on the third picking information of the picking target. The discard flag is used to indicate that the information type of the spectral analysis information corresponding to the picking target is the third type.

[0034] As an optional implementation, in the first aspect of the present invention, the picking type parameter corresponding to the picking target is used to determine the picking execution method for the picking target, the picking execution method including a first execution method or a second execution method, wherein:

[0035] When the picking execution mode is the first execution mode, the fruit picking robot activates the gripper, the first sensor, and the second sensor corresponding to the end effector when picking any of the picking targets.

[0036] When the picking execution mode is the second execution mode, when the fruit picking robot picks any of the picking targets, it activates the gripper, the first sensor, the second sensor, and the moving shearing tool corresponding to the end effector. The moving shearing tool is externally mounted on the end effector to cut off the target fruit stem area corresponding to the picking target.

[0037] A second aspect of this invention discloses a control system for a fruit-picking robot, the system being applied to the fruit-picking robot, the system comprising:

[0038] The determination module is used to determine the picking task to be executed, wherein the picking task includes multiple picking sites and location information corresponding to each picking site;

[0039] A movement control module is used to move to the target picking site according to the location information of the target picking site, and all the picking sites include the target picking site;

[0040] The first picking module is used to perform a first picking processing operation matching the first picking item on all picking targets corresponding to the target picking site, and obtain the first picking information corresponding to all the picking targets. The first picking item is the execution item for the target picking site in the picking task. The first picking processing operation includes information collection operation and picking queue construction operation.

[0041] The second picking module is used to perform a second picking processing operation matching the second picking matter on all the picking targets according to the first picking information, so as to obtain the second picking information corresponding to all the picking targets.

[0042] The third picking module is used to perform a third picking processing operation on all the picking targets based on the second picking information.

[0043] The judgment module is used to determine whether the picking task for the target picking site has been completed;

[0044] The update module is used to update the completion progress of the picking task and move to the next picking site corresponding to the target picking site after the judgment module determines that the picking task for the target picking site has been completed, so as to perform the picking task for the next picking site.

[0045] The second picking process includes multi-level information analysis and processing; the third picking process includes at least one of information collection, information processing, picking force calculation, and fruit picking.

[0046] As an optional implementation, in a second aspect of the present invention, the first picking module performs a first picking processing operation matching the target picking item on all picking targets corresponding to the target picking site, and obtains the first picking information corresponding to all the picking targets in a specific manner including:

[0047] Generate information collection instructions for all picking targets corresponding to the target picking site, and collect first images corresponding to all picking targets according to the information collection instructions and the first sensor configured on it;

[0048] Analyze all the first acquired images to obtain fruit ripening information for each of the harvesting targets; at the same time, construct a target harvesting queue corresponding to all the harvesting targets based on all the first acquired images, the target harvesting queue including the harvesting order and harvesting position corresponding to each harvesting target;

[0049] The fruit ripening information and the target picking queue are determined as the first picking information corresponding to all the picking targets.

[0050] As an optional implementation, in a second aspect of the present invention, the second picking module performs a second picking processing operation matching the second picking matter on all the picking targets based on the first picking information, and obtains the second picking information corresponding to all the picking targets in a specific manner including:

[0051] Perform a first analysis process on the fruit ripening information to obtain first ripening information corresponding to the fruit ripening information. The first ripening information includes a second acquired image, and the ripening degree of the fruit corresponding to the picking target in the second acquired image is greater than the target ripening threshold.

[0052] Based on preset maturity analysis parameters for the harvesting target, the first maturity information is subjected to a second analysis process to obtain second maturity information corresponding to the first maturity information. The target harvesting queue is then updated based on the second maturity information, the position information of the end effector, and preset calibration parameters. The end effector is configured on the device for performing fruit harvesting. The calibration parameters include sensor parameters corresponding to the first sensor and harvesting type parameters corresponding to the harvesting target. The first sensor is externally mounted on the device.

[0053] The second maturity information and the updated target picking queue are determined as the second picking information corresponding to all the picking targets.

[0054] As an optional implementation, in a second aspect of the present invention, the third picking module performs a third picking processing operation on all the picking targets based on the second picking information, specifically including:

[0055] Based on the target picking queue and the actuator coordinates of the end effector, a movement path is generated for the end effector, the movement path including the picking coordinates of each picking target;

[0056] The end effector is controlled to move according to the queue order of the target picking queue and the picking coordinates of each target picking according to the movement path;

[0057] For any of the harvesting targets, when the end effector is determined to have moved to the harvesting coordinates of the harvesting target, the end effector is controlled to collect the third harvesting information corresponding to the harvesting target, and the end effector is controlled to perform a third harvesting processing operation on the harvesting target corresponding to the third harvesting information;

[0058] For any of the picking targets, after determining that the third processing operation for the picking target has been completed, the target picking queue is updated, and it is determined whether the completion progress of the target picking queue has reached a preset progress value. When it is determined that the completion progress of the target picking queue has reached the preset progress value, the determination of whether the picking task for the target picking site has been completed is triggered.

[0059] When it is determined that the completion progress of the target picking queue has not reached the preset progress value, the end effector is controlled to move to the next picking target corresponding to the picking target, so as to perform the third picking processing operation on the next picking target.

[0060] As an optional implementation, in a second aspect of the present invention, the method by which the third picking module controls the end effector to collect the third picking information corresponding to the picking target specifically includes:

[0061] The end effector is controlled to contact the grasper corresponding to the picking target; and the second sensor is controlled to collect the contact information of the picking target. The contact information includes embedded feature point offset information and spectral analysis information of the picking target. The embedded feature point offset information is the information corresponding to the force offset of the second sensor embedded in the grasper after the grasper contacts the picking target.

[0062] Calculate the actual spatial coordinates corresponding to the contact information based on preset multi-camera parameters; and calculate the offset between the actual spatial coordinates and the preset standard spatial coordinates.

[0063] The offset is converted into a target three-dimensional force, which is the force corresponding to the gripper gripping the harvested target.

[0064] The target's three-dimensional force and the spectral analysis information are determined as the third harvesting information corresponding to the harvesting target.

[0065] As an optional implementation, in a second aspect of the present invention, the spectral analysis information is used to determine whether the harvested target meets the conditions of no damage and maturity; the information type of the spectral analysis information includes a first type, a second type, or a third type, wherein the first type indicates that the harvested target simultaneously meets the conditions of no damage and maturity; the second type indicates that the harvested target does not meet the conditions of no damage but meets the conditions of maturity; and the third type indicates that the harvested target does not simultaneously meet the conditions of no damage and maturity.

[0066] The specific methods by which the third harvesting module controls the end effector to perform a third harvesting processing operation on the harvesting target corresponding to the third harvesting information include:

[0067] When the spectral analysis information is of the first type or the second type, the gripper of the end effector is controlled to grip the target based on the target three-dimensional force, and the gripped target is placed in the placement container that matches the information type of the spectral analysis information;

[0068] When the spectral analysis information is of the third type, a discard identifier is generated based on the third harvesting information of the harvesting target. The discard identifier is used to indicate that the information type of the spectral analysis information corresponding to the harvesting target is the third type.

[0069] As an optional implementation, in a second aspect of the invention, the harvesting type parameter corresponding to the harvesting target is used to determine the harvesting execution method for the harvesting target, the harvesting execution method including a first execution method or a second execution method, wherein:

[0070] When the picking execution mode is the first execution mode, the gripper and the second sensor corresponding to the end effector are activated when picking any of the picking targets;

[0071] When the picking execution mode is the second execution mode, when picking any of the picking targets, the gripper, the second sensor and the moving cutting tool corresponding to the end effector are activated. The moving cutting tool is externally mounted on the end effector to cut off the target fruit stem area corresponding to the picking target.

[0072] The third aspect of this invention discloses a fruit-picking robot, which includes an end effector, a first sensor, a second sensor, and a moving shearing tool; the fruit-picking robot is used to execute the control method for the fruit-picking robot disclosed in the first aspect of this invention.

[0073] A fourth aspect of the present invention discloses another control device for a fruit-picking robot, the device comprising:

[0074] Memory containing executable program code;

[0075] A processor coupled to the memory;

[0076] The processor calls the executable program code stored in the memory to execute the control method for the fruit picking robot disclosed in the first aspect of the present invention.

[0077] The fifth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute the control method for the fruit picking robot disclosed in the first aspect of the present invention.

[0078] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0079] In this embodiment of the invention, a method for controlling a fruit-picking robot is provided. This method is applied to the fruit-picking robot and includes:

[0080] The fruit-picking robot determines the current picking task to be executed, which includes multiple picking sites and the location information corresponding to each picking site. It then moves to the target picking site based on the picking task, where all picking sites include the target picking site. The fruit-picking robot performs a first picking processing operation matching the first picking task on all picking targets corresponding to the target picking site, obtaining first picking information corresponding to all picking targets. The first picking task is the execution item for the target picking site within the picking task. The first picking processing operation includes information collection and picking queue construction. Based on the first picking information, the fruit-picking robot performs the first picking processing operation matching the first picking task. The second picking process involves matching two picking tasks to obtain second picking information corresponding to all picking targets. Based on this second picking information, a third picking process is performed on all picking targets. The fruit picking robot determines whether the picking task for the target picking location has been completed. If it determines that the picking task for the target picking location has been completed, it updates the completion progress of the picking task and moves to the next picking location corresponding to the target picking location to perform the picking task for the next picking location. The second picking process includes multi-level information analysis and processing. The third picking process includes at least one of the following: information collection, information processing, picking force calculation, and fruit picking operation. As can be seen, implementing this invention enables automatic movement to the corresponding target picking site based on the determined picking task, and then automatically collecting image information of the target picking site while constructing a picking queue. This image information is used to initially detect the area where all picking targets (such as fruits) are located at the target picking site, and the picking queue is used to determine the picking order for each picking target. The picking range and picking execution order are initially determined, improving picking efficiency. Then, a second picking processing operation is performed on the first picking information, including image information and picking queue, namely, a multi-level information analysis operation, further limiting the area corresponding to the image information and adjusting and updating the picking queue. This improves the accuracy of relevant image information and the accuracy of queue construction. Finally, a third picking processing operation is performed on all picking targets based on the obtained second picking information, that is, the final picking operation is performed for each picking target. During this process, the machine force (picking force) used for picking is calculated, and further analysis of the information corresponding to the fruit contacted in real time is performed. Through the multi-layer algorithm of the first, second, and third picking processing operations, the picking efficiency, picking accuracy, and reliability of fruit picking at the target picking site are improved. Attached Figure Description

[0081] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0082] Figure 1 This is a flowchart illustrating a method for controlling a fruit-picking robot disclosed in an embodiment of the present invention.

[0083] Figure 2 This is a flowchart illustrating another method for controlling a fruit-picking robot disclosed in an embodiment of the present invention;

[0084] Figure 3 This is a schematic diagram of the control system for a fruit picking robot disclosed in an embodiment of the present invention;

[0085] Figure 4 This is a schematic diagram of the structure of a control device for a fruit picking robot disclosed in an embodiment of the present invention;

[0086] Figure 5 This is a schematic diagram of the structure of a control device for another fruit-picking robot disclosed in an embodiment of the present invention;

[0087] Figure 6 This is a flowchart illustrating another method for controlling a fruit-picking robot disclosed in an embodiment of the present invention;

[0088] Figure 7 This is a flowchart illustrating another method for controlling a fruit-picking robot disclosed in an embodiment of the present invention;

[0089] Figure 8 This is a flowchart illustrating another method for controlling a fruit-picking robot disclosed in an embodiment of the present invention;

[0090] Figure 9 This is a schematic diagram showing the local spectral information loss caused by the embedded mark of the flexible transparent film of the second sensor disclosed in the embodiment of the present invention;

[0091] Figure 10 This is a schematic diagram of a fruit harvesting method involving cutting off the fruit stem, as disclosed in an embodiment of the present invention. Detailed Implementation

[0092] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0093] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0094] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0095] This invention discloses a control method and device for a fruit-picking robot. Based on a determined picking task, the robot automatically moves to the corresponding target picking location, automatically collects image information of the target picking location, and simultaneously constructs a picking queue. This image information is used to initially detect the area where all picking targets (such as fruits) are located at the target picking location, and the picking queue is used to determine the picking order for each picking target. This initially determines the picking range and picking execution order, improving picking efficiency. Subsequently, a second picking processing operation, namely multi-level information analysis, is performed on the first picking information, including the image information and the picking queue. The process further restricts the area corresponding to the image information and adjusts and updates the picking queue; this improves the accuracy of relevant image information and queue construction. Finally, based on the obtained second picking information, a third picking processing operation is performed on all picking targets, that is, the final picking operation is performed on each picking target. This includes the calculation of the machine force (picking force) used for picking, and further analysis of the real-time picking contact information with the fruit. Through the multi-layered algorithm of the first, second, and third picking processing operations, the picking efficiency, picking accuracy, and reliability of fruit picking at the target picking site are improved. These are explained in detail below.

[0096] Example 1

[0097] Please see Figure 1 , Figure 1 This is a flowchart illustrating a control method for a fruit-picking robot disclosed in an embodiment of the present invention. Figure 1 The described control method for fruit-picking robots can be applied to fruit-picking robots and their control systems; however, this invention does not limit its application. Figure 1As shown, the control method for this fruit-picking robot may include the following operations:

[0098] 101. The fruit picking robot determines the picking task to be performed, which includes multiple picking sites and the location information corresponding to each picking site; and moves to the target picking site according to the location information of the target picking site.

[0099] In this embodiment of the invention, all picking sites include target picking sites.

[0100] In this embodiment of the invention, optionally, the picking task also includes regional information of the target area where all picking sites are located, including information on the distribution of rugged terrain points and information on crop density;

[0101] Among them, the information on the distribution of rugged terrain points includes information on steep slopes and ground obstacles in the target area; the information on crop density includes the height and density of vegetation in the target area.

[0102] In this embodiment of the invention, step 101, in which the fruit-picking robot moves to the target picking location based on the location information of the target picking location, specifically includes:

[0103] The fruit picking robot generates corresponding path movement information based on the location and area information of each picking site. The path movement information includes the picking order corresponding to each picking site.

[0104] The fruit-picking robot moves to the target picking location according to the path information;

[0105] The movement route of the fruit picking robot to the target picking site is generated based on the current position of the fruit picking robot, the location information of the target picking site, and the regional information.

[0106] Optionally, during actual movement, the fruit-picking robot can also use its configured obstacle avoidance module to replan its path in response to sudden obstacles and adjust its movement path in real time to achieve intelligent obstacle avoidance.

[0107] As can be seen, in this embodiment of the invention, after receiving the picking task, the robot can automatically combine the regional information of the target area to automatically generate path movement information corresponding to the picking task. At the same time, the fruit picking robot can also achieve intelligent obstacle avoidance when moving, which improves the efficiency of the fruit picking robot in performing picking movement.

[0108] 102. The fruit picking robot performs the first picking processing operation matching the first picking task on all picking targets corresponding to the target picking site, and obtains the first picking information corresponding to all picking targets.

[0109] In this embodiment of the invention, the first picking item is the execution item for the target picking site in the picking task, and the first picking processing operation includes information collection operation and picking queue construction operation.

[0110] 103. The fruit picking robot performs a second picking process on all picking targets based on the first picking information, matching the second picking task, and obtains the second picking information corresponding to all picking targets.

[0111] In this embodiment of the invention, the second harvesting processing operation includes a multi-level information analysis and processing operation.

[0112] 104. The fruit picking robot performs the third picking process on all picking targets based on the second picking information.

[0113] In this embodiment of the invention, the third picking process includes at least one of information collection, information processing, picking force calculation, and fruit picking operation;

[0114] 105. The fruit picking robot determines whether the picking task for the target picking site has been completed. When it determines that the picking task for the target picking site has been completed, it updates the completion progress of the picking task and moves to the next picking site corresponding to the target picking site to perform the picking task for the next picking site.

[0115] It is evident that implementation Figure 1 The described control method for the fruit-picking robot can automatically move to the corresponding target picking site based on a determined picking task. It then automatically collects image information of the target picking site and simultaneously constructs a picking queue. This image information is used to initially detect the area where all picking targets (such as fruits) are located at the target picking site, and the picking queue is used to determine the picking order for each picking target. This initially determines the picking range and picking execution order, improving picking efficiency. Subsequently, a second picking processing operation, namely a multi-level information analysis operation, is performed on the first picking information, including the image information and the picking queue. Further limiting the area corresponding to the image information and adjusting and updating the picking queue; improving the accuracy of relevant image information and queue construction; finally, based on the obtained second picking information, a third picking processing operation is performed on all picking targets, that is, the final picking operation is performed on each picking target, including the calculation of the machine force (picking force) used for picking, further analysis of the real-time picking contact information of the fruit, etc. Through the multi-layer algorithm of the first, second and third picking processing operations, the picking efficiency, picking accuracy and reliability of fruit picking at the target picking site are improved.

[0116] Example 2

[0117] Please see Figure 2 , Figure 2This is a flowchart illustrating another method for controlling a fruit-picking robot disclosed in an embodiment of the present invention. Figure 2 The described control method for fruit-picking robots can be applied to control devices for fruit-picking robots, and this invention is not limited to any particular embodiment. Figure 2 As shown, the control method for this fruit-picking robot may include the following operations:

[0118] 201. The fruit picking robot determines the picking task to be performed, which includes multiple picking sites and the location information corresponding to each picking site; and moves to the target picking site according to the location information of the target picking site.

[0119] 202. The fruit picking robot generates information collection instructions for all picking targets corresponding to the target picking site, and collects the first acquisition images corresponding to all picking targets based on the information collection instructions and the first sensor configured on the fruit picking robot.

[0120] In this embodiment of the invention, the first sensor can be an external multispectral-3D sensor. The multispectral-3D multimode sensor consists of N spectral channels (N not less than 4), and each spectral channel's camera only acquires images within a specific wavelength range. The operating spectral range of the multispectral camera includes from near-ultraviolet (UVA) to near-infrared (NIR). 3D sensing can be achieved through binocular or multi-view matching of images from multiple cameras with different spectral wavelength ranges. The light source can be selectively turned on to ensure that the imaging target has reasonable spectral illumination. After acquiring the first acquired images corresponding to all the targets, automatic spectral image alignment correction can be performed.

[0121] 203. The fruit picking robot analyzes all the first acquired images to obtain the fruit ripening information of each picking target; at the same time, it constructs a target picking queue corresponding to all picking targets based on all the first acquired images. The target picking queue includes the picking order and picking position corresponding to each picking target.

[0122] 204. The fruit picking robot determines the fruit ripeness information and the target picking queue as the first picking information corresponding to all picking targets.

[0123] 205. The fruit picking robot performs a second picking process on all picking targets based on the first picking information, matching the second picking task, and obtains the second picking information corresponding to all picking targets.

[0124] 206. The fruit picking robot performs the third picking process on all picking targets based on the second picking information.

[0125] 207. The fruit picking robot determines whether the picking task for the target picking site has been completed. When it determines that the picking task for the target picking site has been completed, it updates the completion progress of the picking task and moves to the next picking site corresponding to the target picking site to perform the picking task for the next picking site.

[0126] For further descriptions of steps 201 and 205-207 in this embodiment of the invention, please refer to the other specific descriptions of steps 101 and 103-105 in Embodiment 1. These descriptions will not be repeated in this embodiment of the invention.

[0127] It is evident that implementation Figure 2 The described control method for the fruit-picking robot can automatically collect image information of the target picking site based on a first sensor after moving to the target picking site, and simultaneously construct a picking queue. This image information is used to initially detect the area where all picking targets (such as fruits) are located at the target picking site, and the picking queue is used to determine the picking order for each picking target. This achieves the initial determination of the picking range and picking execution order, improves picking efficiency, and facilitates subsequent processes to adjust and optimize the initially determined picking range and picking queue, thereby improving the execution efficiency of subsequent processes.

[0128] In an optional embodiment, step 205, where the fruit-picking robot performs a second picking process operation matching the second picking task on all picking targets based on the first picking information, specifically includes the following methods to obtain the second picking information corresponding to all picking targets:

[0129] The fruit picking robot performs a first analysis and processing on the fruit ripening information to obtain the first ripening information corresponding to the fruit ripening information. The first ripening information includes a second acquired image, and the fruit ripening degree corresponding to the picking target in the second acquired image is greater than the target ripening threshold.

[0130] The fruit-picking robot performs a second analysis on the first ripeness information based on preset ripeness analysis parameters for the picking target, obtaining the second ripeness information corresponding to the first ripeness information. It then updates the target picking queue based on the second ripeness information, the position information of the end effector, and preset calibration parameters. The end effector is configured on the fruit-picking robot to perform fruit picking. The calibration parameters include sensor parameters corresponding to the first sensor and picking type parameters corresponding to the picking target. The first sensor is externally mounted on the fruit-picking robot.

[0131] The fruit-picking robot identifies the second ripeness information and the updated target picking queue as the second picking information corresponding to all picking targets.

[0132] In this optional embodiment, the specific implementation process for steps 202-205 can be referred to Figure 6 , Figure 6 This is a flowchart illustrating another method for controlling a fruit-picking robot disclosed in an embodiment of the present invention. The method involves: firstly, detecting the picking targets (fruits) at the target picking sites through a first picking processing operation to obtain the ROI0 regions where all picking targets are located, and constructing a corresponding initial picking queue; then, in steps 202-205, a preliminary determination of fruit maturity is made in the ROI0 regions, extracting the ROI1 regions (first maturity information) that have reached the maturity threshold M1 (target maturity threshold). Next, region expansion and 3D reconstruction are performed on ROI1, and a second extraction and screening is conducted using the estimated fruit size and shape information (maturity analysis parameters) in the ROI regions, along with the maturity threshold M2, to obtain second maturity information; simultaneously, the target picking queue is adjusted and updated using multispectral-3D sensor calibration and hand-eye calibration parameters (calibration parameters).

[0133] As can be seen, in this optional embodiment, a second picking processing operation, namely a multi-level information analysis operation, is further performed on the first picking information, including image information and picking queue, to further limit the area corresponding to the image information and adjust and update the picking queue; thus improving the accuracy of the relevant image information and the accuracy of queue construction.

[0134] In another optional embodiment, step 206, in which the fruit-picking robot performs the third picking processing operation on all picking targets based on the second picking information, specifically includes:

[0135] The fruit picking robot generates a movement path for the end effector based on the target picking queue and the actuator coordinates of the end effector. The movement path includes the picking coordinates of each picking target.

[0136] The fruit picking robot controls the end effector to move according to the queue order of the target picking queue and the picking coordinates of each target, based on the movement path.

[0137] For any picking target, when the end effector moves to the picking coordinates of the picking target, the fruit picking robot controls the end effector to collect the third picking information corresponding to the picking target, and controls the end effector to perform the third picking processing operation corresponding to the third picking information on the picking target.

[0138] For any picking target, after determining that the third processing operation for the picking target has been completed, the fruit picking robot updates the target picking queue and determines whether the completion progress of the target picking queue has reached the preset progress value. When it is determined that the completion progress of the target picking queue has reached the preset progress value, the robot triggers the judgment of whether the picking task for the target picking site has been completed.

[0139] When it is determined that the completion progress of the target picking queue has not reached the preset progress value, the fruit picking robot controls the end effector to move to the next picking target corresponding to the picking target, so as to perform the third picking processing operation on the next picking target.

[0140] In this optional embodiment, the specific implementation process can be referred to Figure 7 After updating the target picking queue, the robot arm is driven to move and pick the fruit based on the coordinates of any picking target (fruit) and the coordinates of the robotic arm.

[0141] In this optional embodiment, the method by which the fruit-picking robot's end effector collects the third picking information corresponding to the picking target specifically includes:

[0142] The fruit picking robot controls the end effector to contact the picking target with the gripper; and controls the second sensor to collect the contact information of the picking target. The contact information includes embedded feature point offset information and spectral analysis information of the picking target. The embedded feature point offset information is the information corresponding to the force offset of the second sensor embedded in the gripper after the gripper contacts the picking target.

[0143] The fruit-picking robot calculates the actual spatial coordinates corresponding to the contact information based on preset multi-camera parameters; and calculates the offset between the actual spatial coordinates and the preset standard spatial coordinates.

[0144] The fruit-picking robot converts the offset into a target three-dimensional force, which is the force corresponding to the gripper gripping the picking target.

[0145] The fruit-picking robot uses the target's three-dimensional force and spectral analysis information to determine the third picking information corresponding to the picking target.

[0146] In this optional embodiment, it should be noted that the second sensor can be a multispectral-tactile multimode sensor composed of N spectral channels (N not less than 4), with each spectral channel's camera only acquiring images within a specific wavelength range. The operating spectral range of the multispectral camera includes from near-ultraviolet (UVA) to near-infrared (NIR). The light source can be selectively turned on to ensure that the imaging target has reasonable spectral illumination. Tactile sensing is achieved by adding a transparent flexible film containing embedded markers in front of the multispectral camera. When no object touches the flexible film, the multispectral camera operates normally; when an object touches the flexible film, the latter deforms under force, causing the embedded marker points to shift. By calculating the marker point shift, the three-dimensional stress situation of the flexible film can be obtained. The transparent flexible film is supported by a transparent glass or plastic substrate to limit its maximum deformation under force.

[0147] In this optional embodiment, the flexible transparent film uses low-cost, common materials as embedded markers (such as black plastic microspheres). Thus, in a multispectral image where the fruit is not in contact with the flexible film, the embedded markers are superimposed on the image of the target object (fruit), resulting in a loss of local spectral information in the image. Figure 9 Since the built-in multispectral sensor of the harvesting actuator is mainly used to confirm the ripeness of fruit and detect whether the fruit has obvious damage, this invention employs a classification method that is insensitive to the local information loss in the image caused by the aforementioned embedded markers when detecting fruit ripeness and damage. For example, it does not use the information-deficient areas. It should be noted that the size of the embedded markers is very small relative to the distance between them, so the proportion of the information-deficient areas in the entire spectral image is very low, and therefore the impact of not using the information-deficient areas is limited. For example, if the spacing between the embedded markers is 1 mm and the marker diameter is 0.1 mm, the area of ​​the information-deficient area in a single channel image is approximately 3.14%.

[0148] In this optional embodiment, spectral analysis information is used to determine whether the target to be harvested meets the conditions of no damage and maturity. The information type of the spectral analysis information includes a first type, a second type, or a third type. The first type indicates that the target to be harvested meets both the conditions of no damage and maturity. The second type indicates that the target to be harvested does not meet the conditions of no damage but meets the conditions of maturity. The third type indicates that the target to be harvested does not meet both the conditions of no damage and maturity.

[0149] The specific methods by which the aforementioned fruit-picking robot's end effector performs the third picking processing operation corresponding to the third picking information on the picking target include:

[0150] When the spectral analysis information is of type one or type two, the gripper of the fruit picking robot's end effector grips the target based on the target's three-dimensional force and places the gripped target into the placement container that matches the information type of the spectral analysis information.

[0151] When the spectral analysis information is of type three, the fruit picking robot generates a discard flag based on the third picking information of the picking target. The discard flag is used to indicate that the information type of the spectral analysis information corresponding to the picking target is type three.

[0152] As can be seen, in this optional embodiment, a third picking process can be performed on all picking targets based on the obtained second picking information, that is, the final picking operation is performed on each picking target. During this process, the machine force (picking force) used for picking is calculated, and further analysis of the information corresponding to the fruit contacted in real time is performed, which improves the picking efficiency, picking accuracy and reliability of picking fruit at the target picking site.

[0153] In another optional embodiment, the harvesting type parameter corresponding to the harvesting target is used to determine the harvesting execution method for the harvesting target. The harvesting execution method includes a first execution method or a second execution method, wherein:

[0154] When the picking execution mode is the first execution mode, the fruit picking robot will activate the gripper, the first sensor and the second sensor corresponding to the end effector when picking any picking target;

[0155] When the picking execution mode is the second execution mode, the fruit picking robot activates the gripper, the first sensor, the second sensor, and the moving shearing tool corresponding to the end effector when picking any target. The moving shearing tool is externally mounted on the end effector and is used to cut off the target fruit stem area corresponding to the picking target.

[0156] In this optional embodiment, the picking actuator fingers (grippers) can effectively pick most fruits. However, for a small number of fruits, such as grapes and strawberries, simply using the picking actuator fingers (grippers) in combination with pulling or torque may damage the fruit or fail to effectively separate the fruit stem. In the latter case, after the picking actuator grasps the fruit, the movable scissors on the actuator are used to position the fruit stem at a suitable location. Figure 10 Cut the stem at the position marked with an "X". For greater flexibility in cutting fruit stems, the movable shears can be mounted on a separate robotic arm; however, this flexibility comes at the cost of higher system costs.

[0157] As can be seen, in this optional embodiment, the picking method can be matched according to the type of the picking target, which helps to reduce the situation of picking unripe and / or poor quality fruits, thereby improving the accuracy and reliability of fruit picking.

[0158] Example 3

[0159] Please see Figure 3 , Figure 3 This is a schematic diagram of a control system for a fruit-picking robot disclosed in an embodiment of the present invention. This control system can be applied to fruit-picking robots to achieve control over them; however, this embodiment of the invention does not limit its application. Figure 3 As shown, the control system of the fruit picking robot may include a determination module 301, a movement control module 302, a first picking module 303, a second picking module 304, a third picking module 305, a judgment module 306, and an update module 307, wherein:

[0160] The determination module 301 is used to determine the current picking task to be executed. The picking task includes multiple picking sites and the location information corresponding to each picking site.

[0161] The movement control module 302 is used to move to the target picking site according to the location information of the target picking site, and all picking sites include the target picking site.

[0162] The first picking module 303 is used to perform a first picking processing operation matching the first picking item on all picking targets corresponding to the target picking site, and obtain the first picking information corresponding to all picking targets. The first picking item is the execution item for the target picking site in the picking task. The first picking processing operation includes information collection operation and picking queue construction operation.

[0163] The second picking module 304 is used to perform a second picking processing operation that matches the second picking items on all picking targets based on the first picking information, so as to obtain the second picking information corresponding to all picking targets.

[0164] The third picking module 305 is used to perform a third picking processing operation on all picking targets based on the second picking information.

[0165] The judgment module 306 is used to determine whether the picking task for the target picking site has been completed.

[0166] The update module 307 is used to update the completion progress of the picking task and move to the next picking site corresponding to the target picking site after the judgment module 306 determines that the picking task for the target picking site has been completed, so as to execute the picking task for the next picking site.

[0167] The second picking and processing operation includes multi-level information analysis and processing; the third picking and processing operation includes at least one of information collection, information processing, picking force calculation, and fruit picking.

[0168] It is evident that implementation Figure 3The described control system for a fruit-picking robot can automatically move to the corresponding target picking location based on a determined picking task. It then automatically collects image information of the target picking location and simultaneously constructs a picking queue. This image information is used to initially detect the area where all picking targets (such as fruits) are located at the target picking location, and the picking queue is used to determine the picking order for each picking target. This initially determines the picking range and picking execution order, improving picking efficiency. Subsequently, a second picking processing operation, namely a multi-level information analysis operation, is performed on the first picking information, including the image information and the picking queue. The algorithm further limits the area corresponding to the image information and adjusts and updates the picking queue; it improves the accuracy of relevant image information and the accuracy of queue construction; finally, based on the obtained second picking information, the third picking processing operation is performed on all picking targets, that is, the final picking operation is performed on each picking target. During this process, the machine force (picking force) used for picking is calculated, and further analysis of the information corresponding to the fruit contacted in real time is performed. Through the multi-layer algorithm of the first, second and third picking processing operations, the picking efficiency, picking accuracy and reliability of fruit picking at the target picking site are improved.

[0169] In an optional embodiment, the first picking module 303 performs a first picking processing operation matching the target picking items on all picking targets corresponding to the target picking site, and the specific method for obtaining the first picking information corresponding to all picking targets includes:

[0170] Generate information collection instructions for all picking targets corresponding to the target picking site, and collect the first acquisition images corresponding to all picking targets according to the information collection instructions and the first sensor configured on it;

[0171] Analyze all the first acquired images to obtain the fruit ripeness information for each picking target; at the same time, construct a target picking queue corresponding to all picking targets based on all the first acquired images. The target picking queue includes the picking order and picking position corresponding to each picking target.

[0172] Fruit ripening information and target picking queues are identified as the first picking information corresponding to all picking targets.

[0173] It is evident that implementation Figure 3The described control system for a fruit-picking robot can automatically collect image information of the target picking site based on a first sensor and simultaneously construct a picking queue after moving to the target picking site. This image information is used to initially detect the area where all picking targets (such as fruits) are located at the target picking site, and the picking queue is used to determine the picking order for each picking target. This system enables the initial determination of the picking range and picking execution order, improves picking efficiency, and facilitates subsequent adjustments and optimizations to the initially determined picking range and picking queue, thereby improving the execution efficiency of subsequent processes.

[0174] In another optional embodiment, the second picking module 304 performs a second picking processing operation matching the second picking task on all picking targets based on the first picking information, and the specific method for obtaining the second picking information corresponding to all picking targets includes:

[0175] Perform a first analysis and processing on the first picking information to obtain the first maturity information corresponding to the first picking information. The first maturity information includes a second acquired image, and the maturity of the fruit corresponding to the picking target in the second acquired image is greater than the target maturity threshold.

[0176] Based on preset maturity analysis parameters for the picking target, the first maturity information is processed by the second analysis to obtain the second maturity information corresponding to the first maturity information. The target picking queue is updated based on the second maturity information, the position information of the end effector, and preset calibration parameters. The end effector is configured on the top and is used to perform fruit picking. The calibration parameters include the sensor parameters corresponding to the first sensor and the picking type parameters corresponding to the picking target. The first sensor is externally mounted on the top.

[0177] The second maturity information and the updated target picking queue are determined as the second picking information corresponding to all picking targets.

[0178] As can be seen, in this optional embodiment, a second picking processing operation, namely a multi-level information analysis operation, is further performed on the first picking information, including image information and picking queue, to further limit the area corresponding to the image information and adjust and update the picking queue; thus improving the accuracy of the relevant image information and the accuracy of queue construction.

[0179] In another optional embodiment, the third picking module 305 performs the third picking processing operation on all picking targets based on the second picking information in the following specific ways:

[0180] Based on the target picking queue and the actuator coordinates of the end effector, a movement path is generated for the end effector, which includes the picking coordinates of each picking target;

[0181] The end effector moves according to the queue order of the target picking queue and the picking coordinates of each picking target, based on the movement path control.

[0182] For any picking target, when the end effector moves to the picking coordinates of the picking target, the end effector is controlled to collect the third picking information corresponding to the picking target, and the end effector is controlled to perform the third picking processing operation corresponding to the third picking information on the picking target.

[0183] For any picking target, after determining that the third processing operation for the picking target has been completed, the target picking queue is updated, and it is determined whether the completion progress of the target picking queue has reached the preset progress value. When it is determined that the completion progress of the target picking queue has reached the preset progress value, the judgment on whether the picking task for the target picking site has been completed is triggered.

[0184] When it is determined that the completion progress of the target picking queue has not reached the preset progress value, the control end effector moves to the next picking target corresponding to the picking target to perform the third picking processing operation on the next picking target.

[0185] Further optionally, the third harvesting module 305 controls the end effector to collect the third harvesting information corresponding to the harvesting target in the following ways:

[0186] The end effector is controlled to contact the picking target with the gripper; and the second sensor is controlled to collect the contact information of the picking target. The contact information includes embedded feature point offset information and spectral analysis information of the picking target. The embedded feature point offset information is the information corresponding to the force offset of the second sensor embedded in the gripper after the gripper contacts the picking target.

[0187] Calculate the actual spatial coordinates corresponding to the contact information based on the preset multi-camera parameters; and calculate the offset between the actual spatial coordinates and the preset standard spatial coordinates.

[0188] The offset is converted into a target three-dimensional force, which is the force corresponding to the gripper gripping and picking the target.

[0189] The target's three-dimensional force and spectral analysis information are identified as the third harvesting information corresponding to the harvesting target.

[0190] In this optional embodiment, spectral analysis information is used to determine whether the target to be harvested meets the conditions of no damage and maturity. The information type of the spectral analysis information includes a first type, a second type, or a third type. The first type indicates that the target to be harvested meets both the conditions of no damage and maturity. The second type indicates that the target to be harvested does not meet the conditions of no damage but meets the conditions of maturity. The third type indicates that the target to be harvested does not meet both the conditions of no damage and maturity.

[0191] The third harvesting module 305 controls the end effector to perform the third harvesting processing operation corresponding to the third harvesting information on the harvesting target in the following ways:

[0192] When the spectral analysis information is of type one or type two, the gripper of the control end effector grips the target based on the target's three-dimensional force and places the gripped target into the placement container that matches the information type of the spectral analysis information.

[0193] When the spectral analysis information is of type three, a discard flag is generated based on the third picking information of the picking target. The discard flag is used to indicate that the information type of the spectral analysis information corresponding to the picking target is type three.

[0194] As can be seen, in this optional embodiment, a third picking process can be performed on all picking targets based on the obtained second picking information, that is, the final picking operation is performed on each picking target. During this process, the machine force (picking force) used for picking is calculated, and further analysis of the information corresponding to the fruit contacted in real time is performed, which improves the picking efficiency, picking accuracy and reliability of picking fruit at the target picking site.

[0195] In another optional embodiment, the harvesting type parameter corresponding to the harvesting target is used to determine the harvesting execution method for the harvesting target. The harvesting execution method includes a first execution method or a second execution method, wherein:

[0196] When the picking execution mode is the first execution mode, the gripper and the second sensor corresponding to the end effector are activated when picking any picking target;

[0197] When the picking execution mode is the second execution mode, when picking any picking target, the gripper, the second sensor and the moving cutting tool corresponding to the end effector are activated. The moving cutting tool is external to the end effector and is used to cut off the target fruit stem area corresponding to the picking target.

[0198] As can be seen, in this optional embodiment, the picking method can be matched according to the type of the picking target, which helps to reduce the situation of picking unripe and / or poor quality fruits, thereby improving the accuracy and reliability of fruit picking.

[0199] Example 4

[0200] Please see Figures 4-5 , Figure 4-5 These are schematic diagrams illustrating the structures of two types of fruit-picking robots disclosed in embodiments of the present invention. Figure 4As shown, the fruit-picking robot may include an end effector, a first sensor, a second sensor, and a moving shearing tool; the fruit-picking robot is used to perform the steps in the control method of the fruit-picking robot described in Embodiment 1 or Embodiment 2 of the present invention.

[0201] In this embodiment of the invention, optionally, the outer wall of the end effector is physically connected to the first sensor; the inner wall of the end sensor is physically connected to the second sensor; and the outer wall of the end effector is physically connected to the moving shearing tool.

[0202] In this embodiment of the invention, optionally, such as Figure 4-5 As shown, the picking actuator 7 (end effect actuator) can be connected to the actuator motor and driver, end effect calculation and communication module. In actual application, the mobile shearing tool can be a mobile scissor.

[0203] In this embodiment of the invention, optionally, such as Figure 5 As shown, the fruit-picking robot may also include: a power module 1, a drive module 2, a positioning, navigation, and obstacle avoidance module 3, a computing, control, and communication module 4, a scheduling and data platform 5, a multi-joint robotic arm 6, a picking actuator 7 (end effector), a spectral-3D multi-mode sensor 8 (first sensor), a spectral-tactile multi-mode sensor 9 (second sensor), and a fruit container 10; wherein:

[0204] Power module 1 is used to provide power to the entire harvesting robot system, including power management and automatic charging.

[0205] Drive module 2 is used to drive wheeled or tracked mechanisms by a motor to achieve movement.

[0206] The positioning, navigation, and obstacle avoidance module 3 mainly consists of GPS, LiDAR, and / or 3D sensors. It is used to combine a pre-established orchard map with the computing control module to realize low-speed autonomous driving functions such as automatic positioning, navigation, obstacle avoidance, and path planning for the robot.

[0207] In addition to enabling the robot system's autonomous driving function, the computing, control, and communication module 4 needs to communicate with the scheduling and data platform to receive tasks, call and update orchard maps, and upload robot perception modules and harvesting data.

[0208] The scheduling and data platform 5 is used for functions such as coordinating and allocating multiple picking robots, planning tasks and paths, creating and updating orchard maps, storing robot perception data, and uploading picking data.

[0209] For details regarding the working principles and execution processes of the picking actuator 7 (end effect actuator), the spectral-3D multi-mode sensor 8 (first sensor), and the spectral-tactile multi-mode sensor 9 (second sensor) in this embodiment of the invention, please refer to the specific content of Embodiment 1 and Embodiment 2. This embodiment of the invention will not elaborate further.

[0210] Example 5

[0211] This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked, they are used to execute the steps in the control method for the fruit picking robot described in Embodiment 1 or Embodiment 2 of this invention.

[0212] Example 6

[0213] This invention discloses a computer program product, which includes a non-transitory computer storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps in the control method for the fruit picking robot described in Embodiment 1 or Embodiment 2.

[0214] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0215] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0216] Finally, it should be noted that the control method and system for a fruit-picking robot disclosed in the embodiments of the present invention, and the fruit-picking robot disclosed herein, are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for controlling a fruit-picking robot, characterized in that, The method is applied to a fruit-picking robot, and the method includes: The fruit picking robot determines the current picking task to be executed, the picking task includes multiple picking sites and the location information corresponding to each picking site; and moves to the target picking site according to the location information of the target picking site, all of which include the target picking site; The fruit picking robot performs a first picking processing operation matching the first picking item on all picking targets corresponding to the target picking site, and obtains the first picking information corresponding to all picking targets. The first picking item is the execution item for the target picking site in the picking task. The first picking processing operation includes information collection operation and picking queue construction operation. The fruit picking robot performs a second picking process operation matching the second picking task on all the picking targets based on the first picking information, thereby obtaining second picking information corresponding to all the picking targets; and performs a third picking process operation on all the picking targets based on the second picking information. The fruit picking robot determines whether the picking task for the target picking site has been completed. When it determines that the picking task for the target picking site has been completed, it updates the completion progress of the picking task and moves to the next picking site corresponding to the target picking site to perform the picking task for the next picking site. The second harvesting operation includes multi-level information analysis and processing; the third harvesting operation includes at least one of information collection, information processing, harvesting force calculation, and fruit harvesting. The fruit-picking robot performs a first picking process operation matching the first picking task on all picking targets corresponding to the target picking site, and obtains first picking information corresponding to all picking targets, including: The fruit picking robot generates information collection instructions for all picking targets corresponding to the target picking site, and collects first images corresponding to all picking targets based on the information collection instructions and the first sensor configured on the fruit picking robot. The fruit-picking robot analyzes all the first acquired images to obtain fruit ripeness information for each picking target; at the same time, it constructs a target picking queue corresponding to all the picking targets based on all the first acquired images, and the target picking queue includes the picking order and picking position corresponding to each picking target. The fruit picking robot determines the fruit ripeness information and the target picking queue as the first picking information corresponding to all the picking targets; The fruit-picking robot performs a third picking process on all the picking targets based on the second picking information, including: The fruit picking robot generates a movement path for the end effector based on the target picking queue and the actuator coordinates of the end effector. The movement path includes the picking coordinates of each picking target. The fruit picking robot controls the end effector to move according to the queue order of the target picking queue and the picking coordinates of each target picking according to the movement path; For any of the picking targets, when the end effector is determined to have moved to the picking coordinates of the picking target, the fruit picking robot controls the end effector to collect the third picking information corresponding to the picking target, and controls the end effector to perform the third picking processing operation corresponding to the third picking information on the picking target; The fruit-picking robot controls the end effector to collect third picking information corresponding to the picking target, including: The fruit picking robot controls the gripper corresponding to the end effector to contact the picking target; and controls the second sensor to collect the contact information of the picking target. The contact information includes embedded feature point offset information and spectral analysis information of the picking target. The embedded feature point offset information is the information corresponding to the force offset of the second sensor embedded in the gripper after the gripper contacts the picking target. The fruit-picking robot calculates the actual spatial coordinates corresponding to the contact information based on preset multi-camera parameters; and calculates the offset between the actual spatial coordinates and the preset standard spatial coordinates. The fruit-picking robot converts the offset into a target three-dimensional force, which is the force corresponding to the gripper gripping the picking target. The fruit-picking robot determines the target's three-dimensional force and the spectral analysis information as the third picking information corresponding to the picking target.

2. The control method for a fruit-picking robot according to claim 1, characterized in that, The fruit-picking robot performs a second picking process operation matching the second picking task on all the picking targets based on the first picking information, and obtains the second picking information corresponding to all the picking targets, including: The fruit picking robot performs a first analysis and processing on the fruit ripening information to obtain first ripening information corresponding to the fruit ripening information. The first ripening information includes a second acquired image, and the ripening degree of the fruit corresponding to the picking target in the second acquired image is greater than the target ripening threshold. The fruit-picking robot performs a second analysis on the first maturity information according to preset maturity analysis parameters for the picking target, to obtain second maturity information corresponding to the first maturity information, and updates the target picking queue according to the second maturity information, the position information of the end effector, and preset calibration parameters. The end effector is configured on the fruit-picking robot for performing fruit picking. The calibration parameters include sensor parameters corresponding to the first sensor and picking type parameters corresponding to the picking target. The first sensor is externally mounted on the fruit-picking robot. The fruit picking robot determines the second ripeness information and the updated target picking queue as the second picking information corresponding to all the picking targets.

3. The control method for a fruit-picking robot according to claim 2, characterized in that, The fruit-picking robot performs a third picking process on all the picking targets based on the second picking information, and further includes: For any of the picking targets, after determining that the third processing operation for the picking target has been completed, the fruit picking robot updates the target picking queue and determines whether the completion progress of the target picking queue has reached a preset progress value. When it is determined that the completion progress of the target picking queue has reached the preset progress value, the determination of whether the picking task for the target picking site has been completed is triggered. When it is determined that the completion progress of the target picking queue has not reached the preset progress value, the fruit picking robot controls the end effector to move to the next picking target corresponding to the picking target, so as to perform the third picking processing operation on the next picking target.

4. The control method for a fruit picking robot according to claim 1, characterized in that, The spectral analysis information is used to determine whether the harvested target meets the conditions of no damage and maturity. The information type of the spectral analysis information includes a first type, a second type, or a third type. The first type indicates that the harvested target simultaneously meets the conditions of no damage and maturity. The second type indicates that the harvested target does not meet the conditions of no damage but meets the conditions of maturity. The third type indicates that the harvested target does not simultaneously meet the conditions of no damage and maturity. The fruit-picking robot controls the end effector to perform a third picking processing operation on the picking target corresponding to the third picking information, including: When the spectral analysis information is of the first type or the second type, the fruit picking robot controls the gripper of the end effector to grip the picking target based on the target three-dimensional force, and places the gripped picking target into the placement container that matches the information type of the spectral analysis information; When the spectral analysis information is of the third type, the fruit picking robot generates a discard flag based on the third picking information of the picking target. The discard flag is used to indicate that the information type of the spectral analysis information corresponding to the picking target is the third type.

5. The control method for a fruit picking robot according to claim 2, characterized in that, The harvesting type parameter corresponding to the harvesting target is used to determine the harvesting execution method for the harvesting target. The harvesting execution method includes a first execution method or a second execution method, wherein: When the picking execution mode is the first execution mode, the fruit picking robot activates the gripper, the first sensor, and the second sensor corresponding to the end effector when picking any of the picking targets. When the picking execution mode is the second execution mode, when the fruit picking robot picks any of the picking targets, it activates the gripper, the first sensor, the second sensor, and the moving shearing tool corresponding to the end effector. The moving shearing tool is externally mounted on the end effector to cut off the target fruit stem area corresponding to the picking target.

6. A control system for a fruit-picking robot, characterized in that, The system is applied to a fruit-picking robot, and the system includes: The determination module is used to determine the picking task to be executed, wherein the picking task includes multiple picking sites and location information corresponding to each picking site; A movement control module is used to move to the target picking site according to the location information of the target picking site, and all the picking sites include the target picking site; The first picking module is used to perform a first picking processing operation matching the first picking item on all picking targets corresponding to the target picking site, and obtain the first picking information corresponding to all the picking targets. The first picking item is the execution item for the target picking site in the picking task. The first picking processing operation includes information collection operation and picking queue construction operation. The second picking module is used to perform a second picking processing operation matching the second picking matter on all the picking targets according to the first picking information, so as to obtain the second picking information corresponding to all the picking targets. The third picking module is used to perform a third picking processing operation on all the picking targets based on the second picking information. The judgment module is used to determine whether the picking task for the target picking site has been completed; The update module is used to update the completion progress of the picking task and move to the next picking site corresponding to the target picking site after the judgment module determines that the picking task for the target picking site has been completed, so as to perform the picking task for the next picking site. The second harvesting operation includes multi-level information analysis and processing; the third harvesting operation includes at least one of information collection, information processing, harvesting force calculation, and fruit harvesting. The first picking module performs a first picking processing operation matching the first picking matter on all picking targets corresponding to the target picking site, and obtains the first picking information corresponding to all the picking targets in the following specific ways: Generate information collection instructions for all picking targets corresponding to the target picking site, and collect first images corresponding to all picking targets according to the information collection instructions and the first sensor configured on it; Analyze all the first acquired images to obtain fruit ripening information for each of the harvesting targets; at the same time, construct a target harvesting queue corresponding to all the harvesting targets based on all the first acquired images, the target harvesting queue including the harvesting order and harvesting position corresponding to each of the harvesting targets; The fruit ripening information and the target picking queue are determined as the first picking information corresponding to all the picking targets; The third picking module performs the third picking processing operation on all the picking targets based on the second picking information in the following ways: Based on the target picking queue and the actuator coordinates of the end effector, a movement path is generated for the end effector, the movement path including the picking coordinates of each picking target; The end effector is controlled to move according to the queue order of the target picking queue and the picking coordinates of each target picking according to the movement path; For any of the harvesting targets, when the end effector is determined to have moved to the harvesting coordinates of the harvesting target, the end effector is controlled to collect the third harvesting information corresponding to the harvesting target, and the end effector is controlled to perform a third harvesting processing operation on the harvesting target corresponding to the third harvesting information; The specific methods by which the third harvesting module controls the end effector to collect the third harvesting information corresponding to the harvesting target include: The end effector is controlled to contact the grasper corresponding to the picking target; and the second sensor is controlled to collect the contact information of the picking target. The contact information includes embedded feature point offset information and spectral analysis information of the picking target. The embedded feature point offset information is the information corresponding to the force offset of the second sensor embedded in the grasper after the grasper contacts the picking target. Calculate the actual spatial coordinates corresponding to the contact information based on preset multi-camera parameters; and calculate the offset between the actual spatial coordinates and the preset standard spatial coordinates. The offset is converted into a target three-dimensional force, which is the force corresponding to the gripper gripping the harvested target. The target's three-dimensional force and the spectral analysis information are determined as the third harvesting information corresponding to the harvesting target.

7. A fruit-picking robot, characterized in that, The fruit-picking robot includes an end effector, a first sensor, a second sensor, and a moving shearing tool; the fruit-picking robot is used to execute the control method for the fruit-picking robot as described in any one of claims 1-5.

8. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which, when invoked, are used to execute the control method for the fruit picking robot as described in any one of claims 1-5.

Images