Picking robot and method based on multi-robot partition layout and cooperation

Abstract

The present application relates to the technical field of picking equipment, and discloses a picking robot and method based on multi-robot arm partition layout and cooperation, which comprises a self-walking device and a vehicle body support device, at least three execution robot arms, a fruit storage and transportation device, a visual positioning device and an intelligent control device. The at least three execution robot arms are distributed along the top two sides of the periphery of the vehicle body support device. The number of the visual positioning devices corresponds to the number of the execution robot arms, and the visual positioning devices are fixed to positions capable of obtaining the visual angle of the corresponding execution robot arms. The intelligent control device controls the execution robot arms to simultaneously collect fruit image data of different regions, and constructs a three-dimensional model of the fruits in the sheltered place. The present application improves the picking success rate in complex environments and reduces fruit damage. The present application realizes the deep cooperation of multi-robot arms in the perception level, forms a distributed visual network, and enhances the adaptability and operation stability to the unstructured orchard environment.

Description

Technical Field

[0001] This invention relates to the field of harvesting equipment technology, and in particular to a harvesting robot and method based on a multi-arm partitioned layout and collaboration. Background Technology

[0002] The statements in this section are merely to provide background information related to the disclosure of this invention and do not necessarily constitute prior art.

[0003] Fruit harvesting is a crucial step in agricultural production, and its efficiency and quality directly impact the economic benefits of the produce. As a major producer of round fruits such as apples and pears, my country generally faces labor shortages and high labor costs in its harvesting operations. Especially during peak fruit ripening periods, failure to harvest in a timely manner can lead to decreased fruit quality and even fruit drop, resulting in significant economic losses. Therefore, the development of efficient and intelligent fruit harvesting robots has become a research hotspot in the field of agricultural engineering.

[0004] Currently, existing harvesting robots mainly fall into two technical categories: single-arm and multi-arm. Single-arm harvesting robots are simple in structure and easy to control, but they suffer from inherent drawbacks such as low operating efficiency and limited coverage area per operation, making it difficult to meet the harvesting needs of large-scale plantations. To overcome these problems, multi-arm collaborative harvesting robots have emerged. For example, existing technologies disclose harvesting robots equipped with multiple robotic arms, which expand the coverage area of ​​a single operation and improve overall harvesting efficiency by distributing multiple robotic arms on both sides of the vehicle body. This type of solution typically divides the fruit tree canopy on both sides of the vehicle body into multiple independent areas and assigns a dedicated robotic arm and vision sensor to each area, achieving an independent parallel operation mode of "one arm per area".

[0005] However, the existing multi-arm harvesting robots still have the following shortcomings in practical applications: There is a lack of effective mechanisms to address fruit occlusion: Orchard environments are highly unstructured, and fruits are often partially or completely obscured by objects such as leaves and branches. In the existing solutions mentioned above, each robotic arm relies solely on a single visual sensor in its corresponding area for perception. When fruit is occluded, the two-dimensional or three-dimensional information obtained from a single viewpoint is extremely limited, making it impossible to accurately reconstruct the complete spatial pose of the occluded fruit. This results in the robotic arm struggling to perform effective grasping, a significant decrease in harvesting success rate, and even damage to the fruit or the robotic arm.

[0006] Multi-arm collaboration only reaches the task allocation level, lacking deep collaboration at the perception and operation levels: In existing solutions, although multiple robotic arms are physically distributed, their operational logic is independent. Each robotic arm is only responsible for picking the "visible" fruits within its own zone, without information exchange or coordinated actions between them. When a task in a zone cannot be completed by a single arm due to complex occlusion, the system cannot mobilize robotic arms in other zones for assistance (such as collecting information from different perspectives or helping to clear away branches and leaves), causing the fruit to be missed and reducing the overall harvest rate.

[0007] Insufficient adaptability to unstructured environments: Existing vision systems mostly operate independently, and the image data collected by each camera lacks effective fusion. When encountering extreme scenarios such as complex tree shapes and dense foliage, the reliability of information acquired by a single sensor decreases, making it impossible to plan a precise and safe collision-free picking path for the robotic arm. This limits the stability and reliability of the robot's operation in real, complex orchard environments.

[0008] Therefore, how to provide a harvesting robot that can effectively address the problem of fruit shading and achieve deep collaboration among multiple robotic arms at the perception and operational levels is a technical challenge that urgently needs to be solved in this field. This invention aims to solve the aforementioned problem. Summary of the Invention

[0009] To overcome the shortcomings of the existing technology, this invention provides a harvesting robot and method based on multi-arm partitioned layout and collaboration. This method achieves three-dimensional reconstruction and precise harvesting of occluded fruits through the collaboration of a distributed visual positioning device and an intelligent control device, integrating multi-view image information. This significantly improves the harvesting success rate in complex environments and reduces fruit damage. Simultaneously, it breaks through the limitations of traditional multi-arm systems that operate independently, achieving deep collaboration among multiple robotic arms at the perception level to form a distributed visual network. This enhances adaptability and operational stability in unstructured orchard environments. Furthermore, this solution retains a parallel operation architecture while addressing the occlusion problem, taking into account harvesting efficiency in conventional scenarios. It achieves unified optimization of overall operational efficiency and success rate.

[0010] The technical solution adopted in this invention is: a fruit-picking robot based on a multi-arm partitioned layout and collaboration, including a self-propelled device and a vehicle body support device connected thereto. The fruit-picking robot also has: A robotic arm harvesting device, comprising at least three robotic arms distributed at intervals along the periphery of the top two sides of a vehicle body support device, each robotic arm having a gravity sensor; and Fruit storage and transportation device, wherein the fruit storage and transportation device is placed on top of the vehicle body support device to transport harvested fruit; and A visual positioning device, the number of which corresponds to the number of the robotic arms, and fixed to a position on the vehicle body support device where the corresponding viewpoint of the robotic arm can be obtained; and An intelligent control device is provided, which connects to and controls a self-propelled device, a visual positioning device, a robotic arm picking device, and a fruit storage and transportation device. The intelligent control device controls the robotic arm on the same side of the vehicle body support to simultaneously collect fruit image data from different regional perspectives to obtain a three-dimensional model of the fruit. This model is used to construct a three-dimensional model of the fruit in the obscured area, thereby achieving virtual modeling of the fruit in the obscured scene. This model is used to guide the picking task of the robotic arm and achieve precise positioning of composite targets such as branches and leaves.

[0011] In this technical solution, the robotic arm harvesting device further includes: Screw rails fixed to the top of the fruit storage and transportation device; A sliding actuator arm connected to a lead screw guide rail, the actuator arm also being provided with at least a flexible gripper.

[0012] In this technical solution, the fruit storage and transportation device includes a conveyor belt and a fruit basket placed on it, and the inside of the fruit basket is provided with shock-absorbing sponge.

[0013] In this technical solution, the visual positioning device is an RGB-D camera.

[0014] In this technical solution, the vehicle body support device is made of aluminum alloy sheet, and intelligent control cabinet door and heat dissipation ventilation holes are set on the aluminum alloy sheet, thereby realizing the overall design of vehicle body lightness and heat dissipation performance.

[0015] In this technical solution, the number of robotic arms is six, the side of the aluminum alloy plate is also equipped with a bracket, and the autonomous walking device also includes tracks, a high-performance battery, a drive motor, GPS, LiDAR and a vision camera.

[0016] A multi-robotic arm collaborative fruit picking method includes fruit picking based on the aforementioned picking robot, and includes the following steps: S1. When fruit is detected in a complex occlusion scene, the canopy point cloud data of the visual positioning device on the same side of the vehicle body support device is obtained, and a three-dimensional scene model of the fruit is generated through multi-view fusion reconstruction technology. S2. Use the Mask R-CNN semantic segmentation algorithm to identify the occluded fruit, combine the geometric features of branches and leaves to calculate the optimal interference path, and determine the initial clamping point for branch and leaf movement. S3. Based on the Q-learning reinforcement learning algorithm, plan the motion trajectory of the robotic arm on the same side to control the robotic arm to perform a precise fruit peeling operation; S4. When the fruit is fully exposed, the positioning information of the first fruit to be picked is transmitted to the robotic arm, and the robotic arm is controlled to grasp and rotate to grasp the fruit. After completion, it moves back a distance L from the current coordinate position. S5. Determine whether the fruit was successfully picked using the gravity sensor on the robotic arm; S6. If so, the robotic arm will transfer the fruit to a distance H directly above the nearest fruit basket, and then the end effector will release, allowing the transport basket to harvest the fruit; Repeat steps S1 to S5 until the occluded fruit in the 3D scene model of the fruit is completed.

[0017] In this technical solution, the fruit harvesting method also includes: S7. The harvesting robot acquires RGB and depth images of most tube layers in the harvesting area of ​​each visual positioning device; S8. Determine whether the acquired RGB image and depth image are clear; If the image is clear, locate the first fruit to be picked in the unobstructed scene of each area, and control the corresponding robotic arm to pick it. If the image is not clear, collect it again. S9. Determine whether the fruit has been picked by acquiring the gravity sensor data on the corresponding robotic arm; S10. If fruit is harvested, obtain fruit information and control the robotic arm to place it into a suitable fruit storage and transportation device.

[0018] In this technical solution, the fruit picking method further includes: It receives harvesting tasks and can locate and move around autonomously; Determine whether there is fruit obstructing the view during harvesting; if there is obstruction, proceed to steps S1 to S5; if there is no obstruction, proceed to steps S6 to S9.

[0019] In this technical solution, after a fruit is picked in step S3, the distance L that moves backward from the current coordinate position is 30-60cm, so as to leave enough moving distance for the next picking and ensure accurate picking next time; In step S5, the robotic arm transfers the fruit to the nearest fruit basket at a distance H of 8-15cm, leaving enough space for the next harvesting step to facilitate better harvesting.

[0020] Compared with the prior art, the beneficial effects of the present invention are: 1. Significantly improved the harvesting success rate of shaded fruits: This invention utilizes a visual positioning device corresponding to the number of robotic arms, and an intelligent control device to control multiple robotic arms on the same side of the vehicle to simultaneously collect fruit image data from different regional perspectives. This allows for the acquisition of a 3D model of the fruit and the construction of a virtual model of the fruit in an occluded scene. Furthermore, by fusing multi-view image information, it can reasonably infer and reconstruct the spatial position, posture, and morphology of the occluded portion of the fruit, providing the robotic arms with far more accurate environmental perception information than a single viewpoint. This effectively guides the robotic arms to accurately harvest occluded fruit, significantly improving the harvesting success rate in complex orchard environments and reducing fruit loss and damage caused by occlusion.

[0021] 2. Deep collaboration among multiple robotic arms at the perception level has been achieved: Existing multi-arm harvesting robots typically only achieve independent parallel operations at the task allocation level, lacking information interaction between the arms. This invention uses an intelligent control device to uniformly schedule all robotic arms on the same side to collaboratively acquire images, fusing data from multiple distributed visual sensors at the control terminal. This transforms the multiple robotic arms from independent "harvesting units" into a distributed visual perception network. This network can cover a wider spatial range and provide multi-baseline stereo vision capabilities, achieving a leap from "independent operation" to "collaborative perception," providing more complete and reliable environmental data support for the robotic arms' path planning and obstacle avoidance.

[0022] 3. Enhance the robot's adaptability and stability in unstructured orchard environments: This invention, through the coordinated use of distributed visual positioning devices and intelligent control devices, enables robots to effectively handle unstructured scenarios such as complex tree shapes and dense foliage. The ability to generate 3D fruit models and virtual modeling based on multi-view image data allows harvesting path planning to move beyond relying on single, potentially occluded local views, instead establishing a virtual fruit model based on fused global image data. This significantly improves the success rate and safety of the robotic arm in complex scenarios, reduces the risk of collisions due to perceptual errors, and allows the harvesting robot to adapt to a wider range of realistic orchard environments.

[0023] 4. While addressing the issue of obstruction, overall harvesting efficiency should also be considered: This invention endows multiple robotic arms with collaborative sensing and occlusion handling capabilities while retaining the parallel operation architecture of "at least three robotic arms distributed in a distributed manner." In unoccluded scenarios, each robotic arm can still independently execute the harvesting task within its own area, ensuring overall harvesting efficiency. When encountering occluded scenarios, it can flexibly switch to collaborative sensing and harvesting mode. Therefore, this invention significantly improves adaptability to complex scenarios while achieving a unified optimization of operational efficiency and success rate.

[0024] In summary, this invention, through the collaboration of a distributed visual positioning device and an intelligent control device, integrates multi-view image information to achieve three-dimensional reconstruction and precise harvesting of occluded fruits, significantly improving the harvesting success rate and reducing fruit damage in complex environments. It also achieves deep collaboration among multiple robotic arms at the perception level, forming a distributed visual network that enhances adaptability and operational stability in unstructured orchard environments. Furthermore, while addressing the occlusion problem, it retains a parallel operation architecture, taking into account harvesting efficiency in conventional scenarios, thus achieving a unified optimization of overall operational efficiency and success rate. Attached Figure Description

[0025] Figure 1 A three-dimensional structural diagram of one embodiment of a harvesting robot; Figure 2 A three-dimensional structural diagram of one embodiment of the vehicle body support device; Figure 3 A 3D structural diagram of the robotic arm picking device in conjunction with a vision positioning device; Figure 4 A three-dimensional structural diagram of one embodiment of a fruit storage and transportation device; Figure 5 A flowchart of an embodiment of a multi-robotic arm collaborative fruit picking method; Figure 6 A flowchart of another embodiment of a multi-robotic arm collaborative fruit picking method; Figure 7 A flowchart of a multi-robotic arm collaborative fruit picking method; Figure 8 A three-dimensional structural diagram of one embodiment of the self-propelled device; Among them: 1-self-propelled device, 2-vehicle support device, 21-aluminum alloy plate, 22-intelligent control cabinet door, 23-heat dissipation and ventilation holes, 24-bracket; 3-visual positioning device; 4-robotic arm picking device, 41-screw slide rail, 42-execution robotic arm; 5-fruit storage and transportation device, 51-fruit basket, 52-shock-absorbing sponge, 53-conveyor belt; 6-intelligent control device. Detailed Implementation

[0026] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0027] In the description of this invention, it should be understood that the terms "center," "upper," "lower," "front," "rear," "left," and "right," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the combination or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, in the description of the embodiments of this invention, the positional relationships of devices such as "upper," "lower," "front," "rear," "left," and "right" in all figures are based on… Figure 1 As the standard.

[0028] like Figure 1 and Figure 3 As shown, a fruit-picking robot based on a multi-arm partitioned layout and collaboration includes a self-propelled device 1 and a vehicle body support device 2 connected thereto. The fruit-picking robot also has: A robotic arm harvesting device 4, comprising at least three robotic arms 42, which are distributed at intervals along the periphery of the top two sides of the vehicle body support device 2. Each robotic arm 42 is equipped with a gravity sensor. Fruit storage and transportation device 5, which is placed on top of vehicle body support device 2 to transport harvested fruit; and A visual positioning device 3, the number of which corresponds to the number of the robotic arms 42, and fixed to the vehicle body support device 2 at a position where the corresponding view of the robotic arm 42 can be obtained; and The intelligent control device 6 connects to and controls the self-propelled device 1, the visual positioning device 3, the robotic arm picking device 4, and the fruit storage and transportation device 5. The intelligent control device 6 controls the robotic arm 42 on the same side of the vehicle body support device 2 to simultaneously collect fruit image data from different regional perspectives to obtain a three-dimensional model of the fruit, so as to construct a three-dimensional model of the fruit in the occluded area, realize virtual modeling of the fruit in the occluded scene, and guide the picking task of the robotic arm 42 to achieve precise positioning of the composite target of branches and leaves.

[0029] In the specific implementation process, the autonomous walking device 1 includes tracks, a high-performance battery, drive motors, GPS, LiDAR, and a vision camera. The high-performance battery and drive motors provide power, while the tracks adapt to the uneven terrain of the orchard, enabling the robot to move forward, backward, and turn. Controlling the rotation speed of the drive motors on both sides ensures the robot can flexibly adjust its walking direction in complex orchard environments. The autonomous walking device employs a high-precision GPS navigation system and multi-sensor fusion technology, ensuring the robot's autonomous, safe, and precise movement in complex terrain, thus improving its adaptability and stability.

[0030] The harvesting robot operates according to the following steps: The robot moves into the orchard via a self-propelled device, and the intelligent control device coordinates the self-propelled device, the visual positioning device, the robotic arm picking device, and the fruit storage and transportation device. When encountering a scenario where branches and leaves obscure the fruit, the intelligent control device will control multiple robotic arms on the same side of the vehicle to simultaneously capture multiple images of the fruit from different perspectives using their respective visual positioning devices. Subsequently, the harvesting robot fuses these multi-view image data to reconstruct a realistic 3D model of the fruit and makes reasonable inferences about the shape and position of the obscured parts, constructing a complete virtual model of the fruit. Based on this precise perception information, the intelligent control device plans the optimal harvesting path for each robotic arm, guiding it to accurately grasp and harvest the obscured fruit, and finally placing the fruit into the fruit storage and transportation device. Throughout the process, each robotic arm can operate independently in sections to ensure efficiency, and can also effectively solve the occlusion problem through collaborative perception and virtual modeling when needed, achieving precise target positioning and harvesting in complex environments with branches and leaves.

[0031] In at least one embodiment, such as Figure 3As shown, the robotic arm harvesting device 4 also includes: a lead screw slide rail 41 fixed to the top of the fruit storage and transportation device 5; and a sliding execution robotic arm 42 connected to the lead screw slide rail 41. The intelligent control device drives the lead screw slide rail 41 to move the execution robotic arm 42 along the lead screw slide rail 41 according to the fruit's position and obstruction, allowing the execution robotic arm 42 to flexibly adjust its harvesting position and cover a wider working area. In this structure, the lead screw slide rail 41 converts rotational motion into linear motion, achieving high-precision, low-friction position adjustment, ensuring the robotic arm reaches the target point quickly and smoothly. The execution robotic arm 42 is also equipped with at least a flexible gripper 421. The execution robotic arm 42 and the flexible gripper 421 can be pre-made components in the specific implementation process, without the need for separate design, as long as they can complete the action. Alternatively, they can be designed according to the functional requirements. Overall, they can adaptively adjust according to the texture and shape of different fruits, effectively reducing damage during harvesting and ensuring the integrity of the fruit.

[0032] In at least one embodiment, such as Figure 4 As shown, the fruit storage and transportation device 5 includes a conveyor belt 53 and fruit baskets 51 placed on it. The fruit baskets 51 are equipped with shock-absorbing sponges 52. When a fruit basket 51 is full, the conveyor belt 53 automatically transports it backwards, while simultaneously sending empty fruit baskets into the harvesting area. The conveyor belt 53 is driven by a motor to operate continuously, enabling the cyclical movement of the fruit baskets 51. The shock-absorbing sponges 52 utilize flexible materials to absorb the impact energy of falling fruits or during transportation, reducing the breakage rate. The conveyor belt enables automatic circulation of fruit baskets, avoiding frequent manual handling, improving continuous operation efficiency, and making the harvesting process more automated and smooth. In specific operations, at least two fruit baskets 51 are used for easy replacement and harvesting; three baskets can simultaneously perform harvesting, transportation, and preparation, resulting in even better efficiency.

[0033] In at least one embodiment, the visual positioning device 3 is an RGB-D camera. The RGB-D camera integrates a traditional camera with a depth sensor (such as structured light or Time-of-Flight (ToF) technology), enabling it to output the three-dimensional spatial coordinates corresponding to each pixel. This allows it to acquire a dense point cloud on the fruit surface, enabling the RGB-D camera to simultaneously capture both color images and depth data of the fruit and transmit these two types of information to the intelligent control device in real time. The RGB-D camera can directly provide precise distance information between the fruit and branches, significantly improving the accuracy of three-dimensional reconstruction of the spatial position, posture, and shape of occluded fruits. Even in orchard environments with varying lighting, it can provide reliable visual guidance for the robotic arm, effectively enhancing the harvesting success rate and obstacle avoidance capabilities.

[0034] In at least one embodiment, the vehicle body support device 2 is made of aluminum alloy sheet 21, and an intelligent control cabinet door 22 and heat dissipation ventilation holes 23 are provided on the aluminum alloy sheet 21, thereby achieving a coordinated design of vehicle body lightweight and heat dissipation performance. The vehicle body support device 2 uses high-strength aluminum alloy sheet to ensure that the robot can maintain a stable structure when carrying heavy objects such as six robotic arms and fruit storage devices. In specific implementation, six robotic arm sliding tracks are distributed on both sides of the upper part of the vehicle body support device 2, providing six robotic arms 42 with individual connection and movement, which can control the six robotic arms 42 to achieve longitudinal translational movement.

[0035] In at least one embodiment, the number of robotic arms 42 is six. Six robotic arms 42 significantly improve parallel harvesting efficiency and coverage. The support frame enhances the stability of the fruit basket when fully loaded or during dynamic operations, reducing the risk of fruit collisions and tipping over, further lowering the damage rate, and ensuring smooth operation of the conveyor belt, making the overall structure more reliable. The aluminum alloy plate 21 is also equipped with a support frame 24 on its side to support the fruit basket 51, facilitating transport and convenient use. Figure 8 As shown, the autonomous walking device 1 also includes tracks 11, a high-performance battery 12, a drive motor 13, a GPS 14, a LiDAR 15, and a vision camera 16. The high-performance battery and drive motor provide power, and the tracks can adapt to the uneven terrain of the orchard, enabling the robot to move forward, backward, and turn. By controlling the rotation speed of the drive motors on the left and right sides, the robot can flexibly adjust its walking direction in complex orchard environments. The autonomous walking control system is based on data from multiple sensors such as GPS, LiDAR, and vision cameras to achieve autonomous navigation and obstacle avoidance for the robot, ensuring that the picking robot moves stably and safely within the orchard.

[0036] In the specific implementation process, the number of robotic arms 42 is six. The zonal visual guidance system formed by the robot consists of six sets of RGB-D cameras and image processing modules. The canopy of fruit trees on both sides of the robot is divided into six independent working areas, and each area is assigned an RGB-D camera and a robotic arm 42, thus constructing a distributed working architecture of "one machine per area".

[0037] The zoned visual guidance system executes the following intelligent process: Six RGB-D cameras are distributed on both sides of the vehicle body, capable of capturing real-time image information of the fruit tree canopy within six zones; the image processing module integrates a multi-core parallel computing architecture, receiving six visual streams in real time, and using a GPU acceleration engine to complete image denoising, color correction, and depth map registration, generating high-precision 3D point cloud data; a fruit detection model based on a deep learning framework, combined with geometric feature analysis technology, accurately locates unobstructed fruit targets; a fruit-robotic arm spatial relationship topology map is established in the multi-actuator collaborative system, and an improved Dijkstra algorithm is used to generate the optimal picking sequence. During the picking process, a collision detection mechanism ensures spatial isolation of the multiple robotic arms, efficiently completing precise harvesting operations. This zoned independent picking operation method, through spatial partitioning decoupling and intelligent task scheduling, significantly improves the automated harvesting efficiency in complex orchard environments while ensuring operational accuracy. This solution not only ensures that each robotic arm does not interfere with each other during operation but also significantly improves the overall efficiency of the robot's harvesting operation, providing a revolutionary multi-robotic arm collaborative operation solution for the field of agricultural robotics.

[0038] like Figure 5 As shown, a multi-robotic arm collaborative fruit picking method includes fruit picking based on the aforementioned picking robot, and includes the following steps: S1. When fruit is detected in a complex occlusion scene, the canopy point cloud data of the visual positioning device 3 on the same side of the vehicle body support device 2 is obtained, and a three-dimensional scene model of the fruit is generated through multi-view fusion reconstruction technology. S2. The Mask R-CNN semantic segmentation algorithm is used to identify occluded fruits. The optimal interference path is calculated by combining the geometric features of branches and leaves, and the initial clamping point for branch and leaf movement is determined. The Mask R-CNN semantic segmentation algorithm is an instance segmentation method; its goal is not only to perform pixel-level classification but also to distinguish different individuals of the same category. In the canopy point cloud data of the aforementioned visual positioning device 3, it generates an independent and different mask for each image and depth data obtained by each visual positioning device 3. It adopts a two-stage strategy of "detection first, then segmentation" to achieve more refined differentiation. S3. Based on the Q-learning reinforcement learning algorithm, plan the motion trajectory of the robotic arm 42 on the same side to control the robotic arm 42 to perform precise fruit peeling operation. Specifically, Q-learning is a classic model-free, value-based off-policy algorithm in reinforcement learning. It aims to enable the picking robot to learn a policy that can choose the optimal action in different states through trial and error learning, thereby achieving optimal fruit peeling. S4. When the fruit is fully exposed, the positioning information of the first fruit to be picked is transmitted to the execution robotic arm 42, and the execution robotic arm 42 is controlled to grasp and rotate to grasp the fruit. After completion, it moves back a distance L from the current coordinate position. S5. Determine whether the fruit was successfully picked using the gravity sensor on the robotic arm; If so, the robotic arm will transfer the fruit to a distance H directly above the nearest fruit basket, and then the end effector will release, allowing the transport basket to harvest the fruit; Repeat steps S1 to S5 until the occluded fruit in the 3D scene model of the fruit is completed.

[0039] This fruit-picking method combines a collaborative workflow between occluded and unoccluded scenarios, and its process is as follows: First, when complex shading of fruit is detected, the robot simultaneously collects canopy point cloud data using multiple RGB-D cameras on the same side of the robot body, and reconstructs a 3D scene model of the fruit through multi-view fusion. Then, the Mask R-CNN semantic segmentation algorithm is used to identify the shading fruit, and the optimal interference path is calculated based on the geometric features of branches and leaves to determine the initial gripping point for branch and leaf movement. Next, Q-learning reinforcement learning is used to plan the motion trajectory of multiple robotic arms on the same side, controlling the robotic arms to accurately peel off branches and leaves. The moment the fruit is fully exposed, the target fruit positioning information is transmitted to the robotic arms, enabling them to grasp and rotate for harvesting. After harvesting, the robot moves back a distance L. The gravity sensor determines whether the harvest was successful. If successful, the fruit is transferred to the top of the fruit basket and released, completing the harvest. The above steps are repeated until all shading fruit in the scenario has been harvested.

[0040] In unobstructed scenarios, the robot acquires RGB and depth images from various visual positioning devices, determines image clarity, locates the first fruit to be picked in each area, controls the corresponding robotic arm to pick it directly, and uses a gravity sensor to confirm successful picking. Finally, the fruit is placed into a suitable fruit storage and transportation device.

[0041] This method effectively solves the problem of fruit shading in complex orchard environments, significantly improving the harvesting success rate; multiple robotic arms can coordinate perception and action in shading scenarios, and operate independently and in parallel in unshading scenarios, taking into account both high adaptability and overall harvesting efficiency; combined with gravity sensors and automatic transfer, it reduces the fruit damage rate and achieves intelligent and highly robust fruit harvesting.

[0042] In at least one embodiment, such as Figure 5 As shown, fruit picking methods also include: S6. The harvesting robot acquires RGB and depth images of most tube layers in the harvesting area of ​​each of the visual positioning devices 3; S7. Determine whether the acquired RGB image and depth image are clear; If the image is clear, locate the first fruit to be picked in the unobstructed scene of each area, and control the corresponding robotic arm 42 to pick it. If the image is not clear, pick it again. S8. Determine whether the fruit has been picked by acquiring the gravity sensor data on the corresponding robotic arm 42; S9. If fruit is harvested, obtain fruit information and control the robotic arm 42 to place it into a suitable fruit storage and transport device 5.

[0043] This method enables multiple robotic arms to operate independently and in parallel in unobstructed, conventional environments, eliminating the need for complex 3D reconstruction and reinforcement learning planning, thus significantly improving overall harvesting efficiency. Simultaneously, dual verification via image sharpness assessment and gravity sensors ensures the reliability and accuracy of the harvesting actions, reducing missed and incorrect harvests. Combining fruit information with appropriate basket allocation further reduces the risk of fruit damage, enabling the robot to operate efficiently in unstructured orchard environments while maintaining a high success rate and fruit quality.

[0044] In at least one embodiment, such as Figure 7 As shown, the fruit picking method further includes: Obtaining the harvesting task means completing the initialization and task allocation preparation of the harvesting robot control system; and the harvesting robot can autonomously locate and move. During positioning and movement, the robot automatically plans the optimal autonomous walking path based on the orchard's geographical coordinates and the distribution of fruit trees, combined with the robot's GPS positioning system. It then activates the autonomous walking device, driving the motor to rotate the tracks, allowing the robot to move autonomously along the planned path. When the robot approaches the target fruit tree, it calculates the precise relative position between the robot and the tree using integrated sensor data, and controls the autonomous walking device to make fine adjustments, ensuring the robot reaches the most suitable position for harvesting.

[0045] Determine if there is fruit obstruction during harvesting; for fruits in complex obstruction scenarios, perform multi-robotic arm collaborative harvesting operations, executing steps S1 to S5; for fruits in unobstructed scenarios, perform independent harvesting operations in designated areas, executing steps S6 to S9, which can improve the efficiency of fruit harvesting; then perform fruit collection and transfer, continuous monitoring and adjustment, until the task is completed and the robot returns. The robot autonomously walks to the starting point or the designated parking position according to the preset return path.

[0046] During the harvesting process, when collecting and storing the fruit, after the robotic arm completes the harvesting task of a single fruit, it automatically places the fruit into a nearby fruit basket. When the weight sensors in the three fruit baskets detect that the baskets are about to be full, the intelligent control device will send a signal to notify the worker to control the transport vehicle to come to the back of the harvesting robot. Through the transfer of the conveyor belt, the fruit baskets filled with fruit are loaded onto the transport vehicle and replaced with empty fruit baskets.

[0047] In at least one embodiment, in step S3, after a fruit is picked, the distance L that moves backward from the current coordinate position is 30-60cm, so as to leave enough moving distance for the next picking and ensure accurate picking next time. In step S5, the robotic arm transfers the fruit to the nearest fruit basket at a distance H of 8-15cm, leaving enough space for the next harvesting step to facilitate better harvesting.

[0048] The embodiments disclosed herein are preferred embodiments, but are not limited thereto. Those skilled in the art can readily grasp the spirit of the present invention based on the above embodiments and make different extensions and variations, but as long as they do not depart from the spirit of the present invention, they are all within the protection scope of the present invention.

Claims

1. A harvesting robot based on a multi-arm partitioned layout and collaboration, comprising a self-propelled device (1) and a vehicle body support device (2) connected thereto, characterized in that, The fruit-picking robot also has the following features: The robotic arm picking device (4) has at least three robotic arms (42), which are distributed at intervals along the top two sides of the vehicle body support device (2), and each robotic arm (42) has a gravity sensor. as well as Fruit storage and transportation device (5), which is placed on top of vehicle body support device (2) to transport picked fruits; as well as The number of visual positioning devices (3) is set in accordance with the number of the execution robotic arms (42), and is fixed on the vehicle body support device (2) at a position that can obtain the corresponding view of the execution robotic arms (42); and The intelligent control device (6) connects to and controls the self-propelled device (1), the visual positioning device (3), the robotic arm picking device (4), and the fruit storage and transportation device (5); the intelligent control device (6) controls the robotic arm (42) on the same side of the vehicle body support device (2) to simultaneously collect fruit image data from different regional perspectives to obtain a three-dimensional model of the fruit, so as to construct a three-dimensional model of the fruit in the occluded area, realize virtual modeling of the fruit in the occluded scene, and guide the picking task of the robotic arm (42).

2. The harvesting robot based on multi-arm partitioned layout and collaboration according to claim 1, characterized in that, The robotic arm harvesting device (4) also includes: Screw rail (41) fixed on top of fruit storage and transportation device (5); And a sliding actuator (42) connected to a lead screw guide (41), the actuator (42) also being provided with at least a flexible gripper (421).

3. The harvesting robot based on multi-arm partitioned layout and collaboration according to claim 2, characterized in that: The fruit storage and transportation device (5) includes a conveyor belt (53) and a fruit basket (51) placed thereon, the fruit basket (51) being provided with shock-absorbing sponge (52).

4. The harvesting robot based on multi-arm partitioned layout and collaboration according to claim 3, characterized in that: The visual positioning device (3) is an RGB-D camera.

5. The harvesting robot based on multi-arm partitioned layout and collaboration according to claim 4, characterized in that: The vehicle body support device (2) is made of aluminum alloy sheet (21), and an intelligent control cabinet door (22) and heat dissipation ventilation holes (23) are provided on the aluminum alloy sheet (21).

6. The harvesting robot based on multi-arm partitioned layout and collaboration according to claim 5, characterized in that: The number of the robotic arms (42) is six, and the aluminum alloy plate (21) is also equipped with a bracket (24) on the side. The autonomous walking device (1) also includes tracks (11), a high-performance battery (12), a drive motor (13), a GPS (14), a lidar (15), and a vision camera (16).

7. A multi-robotic arm collaborative fruit picking method, characterized by: This includes fruit picking based on the picking robot described in any one of claims 1-6, and includes the following steps: S1. When fruit is detected in a complex occlusion scene, the canopy point cloud data of the visual positioning device (3) on the same side as the vehicle support device (2) is obtained, and a three-dimensional scene model of the fruit is generated through multi-view fusion reconstruction technology. S2. Use the Mask R-CNN semantic segmentation algorithm to identify the occluded fruit, combine the geometric features of branches and leaves to calculate the optimal interference path, and determine the initial clamping point for branch and leaf movement. S3. Based on the Q-learning reinforcement learning algorithm, plan the motion trajectory of the same-side execution robotic arm (42) to control the execution robotic arm (42) to perform precise fruit peeling operation; S4. When the fruit is fully exposed, the positioning information of the first fruit to be picked is transmitted to the execution robotic arm (42), and the execution robotic arm (42) is controlled to grab and rotate to grab the fruit. After completion, it moves back a distance L from the current coordinate position. S5. Determine whether the fruit was successfully picked using the gravity sensor on the robotic arm; If so, the robotic arm will transfer the fruit to a distance H directly above the nearest fruit basket, and then the end effector will release, allowing the transport basket to harvest the fruit; Repeat steps S1 to S5 until the occluded fruit in the 3D scene model of the fruit is completed.

8. The multi-robotic arm collaborative fruit picking method according to claim 7, characterized in that, Fruit picking methods also include: S6. The picking robot acquires RGB and depth images of most tube layers in the picking area of ​​each visual positioning device (3); S7. Determine whether the acquired RGB image and depth image are clear; If clear, locate the first fruit to be picked in the unobstructed scene of each area, and control the corresponding robotic arm (42) to pick it; if unclear, pick it again. S8. By acquiring the gravity sensor data on the corresponding execution robotic arm (42), determine whether the fruit has been picked; S9. If fruit is harvested, obtain fruit information and control the execution robotic arm (42) to place it into a suitable fruit storage and transportation device (5).

9. The multi-robotic arm collaborative fruit picking method according to claim 7, characterized in that: The fruit harvesting method also includes: It receives harvesting tasks and can locate and move around autonomously; It also determines whether there is fruit obstructing the view during harvesting; if there is obstruction, proceed to steps S1 to S5; if there is no obstruction, proceed to steps S6 to S9.

10. The multi-robotic arm collaborative fruit picking method according to claim 9, characterized in that: In step S3, after a fruit is picked, the distance L that moves backward from the current coordinate position is 30-60cm; in step S5, the robotic arm transfers the fruit to the nearest fruit basket at a distance H of 8-15cm.

Images