A collaborative mushroom harvesting humanoid robot system

Abstract

The present application provides a kind of collaborative work mushroom harvesting humanoid robot system, including: vision system, central controller, mobile chassis, lifting mechanism, operating platform and setting on the picking SCARA arm of operating platform, collaborative processing mechanical arm, rotary cutting assembly, collection device;Central controller according to the information collected by vision system cooperatively controls the working state of other modules;Mobile chassis and lifting mechanism are used to adjust the position of operating platform;Picking SCARA arm is used for picking mushroom and handover to collaborative processing mechanical arm;Collaborative processing mechanical arm is used for holding mushroom and cooperating with rotary cutting assembly to cut root;Central controller determines mushroom quality classification according to the information collected by vision system, and controls picking SCARA arm and / or collaborative processing mechanical arm movement and puts mushroom into the corresponding category container of collection device.The present application distributes picking and fine processing to independent picking SCARA arm and collaborative processing mechanical arm, interacts through "collaborative handover point", and realizes the order of magnitude improvement of harvesting throughput.

Description

Technical Field

[0001] This invention relates to the field of harvesting robot technology, and in particular to a collaborative mushroom harvesting humanoid robot system. Background Technology

[0002] Edible fungi are important consumer products, and consumers have extremely high requirements for the uniformity of fresh mushroom specifications and the absence of damage to their appearance, making the harvesting process a key bottleneck that determines the value of the product.

[0003] In current industrialized production of edible fungi, modern mushroom houses generally adopt the "Dutch shelving system" structure. For example... Figure 1 As shown, the structure mainly consists of vertically arranged multi-layer bed frames ( Figure 1 (Only two layers are shown). The bed frame width is typically 1.2 to 1.5 meters, with narrow spaces between layers (usually about 28-40 centimeters), and the environment is characterized by high humidity and constant temperature. Mushroom harvesting is a crucial step in determining its commercial value and production cost. To address the many drawbacks of manual harvesting, automated harvesting technology has been continuously developing, but existing mainstream technologies all have significant limitations that are difficult to balance.

[0004] 1. Fixed structure or partially mobile equipment: The harvesting mechanism moves on a fixed frame, lacks overall mobility, cannot autonomously cover multiple beds in the entire mushroom house, has a limited degree of automation, and is essentially still a semi-automated workstation, which cannot meet the needs of large-scale factories for fully automated inspection and harvesting.

[0005] 2. Rail-mounted or rack-mounted mobile robots: These devices represent a high level of automation. For example, CN202111151206 and CN202511458618 both utilize rails on the side of the modified mushroom bed, along which the robot moves. The US 4AG patent (US 2025 / 0176481 A1) is a prime example, employing a gantry-type vertical carriage with SCARA arms, a clever design that solves the problem of achieving large-area horizontal coverage in narrow spaces. However, a common drawback of these solutions is the need for rail installation and infrastructure modifications in existing mushroom houses, resulting in high initial investment, poor deployment flexibility, and maintenance issues with the rail system in long-term high-humidity environments. Furthermore, these devices typically rely heavily on a pre-set rail coordinate system, making cross-bed and cross-area operation scheduling still quite complex.

[0006] 3. Ground-based Autonomous Mobile Robots: The applicant, Qiwu Technology's first-generation robot patent (CN 121195787 A), utilizes a mobile chassis, demonstrating good spatial freedom and ease of deployment. However, its six-axis arm suffers from limited range of motion and slow speed; the processing system faces challenges such as difficulty in mushroom placement, low throughput, and susceptibility to mushroom damage. Other similar mobile robot solutions in the industry, such as CN202511458618, also exhibit issues such as relatively crude subsequent processing methods and difficulty in controlling root cutting and damage.

[0007] In summary, the field of automated mushroom harvesting currently faces a systemic challenge: how to deploy a highly integrated and intelligent autonomous mobile robot system without modifying existing mushroom house infrastructure. This system needs to move freely between complex multi-layered shelves, like a highly skilled worker, and collaboratively complete a series of delicate operations, including rapid identification, non-destructive grasping, precise transfer, high-quality root cutting, and real-time sorting, while ensuring extremely high operational efficiency and reliability (i.e., low damage rate). Existing technological solutions all have varying degrees of deficiencies in balancing the four core requirements of deployment flexibility, functional completeness, operational efficiency, and harvesting quality.

[0008] To address, or at least partially address, the performance deficiencies of existing mushroom harvesting robots in terms of deployment flexibility, functional completeness, operational efficiency, and harvesting quality, this invention provides a collaborative mushroom harvesting humanoid robot system with higher overall performance. Summary of the Invention

[0009] This invention provides a collaborative mushroom harvesting humanoid robot system with higher overall performance, which solves or at least partially solves the performance defects of existing mushroom harvesting robots in terms of deployment flexibility, functional completeness, operational efficiency and harvesting quality.

[0010] This invention provides a collaborative mushroom harvesting humanoid robot system, comprising: a vision system, a central controller, a mobile chassis, a lifting mechanism, an operating platform, and a harvesting SCARA arm, a collaborative processing robotic arm, a rotary cutting assembly, and a collection device mounted on the operating platform;

[0011] The central controller coordinates and controls the working status of other modules based on the information collected by the vision system.

[0012] The mobile chassis and the lifting mechanism are used to adjust the position of the operating platform;

[0013] The SCARA harvesting arm is used to harvest mushrooms and hand them over to the collaborative processing robotic arm;

[0014] The collaborative processing robotic arm is used to grip the mushroom and work with the rotary cutting component to cut the root;

[0015] The central controller determines the mushroom quality classification based on the information collected by the vision system, and controls the movement of the picking SCARA arm and / or the collaborative processing robotic arm to place the mushrooms into the corresponding category containers of the collection device.

[0016] Optionally, the top of the lifting mechanism is equipped with a lidar, which can collect environmental information to assist in obstacle avoidance.

[0017] Optionally, the harvesting SCARA arm is a three-axis or four-axis SCARA arm, with degrees of freedom for adjustment in the height direction and in the horizontal plane. The end of the harvesting SCARA arm is provided with a silicone suction cup for adsorbing mushrooms.

[0018] Furthermore, the silicone suction cup is funnel-shaped, including a tubular handle and a conical suction part. The tubular handle is used to be installed at the end of the harvesting SCARA arm, and the outer surface of the tubular handle is provided with a flexible reinforcement structure.

[0019] Optionally, the harvesting SCARA arm is also equipped with a lighting module that can provide illumination during the harvesting process.

[0020] Optionally, the collaborative processing robotic arm includes a lifting module, a rotary table, and a two-finger gripper. The lifting module adjusts the height of the rotary table and the two-finger gripper, and the rotary table adjusts the orientation of the two-finger gripper.

[0021] Furthermore, the two-finger gripper is capable of gripping and releasing the mushroom. The part of the two-finger gripper that grips the mushroom body is bowl-shaped, and its bottom is provided with a hole for the mushroom root to pass through.

[0022] Optionally, the system includes two or more sets of SCARA harvesting arms and collaborative processing robotic arms; the same set of SCARA harvesting arms and collaborative processing robotic arms hand over mushrooms at a fixed position, or the same set of SCARA harvesting arms and collaborative processing robotic arms adaptively selects a position to hand over mushrooms according to the rhythm.

[0023] Optionally, the rotary cutting assembly includes a drive motor and a cutter;

[0024] After the collaborative processing robotic arm holds the mushroom to be cut at the cutting point, the drive motor drives the cutter to rotate and cut off the mushroom root.

[0025] Optionally, the collection device includes at least three containers, which are distributed in the same layer or different layers; each container is equipped with a weighing sensor at the bottom, which can measure the weight of the mushrooms in the container.

[0026] Optionally, the vision system includes:

[0027] A visual sensor is used to collect video data of the mushroom bed area to determine the current harvesting status.

[0028] A depth vision camera mounted on the harvesting SCARA arm is used to acquire three-dimensional information about the mushroom.

[0029] A first high-definition camera is used to capture images of the mushrooms harvested by the SCARA arm.

[0030] A second high-definition camera is used to capture images of the mushrooms after they have been cut and are held by the collaborative processing robotic arm.

[0031] Furthermore, the central controller controls the harvesting SCARA arm to harvest mushrooms based on the three-dimensional information of the mushrooms collected by the visual sensor; the central controller also identifies abnormal mushrooms based on the mushroom image information collected by the visual sensor, and controls the harvesting SCARA arm to harvest the abnormal mushrooms and put them into the waste container.

[0032] Optionally, the central controller identifies abnormal mushrooms based on images captured by the first high-definition camera and controls the harvesting SCARA arm to place the abnormal mushrooms into a waste container, or controls the harvesting SCARA arm to transfer the abnormal mushrooms to the collaborative processing robotic arm, which then places the abnormal mushrooms into the waste container.

[0033] Optionally, the central controller identifies abnormal mushrooms based on images captured by the second high-definition camera, and the collaborative processing robotic arm places the abnormal mushrooms into a waste container. If no abnormal mushrooms are identified, the visual data collected by the depth vision camera, the first high-definition camera, and the second high-definition camera are combined to classify the quality of the mushrooms and place them into containers of the corresponding category in the collection device.

[0034] The collaborative mushroom harvesting humanoid robot system provided by this invention has at least the following beneficial effects:

[0035] 1. No need to modify the mushroom house

[0036] The mobile chassis and lifting mechanism allow for flexible adjustment of the operating platform's position, precisely raising and lowering the work surface to any target bed layer without requiring any modifications to existing Dutch-style mushroom houses' tracks or infrastructure. Integrated SLAM navigation enables the robot to move autonomously and avoid obstacles within the mushroom house's passageways, much like a human. This fundamentally solves the problems of high deployment costs and poor flexibility associated with fixed, track-based solutions, achieving a wide-area automated harvesting capability of "one device covering the entire house."

[0037] 2. Automated assembly line operations mimicking human hand coordination achieve a dual improvement in throughput and quality.

[0038] The "harvesting" and "refining" functions are assigned to independent "harvesting SCARA arms" and "cooperative processing robotic arms," ​​which physically interact through "cooperative handover points" and work in parallel under the scheduling of a central controller. Once the harvesting arm completes the handover, it can return to start the next harvest, and its work cycle is not affected by the time consumed by subsequent root cutting and sorting.

[0039] The mushrooms are subjected to only two controllable and flexible forces throughout the entire processing: the first is the non-destructive gripping and separation achieved by a flexible silicone suction cup; the second is the virtual fixation in a static state by the cup-shaped two-finger gripper of the collaborative processing arm, and the precise cutting is completed by the active rotating cutter, ensuring a flat cut surface and avoiding physical damage caused by conveying and secondary gripping in traditional solutions.

[0040] The aforementioned parallel pipeline architecture means that the overall efficiency of the system is determined by the slower of the "harvesting speed" and "processing speed," rather than the sum of the two. In the preferred scheme, the handover point of the mushrooms (including height and position on the horizontal plane) is adaptively adjusted according to the harvesting rhythm of the SCARA arm and the collaborative processing arm. This allows the utilization rate of the collaborative processing arm to approach saturation, thereby achieving an order-of-magnitude increase in the overall harvesting throughput of the system while ensuring an ultra-low damage rate.

[0041] 3. High-speed, high-precision, blind-spot-free intensive harvesting

[0042] Employing a SCARA arm optimized for horizontal operations, the time for a single "identify-harvest-transfer" cycle is significantly reduced. Through a "forward / reverse folding" joint configuration strategy and real-time collision detection, the SCARA arm can dynamically bypass mushroom bed supports, ensuring 100% coverage of the standard workspace. Front-end perception uses a unified deep learning vision model, simultaneously completing mushroom detection, key point (grasping point) localization, and disease identification in a single scan, with a latency of less than 30 milliseconds. At its core, the system, through a harvesting algorithm based on pickability assessment and dynamic programming, can perform real-time analysis of dense mushroom clusters, automatically planning a low-damage, high-efficiency harvesting sequence, effectively solving the industry challenges of "crushing damage" and "harvesting losses."

[0043] 4. Closed-loop intelligent sorting and refined operation management

[0044] By using a cascaded vision system consisting of a depth vision camera at the end of the SCARA arm, a first small high-definition camera, and a second small high-definition camera, mushrooms are subjected to a three-level closed-loop inspection: "preliminary screening - handover verification - final quality control." This significantly reduces the false judgment rate and ensures that only mushrooms that meet the predetermined standards in the "harvesting settings" enter the corresponding grade of collection container.

[0045] Integrated weighing sensors enable real-time weight monitoring of the collection containers. Combined with a central controller, this can automatically trigger basket-changing reminders or pause operations, achieving unmanned material management. All processes operate based on configurable "harvesting settings," adapting to the harvesting needs of different mushroom varieties and commercial standards.

[0046] 5. A data-driven and continuously evolving intelligent decision-making core

[0047] The system's further innovation lies in its evolvable intelligent decision-making capabilities. In this further development, the central controller does not run a fixed program but instead deploys a reinforcement learning agent based on an embodied large model. This system offers dual advantages: First, it can directly understand natural language commands such as "harvesting settings" and generate optimal action sequences end-to-end based on real-time visual input, demonstrating strong scene generalization and task understanding capabilities. More importantly, the system possesses online learning and continuous optimization capabilities: the multimodal data (images, actions, results) generated during operation, the structured rewards automatically converted from three levels of visual feedback, and the post-harvest human evaluation feedback together constitute a continuous reinforcement learning loop, driving the decision-making model to continuously self-calibrate and evolve. This enables the robot to adapt to new production environments, mushroom varieties, and quality standards, consistently maintaining optimal operational performance and achieving a leap from "automation" to "autonomous intelligence."

[0048] In summary, this invention, through a three-layer integrated design of a "flexible mobile platform + humanoid dual-robotic arm collaborative system + data-driven self-evolving intelligent kernel," successfully achieves an excellent balance between the four core requirements of deployment flexibility, functional completeness, operational efficiency, and high-quality harvesting. This system is not only an automated device but also an intelligent harvesting unit that can adapt to complex mushroom house environments, simulate optimal human decision-making, and achieve hand-eye coordination, providing a brand-new, fully automated solution for factory-scale mushroom production. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram of a Dutch multi-tiered bed frame in the prior art;

[0051] Figure 2 This is one of the structural schematic diagrams of a collaborative mushroom harvesting humanoid robot system according to the present invention;

[0052] Figure 3 This is a schematic diagram of the rear structure of a collaborative mushroom harvesting humanoid robot system according to the present invention;

[0053] Figure 4 This is a schematic diagram of the rear side structure of a collaborative mushroom harvesting humanoid robot system according to the present invention;

[0054] Figure 5 This is a schematic diagram of the harvesting SCARA arm of a collaborative mushroom harvesting humanoid robot system according to the present invention;

[0055] Figure 6 This is a schematic diagram of the silicone suction cup of a collaborative mushroom harvesting humanoid robot system according to the present invention;

[0056] Figure 7 This is a schematic diagram of the structure of the collaborative processing robotic arm of a humanoid robot system for collaborative mushroom harvesting according to the present invention;

[0057] Figure 8 This is a schematic diagram of the rotary cutting component of a collaborative mushroom harvesting humanoid robot system according to the present invention;

[0058] Figure 9 This is a schematic diagram of the collection device of a collaborative mushroom harvesting humanoid robot system according to the present invention;

[0059] Figure 10 This is a schematic diagram of the rear structure of the collection device of a collaborative mushroom harvesting humanoid robot system according to the present invention;

[0060] Figure 11 This is a schematic diagram of the workspace of a collaborative mushroom harvesting humanoid robot system according to the present invention;

[0061] Figure 12 This is a schematic diagram of the harvestable range of the SCARA arm joint in the forward-folded state of a collaborative mushroom harvesting humanoid robot system according to the present invention;

[0062] Figure 13 This is a schematic diagram of the harvestable range of a SCARA arm in a collaborative mushroom harvesting humanoid robot system of the present invention, with the joints in a retracted state.

[0063] Figure 14 This is a schematic diagram of the structure of an expert data acquisition system according to the present invention;

[0064] Figure 15 This is a schematic diagram showing the field of view of the camera at a series of points on the depth vision arm within the workspace.

[0065] Figure label:

[0066] 100-Mobile chassis, 101-LiDAR, 102-Vision sensor, 200-Lifting mechanism, 210-Operating platform; 300-SCARA harvesting arm, 310-Depth vision sensor, 320-Silicone suction cup, 321-Adsorption part, 322-Tube-shaped handle, 330-Lighting module; 341-Workspace, 342-Column, 343-Blind zone, 344-Motion boundary of the robotic arm end effector, 345-Field of view of the first high-definition camera at each point, 343- The workspace that the robotic arm should reach; 400 - Collaborative processing robotic arm, 410 - Two-finger gripper, 420 - Rotary table, 430 - Lifting module; 440 - Rotary cutting assembly, 441 - Drive motor, 442 - Cutter, 451 - First high-definition camera, 452 - Second high-definition camera; 500 - Collection device, 501 - Collection container, 502 - Collection container, 503 - Mushroom root collection container, 505 - Waste container, 504 - Weighing sensor; 600 - Central controller. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0068] The following is combined Figures 2-15 This invention describes a collaborative mushroom harvesting humanoid robot system.

[0069] Reference Figure 2-15 The present invention provides a collaborative mushroom harvesting humanoid robot system, which includes: a vision system, a central controller 600, a mobile chassis 100, a lifting mechanism 200, an operating table 210, and a harvesting SCARA arm 300, a collaborative processing robotic arm 400, a rotary cutting component 440, and a collection device 500 set on the operating table.

[0070] The central controller 600 coordinates and controls the working status of other modules based on the information collected by the vision system.

[0071] The movable chassis 100 and the lifting mechanism 200 are used to adjust the position of the operating platform 210;

[0072] The SCARA arm 300 is used to pick mushrooms and then hand them over to the collaborative processing robotic arm 400.

[0073] The collaborative processing robotic arm 400 is used to grip mushrooms and work with the rotary cutting component 440 to cut the roots;

[0074] The central controller 600 determines the quality classification of mushrooms based on the information collected by the vision system, and controls the movement of the picking SCARA arm 300 and / or the co-processing robotic arm 400 to place the mushrooms into the corresponding category containers of the collection device 500.

[0075] Specifically, refer to Figure 2 , Figure 3 The mobile chassis 100 is driven by wheels or tracks, enabling the robot system to move on the ground. The lifting mechanism 200 (such as a high-rigidity screw module) is fixedly mounted on the mobile chassis 100, and its operating platform 210 carries all subsequent work modules. By controlling the lifting mechanism 200, the entire work surface can be precisely positioned to any layer of the target bed frame, adapting to different layer heights of the "Dutch" bed frame without requiring any track modifications to the existing mushroom house. The operating platform is equipped with modules such as a harvesting SCARA arm, a collaborative processing robotic arm, a rotary cutting component, and a collection device. It should be noted that the humanoid robot system of this invention may include one harvesting SCARA arm, two harvesting SCARA arms, or more harvesting SCARA arms.

[0076] Reference Figure 4 The central controller 600 is electrically connected to all sensors and actuators. It is programmed to execute core coordination logic: while controlling the picking SCARA arm 300 to complete the "identification-picking-transfer to handover point" cycle, it simultaneously controls the collaborative processing robotic arm 400 to perform a serial process cycle of "receiving-root cutting-visual sorting-movement to corresponding container delivery". The central controller 600 precisely plans the timing of both, ensuring that the picking SCARA arm 300 delivers the mushroom precisely when the two-finger gripper 410 of the collaborative processing robotic arm 400 is in the receiving-ready position; and ensuring no interference between the two arms during operation.

[0077] Reference Figure 2 Based on the previous embodiment, in some embodiments, a lidar 101 is installed on the top of the lifting mechanism 200, which can collect environmental information, sense obstacles, and assist the robot in obstacle avoidance. Furthermore, the system also integrates a SLAM (Simultaneous Localization and Mapping) navigation system, which can sense, record, and update the environmental map to achieve precise positioning and movement path planning.

[0078] Reference Figure 5Based on the previous embodiment, in some embodiments, the picking SCARA arm 300 is mounted on the operating table 210. The number of SCARA arms can be one, two, or more. The picking SCARA arm is a three-axis or four-axis SCARA arm, preferably a three-axis configuration, possessing degrees of freedom for height adjustment and horizontal adjustment. It features a large working range, high speed, and high repeatability in the horizontal plane, making it particularly suitable for performing rapid reciprocating movements between dense "picking points" and "intersection points" within the two-dimensional plane of the bed frame. Its cycle time is significantly better than that of a six-axis arm, which requires coordinating multiple joints in a confined space. A silicone suction cup 320 is installed on the flange at the end of the picking SCARA arm for adsorbing mushrooms.

[0079] Further reference Figure 6 The silicone suction cup 320 is funnel-shaped, comprising a tubular handle 322 and a conical suction part 321. The suction part 321 is made of low-hardness silicone, which can adaptively wrap around mushroom caps of different sizes and angles, achieving non-destructive suction. The tubular handle 322 is used to install onto the end flange of the harvesting SCARA arm 300. Its relatively large thickness ensures sufficient longitudinal rigidity during suction and harvesting, preventing excessive axial deformation of the suction cup during mushroom adsorption. Simultaneously, the outer surface of the tubular handle 322 features flexible reinforcement structures, such as spiral grooves or circumferential grooves of a certain depth, ensuring that the suction cup does not become too rigid and detach from the mushroom body or damage it during the bending and harvesting process with the mushroom root as the center point. This design achieves a balance between flexibility during suction and necessary rigidity during harvesting through physical structure. The tubular handle 322 of the suction cup is connected via an air pipe to an air pump located in the rear chassis of the robot, which can provide both positive and negative pressure. When sucking up mushrooms, negative pressure is activated; when releasing mushrooms, positive pressure is activated. Simultaneously, a pressure sensor is connected to the air circuit; the negative pressure value from the sensor indicates the mushroom sucking status.

[0080] Still refer to Figure 5 Based on the foregoing embodiments, in some embodiments, the harvesting SCARA arm 300 is also equipped with a lighting module 330. Preferably, it is located closer to the end of the arm segment and faces downwards (it can be a strip lighting unit) to illuminate the mushrooms in the mushroom bed. A bright environment helps improve the perception accuracy of the depth camera and enhances the clarity of the images captured by the camera. At the same time, the lighting module 330 enables the robot to work in the dark conditions of the factory.

[0081] Reference Figure 2 , Figure 7Based on the foregoing embodiments, in some embodiments, the base of the collaborative processing robotic arm 400 is fixedly mounted on the operating table 210, located near the workspace of the picking SCARA arm 300, and has 3-4 degrees of freedom. The collaborative processing robotic arm 400 can be a compact SCARA arm or a Delta arm. Taking a composite robotic arm that moves in a vertical and horizontal fan-shaped pattern as an example, the collaborative processing robotic arm 400 includes a lifting module 430, a rotary table 420, and a two-finger gripper 410. The lifting module 430 adjusts the height of the rotary table 420 and the two-finger gripper 410, and the rotary table 420 adjusts the orientation of the two-finger gripper 410.

[0082] Furthermore, the two-finger gripper can grip and release mushrooms. The part of the two-finger gripper that grips the mushroom body is bowl-shaped, which improves the gripping adaptability to tilted mushrooms. The bottom of the gripper is provided with a hole for the mushroom root to pass through, so as to stabilize the root without damaging the mushroom root.

[0083] Reference Figures 2-4 Based on the foregoing embodiments, in some embodiments, the system includes two or more sets of harvesting SCARA arms and collaborative processing robotic arms. Preferably, two sets of harvesting SCARA arms and collaborative processing robotic arms are used, and the two sets of harvesting SCARA arms and collaborative processing robotic arms can be arranged symmetrically. Furthermore, in some implementations, the same set of harvesting SCARA arms and collaborative processing robotic arms hand over the mushroom at a fixed position to improve control stability and reduce control difficulty. Considering that the movement rhythm of the SCARA arm may vary depending on the specific location of mushroom picking, and that the time required to place the mushrooms into different types of containers may also vary depending on the type of mushroom, the movement rhythm of the collaborative processing robot arm may also vary. Therefore, in some implementations, the same group of SCARA arms and collaborative processing robots adaptively select positions to hand over mushrooms based on their rhythm. For example, if the collaborative processing robot arm has just finished cutting the root of a mushroom and is about to sort and place it, it will take 3 seconds to return to the default handover position. If the SCARA arm has finished picking a mushroom and it will take 2 seconds to move to the default handover position, a new handover position closer to the collaborative processing robot arm can be determined based on the movement trajectory of the collaborative processing robot arm to improve the picking rhythm and picking efficiency.

[0084] Reference Figure 8 Based on the foregoing embodiments, in some embodiments, the rotary cutting assembly 440 includes a drive motor 441 and a cutter 442; after the collaborative processing robotic arm 400 holds the mushroom to be cut at the cutting point, the drive motor 441 drives the cutter 442 to rotate and cut off the mushroom root.

[0085] Specifically, the cutter 442, driven by the drive motor 441, moves in a horizontal fan-shaped motion. The cutter has a retractable position and a root-cutting position (the cutter's rotation plane covers the point where the collaborative processing robotic arm 400 holds the mushroom and awaits root cutting). The collaborative processing robotic arm 400 first positions itself at the junction point. After receiving the mushroom, its two-finger gripper 410 closes to hold the mushroom. The rotary table 420 moves the mushroom to the root-cutting position, activating the rotary cutting assembly 440 to cut the mushroom root. The micro-motor drives the cutter to rotate and cleanly cut off the root. During this process, the mushroom is loosely fixed by the gripper to prevent damage from clamping marks. By adjusting the appropriate cutter speed, the stability of the cutting action and the smoothness and perfection of the cut surface are ensured. Figure 8 The diagram shows the blade groove. When the cutter 442 is inside the groove, it is in the retracted position. When the cutter 442 leaves the groove, it cuts the mushroom root. After cutting, it retracts into the groove, improving safety. Of course, this design can also be modified to change the blade direction, cutting the mushroom root when returning to the groove.

[0086] Those skilled in the art will understand from this invention that the positions of the SCARA harvesting arm, the collaborative processing robotic arm, the rotary cutting component, and the collection device on the operating table can be flexibly arranged so that the robot can smoothly complete steps such as harvesting, handing over, cutting roots, and sorting and placing. Specifically, the mushroom handing point (i.e., the handing point mentioned above) is within the reachable range of the SCARA harvesting arm and the collaborative processing robotic arm, the root cutting point is within the reachable range of the collaborative processing robotic arm and within the cutting plane of the rotary cutting component, and the collection device is within the reachable range of the SCARA harvesting arm and the collaborative processing robotic arm.

[0087] Reference Figure 9 Based on the foregoing embodiments, in some embodiments, the collecting device 500 includes at least three containers, each distributed in the same layer or different layers; each container is provided with a weighing sensor at the bottom, which can measure the weight of the mushrooms in the container.

[0088] Specifically, the collection device 500 is fixedly installed above the operating table 210, within the working range of the collaborative processing robotic arm 400. It includes at least two collection containers 501 and 502 positioned at different heights; for example, the upper container collects Grade A mushrooms, and the lower container collects Grade B mushrooms (Grade A and Grade B mushrooms have different quality grades, such as differences in cap integrity, damage, and cut surface quality). The mushroom root collection container 503 has a baffle above it to block mushroom roots that are cut off by the active blade, forming an irregularly shaped container with a rear baffle and left and right guide grooves. A door panel is located below it, which can be manually opened to allow the mushroom roots to fall out due to gravity. A waste container 505 is located below to collect abnormal mushrooms identified during the visual sorting stage. A weighing sensor 504 measures the weight of the mushrooms in each collection container and sends the information to the central controller.

[0089] in addition, Figure 9 This illustration only shows the containers distributed across different layers. This solution requires structural design, including container supports, which increases the data collection capacity. Alternatively, the number of containers could be reduced, and they could be placed directly on the workbench.

[0090] Reference Figure 2 , Figure 5 , Figure 8 Based on the foregoing embodiments, in some embodiments, the vision system includes a vision sensor 102, a depth vision camera 310, a first high-definition camera 451, and a second high-definition camera 452.

[0091] The visual sensor 102 is used to collect video data of the mushroom bed area to determine the current harvesting situation, that is, to determine whether there are enough mushrooms to be harvested in the current mushroom bed area, and whether harvesting is necessary. The video sensor 102 can also be used to assist in collecting data for training the embodied large model (see below for details on the embodied large model).

[0092] A depth vision camera 310, such as an RGB-D camera, is mounted on the harvesting SCARA arm 300, preferably on the arm segment near the end. The depth vision camera 310 can accurately acquire the three-dimensional position, size, and posture information of the mushrooms below, facilitating accurate mushroom harvesting. It can also make a preliminary judgment on the quality of the mushrooms, identify abnormal mushrooms, and issue instructions to the harvesting SCARA arm 300 to discard the abnormal mushrooms directly into the waste container.

[0093] The placement of the first high-definition camera 451 and the second high-definition camera 452 can be flexibly configured. For example, they can be placed on the support column of the SCARA harvesting arm 300, the support column of the collaborative processing robotic arm 400, on the rotary cutting assembly, or on a dedicated bracket / mounting position. The first high-definition camera 451 is used to capture images of the mushrooms harvested by the SCARA harvesting arm to determine their quality. Preferably, the first high-definition camera 451 is placed near the location where the mushrooms are harvested by the SCARA harvesting arm 300 and the collaborative processing robotic arm 400 (e.g., near the junction of the SCARA harvesting arm 300 and the collaborative processing robotic arm 400). Figure 8 (Shown below the rotary cutting component 440) to capture images more clearly at close range and also avoid affecting the beat.

[0094] The second high-definition camera 452 is used to collect images of mushrooms after root cutting held by the collaborative processing robotic arm, and to determine the quality of the mushrooms after root cutting. Preferably, the second high-definition camera 452 is set near the root cutting position so as to collect images more clearly at close range, and also to avoid affecting the rhythm.

[0095] Based on the previous embodiment, in some embodiments, the central controller 600 controls the harvesting SCARA arm 300 to harvest mushrooms based on the three-dimensional information of the mushrooms collected by the depth vision camera 310; the central controller 600 also identifies abnormal mushrooms based on the mushroom image information collected by the depth vision camera 310, and controls the harvesting SCARA arm to harvest the abnormal mushrooms and put them into a waste container.

[0096] Specifically, the depth vision camera 310 on the harvesting SCARA arm 300 performs real-time detection and preliminary classification of mushrooms before harvesting. The central controller 600 processes the captured images and outputs the mushroom bounding box and its corresponding disease attributes (such as normal, yellow spot disease, mold, insect pests). If a severely diseased or deformed mushroom is detected, the system directly marks it as "abnormal." Depending on the harvesting settings, the system chooses to use the harvesting SCARA arm 300 to collect the mushroom and discard it into the abnormal basket, or not to harvest it at all.

[0097] Based on the foregoing embodiments, in some embodiments, the central controller 600 identifies abnormal mushrooms based on images captured by the first high-definition camera 451 and controls the picking SCARA arm 300 to place the abnormal mushrooms into a waste container, or controls the picking SCARA arm to transfer the abnormal mushrooms to the collaborative processing robotic arm, which then places the abnormal mushrooms into a waste container.

[0098] Specifically, the first high-definition camera can be positioned below the active cutter assembly, with its optical axis pointing towards the direction of the collaborative processing robotic arm 400 when the mushroom is handed over. It is used to take a second picture before the mushroom is received by the gripper to verify the mushroom's shape and integrity. It determines whether the mushroom's posture is stable, whether the sides of the cap are intact, and whether there are any yellow spots or lesions. If any abnormalities are found, the picking SCARA arm 300 directly discards it into the abnormality basket, or places it on the gripper mechanism for direct discarding by the collaborative processing robotic arm 400. In addition, the first high-definition camera 451 also judges the mushroom's posture, cap thickness, and root length to determine the optimal placement location.

[0099] Based on the foregoing embodiments, in some embodiments, the central controller 600 identifies abnormal mushrooms based on images captured by the second high-definition camera 452 and coordinates with the robotic arm 400 to place the abnormal mushrooms into a waste container. If no abnormal mushrooms are identified, the visual data collected by the depth vision camera 310, the first high-definition camera 451, and the second high-definition camera 452 are combined to classify the quality of the mushrooms and place them into containers of the corresponding category in the collection device.

[0100] Specifically, the second high-definition camera 452 is located next to the first high-definition camera 451, with its optical axis vertically upward, directly facing the bottom of the mushroom after root cutting. After root cutting, the clamping turntable rotates above the camera, which immediately takes a picture for quality analysis, judging the final quality indicators such as the smoothness of the cut surface, the integrity of the stem, and the presence of hidden diseases. If the cut surface is uneven, has hidden diseases, or does not meet the morphological standards, it is marked as "substandard" and discarded into the abnormal basket. If it meets the A or B grade standards, its classification is determined based on the comprehensive score of the images acquired by the depth vision sensor 310 at the end of the harvesting SCARA arm, the first high-definition camera 451, and the second high-definition camera 452. The collaborative processing robotic arm 400 then accurately places it into the corresponding container in the collection device 500.

[0101] Based on the collaborative mushroom harvesting humanoid robot system described in the above embodiments, the present invention also proposes algorithm improvements, which are further described below.

[0102] On the one hand, this invention improves the obstacle avoidance algorithm for harvesting arms.

[0103] Figure 11 The working space for harvesting SCARA arms is explained. The "Dutch" mushroom bed has a standard depth and length. A typical configuration is as follows: the entire mushroom bed is 30m long, divided into 20 sections of 1.5m each; the width of the mushroom bed is 1.4m, usually divided into two 0.7m sections on either side of the center of the width. Workers harvest the mushrooms along the length of the mushroom bed from both sides. Figure 11 The diagram illustrates a workspace 341 in the first gear, measuring 1.5m in length and 0.7m in width. A support column 342, supporting the mushroom bed vertically, stands in front of it. With appropriate arm spacing and SCARA arm length settings, a robotic system composed of dual-arm robots can cover the entire work area when stationary only at the center point in the first gear.

[0104] To achieve high-speed continuous movement and avoid collisions with the upright column 342 in front of the mushroom bed, this system adopts an optimized joint configuration and a real-time collision detection strategy.

[0105] 1. Joint antagonistic configuration and motion efficiency optimization

[0106] When the SCARA arm moves in the horizontal plane, the orientation of its second joint (the rotational joint between the forearm and upper arm) directly affects the extension direction and movement efficiency of the arm in the plane. Since the placement point is located in the middle region of the two arms, the following "forward-folding" joint configuration achieves the highest speed and smoothest movement trajectory:

[0107] Left arm: Keep the first joint facing to the left (i.e., the "elbow" of the robotic arm points to the left);

[0108] Right arm: Keep the first joint facing to the right (i.e., the "elbow" of the robotic arm points to the right).

[0109] 2. Movement Reversal Strategy in the Column Area

[0110] However, if the aforementioned "positive hand position" movement continues in the area near the column 342 in front of the mushroom bed, the robotic arm's main arm linkage will collide with the column. For example... Figure 12 As shown, to avoid collisions, the range of motion of the robotic arm is limited, and the final reachable point has a blind spot 343 near the column.

[0111] To address this, the system employs a joint reversal strategy (i.e., changing the orientation of the first joint so that the "elbow" points in the opposite direction) to avoid the column. For example... Figure 13 As shown, after folding back, the reachable site can cover the entire working area.

[0112] The system calculates the joint angles from the current position to the target picking point in real time using inverse kinematics, and determines whether interference with the pillar model (pre-placed in the environment map) will occur in the "positive hand position". If a collision is predicted, it automatically switches to the "negative hand position" (i.e., left arm joint facing right, right arm joint facing left) for movement. Although the movement efficiency is slightly reduced after the arm is folded back, it can safely bypass the pillar area, ensuring that there are no blind spots in the picking coverage.

[0113] 3. Safety margin and real-time collision detection

[0114] The key to the anti-folding strategy is obtaining the angle of the upper arm at the time of collision. In practice, the algorithm can be simplified by assuming the robot is at a fixed position. In this case, the upper arm angle at the time of collision can be directly calculated based on the upper arm width. Simultaneously, based on navigation accuracy, a large static safety margin is set for the actual collision angle to further reduce the risk of collision. Furthermore, a higher-precision detection method can be used to detect the column position online using environmental point cloud data, thereby calculating the collision threshold angle. The central controller performs collision prediction before issuing each motion command to avoid collisions.

[0115] 4. Dynamic path optimization

[0116] Based on the above strategies, the system achieves dynamic path optimization: high-speed "forward" movement is adopted in open areas without pillar obstruction; it automatically switches to "reverse" position to avoid obstacles near pillars; and in the picking sequence planning, picking points that can be completed continuously in "forward" position are prioritized to reduce the number of joint reversals, thereby maximizing the overall picking efficiency while ensuring safety.

[0117] On the other hand, this invention proposes a deeply integrated "intelligent system layer" as the core of the robot's decision-making and control. This intelligent layer comprises two main pillars: 1. an expert-experience-driven data acquisition and rule generation module, used to efficiently execute harvesting tasks and accumulate high-quality data in structured scenarios; 2. an autonomous decision-making and reinforcement learning optimization module based on an embodied large model, capable of understanding natural language instructions and achieving self-iteration and evolution of harvesting strategies based on real-time visual feedback and human feedback. These two pillars work together, enabling the system not only to complete efficient and lossless harvesting tasks but also to possess adaptive and continuous learning capabilities for complex scenarios. The following sections will describe these capabilities in two parts:

[0118] Expert data acquisition system.

[0119] To achieve intelligent data collection and accumulate high-quality decision-making data, this system constructs a closed-loop expert data acquisition system. Its core process is as follows: Figure 14 As shown, this forms a complete closed loop from parameterized task-driven to data-driven experience output.

[0120] The process can be summarized as follows:

[0121] Task configuration and data acquisition: The system generates operation parameters based on the preset "harvesting settings" (harvesting standards, disease strategies, sorting grades, etc.), guides the robotic arm to scan the mushroom bed point by point, and simultaneously acquires high-resolution images of the mushroom growth scene and related auxiliary information (such as depth and location).

[0122] Perception and Planning under Expert Rules: The acquired images are input into a unified visual perception model for inference, resulting in the detection box, segmentation mask, key points, and disease attributes of the mushrooms. Subsequently, based on a set of expert rules (such as convex hull peeling algorithm, neighbor interference evaluation, and root break direction optimization), the system assesses the pickability of mushrooms within the current field of view and plans the picking sequence, generating a series of robotic arm action instructions that conform to the principles of "high efficiency and low loss".

[0123] Action Execution and Data Recording: The planned action instructions are executed precisely to complete the grasping, transfer, handover, and processing of the mushrooms. Simultaneously, the system structurally aligns and records all key data generated in this cycle—including input images, perception results, planning decisions (action sequences), and execution results (success / failure, mushroom final level). This data constitutes an expert experience dataset rich in the correspondence between "state-action-result."

[0124] Data Output and Model Training: The above steps continue to run, accumulating data. The final high-quality, multimodal dataset (language instructions-images-actions-results) is primarily used to provide training samples for the embodied large model in Part II below, driving it to learn and internalize the decision-making logic of the expert system, ultimately achieving intelligent evolution from rule-driven to model-driven.

[0125] The following section will elaborate on the key technical modules in this process, including scanning strategies, unified perception models, expert planning algorithms, and sorting vision systems.

[0126] System decision-making basis – harvesting settings

[0127] To achieve intelligent harvesting, this system relies on a set of predefined or real-time "harvesting settings." These settings are the core basis for the central controller 600 to plan harvesting tasks, identify targets, make action decisions, and sort and place items. Specifically, they include, but are not limited to:

[0128] Harvesting criteria: minimum / maximum size (diameter or height) and maturity threshold of the target mushroom.

[0129] Quality and disease management strategies: instructions for handling mushrooms with different identified disease types (such as yellow spot, mold, and insect pests) (such as "pick and discard into the abnormal basket" or "skip picking").

[0130] Sorting grade standards: Define the classification rules for different product grades such as Grade A, Grade B, and substandard products (e.g., cap integrity, damage, cut surface quality, etc.).

[0131] Operation mode parameters: such as whether to enable "dark operation" (dependent on lighting module 330), whether to enable automatic basket changing function (based on feedback from weighing sensor 504), target area of ​​harvesting task, etc.

[0132] In operation, all components described below (such as the picking SCARA arm, the collaborative processing robotic arm, the vision system, etc.) perform corresponding recognition, judgment and actions based on this "picking settings".

[0133] Scanning points

[0134] Before harvesting, the robotic arm's vertically downward depth vision sensor 310 photographs the mushroom bed to determine the mushroom's location for picking. To achieve comprehensive visual coverage of the 1.5m × 0.7m working area of ​​the "Dutch style" mushroom bed, the positions of each scanning point need to be rationally designed. At a given height, the camera's field of view is approximately rectangular. Through computer simulation, the coverage of the robotic arm's reach by the camera's field of view under a given sequence of scanning points can be calculated. Scanning points can be defined manually or generated according to rules. Since scanning takes time (typically 1.5 seconds), the optimal scanning point design method can cover the entire area with the fewest possible points.

[0135] Here is a suggested honeycomb layout method. Camera positions are arranged in a honeycomb grid, with the distance between adjacent points equal to the camera's field of view. The first point is where the robotic arm is extended, and the layout extends downwards in a honeycomb pattern, ensuring continuous coverage of the field of view while minimizing overlap. For example... Figure 15 As shown, taking the scanning range of the left arm as an example, boundary 344 represents the motion boundary of the left arm's end, each small rectangle 345 represents the visible range of each scanning point, and the large rectangle 346 represents the mushroom bed area (or an area extended a certain distance from the mushroom bed area, with some redundancy). Rectangle 346 also represents the workspace that the robotic arm should reach. With this cellular configuration, the cumulative field of view of each camera point can cover the entire working area. Scanning is performed sequentially from left to right. After each scan, a virtual model of the actual mushroom bed environment is completed using a perception algorithm.

[0136] Perception Algorithm

[0137] To achieve efficient and unified multi-task visual perception, the expert system employs a unified visual perception model based on deep learning. This model can simultaneously and in real-time complete target detection, instance segmentation, key point localization, and disease attribute identification of mushrooms, meeting the dual requirements of perception accuracy and real-time performance in high-speed harvesting scenarios. For example, upon detecting a mushroom and its key point (the center of the cap), the mushroom is numbered and recorded (for counting purposes only).

[0138] 1. Unified multi-task-aware architecture

[0139] The system employs an end-to-end deep learning network as its perception model, capable of processing visual images (RGB images) and supporting the completion of multiple visual tasks in a single forward propagation, including but not limited to:

[0140] Object detection: Output the bounding box position of the mushroom and its confidence score.

[0141] Instance segmentation: Generate pixel-level segmentation masks for each detected mushroom to accurately outline its contours.

[0142] Key point estimation: Locate preset key points (such as the best grab point) on the mushroom and output their coordinates and visibility status.

[0143] Attribute Classification: Perform multi-label classification on attributes such as mushroom diseases and maturity, and output the corresponding classification scores.

[0144] This architecture achieves a high degree of integration of perception tasks and intensive utilization of computing resources by sharing common features extracted from the backbone network and using multi-task decoupling heads to output the results of each task in parallel.

[0145] 2. Model Training and Optimization

[0146] The model was trained on a large-scale, diverse dataset of manually labeled mushroom images, covering different growth stages, lighting conditions, poses, and typical disease morphologies to ensure the model's robustness and generalization ability. During training, a multi-task joint loss function was employed, using weighted fusion of losses from sub-tasks such as detection, segmentation, keypoint regression, and attribute classification to guide the model in learning feature representations that take into account various perceptual targets.

[0147] Planning Algorithm

[0148] Mushrooms grow densely, and pulling them vertically upwards directly will damage them, causing root breakage. Using fern root movements can reduce root breakage, but if there are mushrooms along the fern root direction, it will crowd other mushrooms. Therefore, the planning algorithm needs to effectively plan for dense scenes to minimize damage. The core idea is to assess the ease of picking a mushroom based on its neighbors, classifying currently pickable and unpickable mushrooms. Simultaneously, clustering is performed based on proximity, dividing the mushrooms within the field of view into different clusters. Further, the convex hull algorithm is used to calculate the outermost pickable mushrooms, and then they are picked sequentially from the outside in, with the neighbor information dynamically updated.

[0149] This system proposes a dynamic harvesting sequence planning method based on pickability assessment and convex hull peeling. By prioritizing the harvesting of outer, harvestable mushrooms and gradually moving inward, it achieves low-damage, high-efficiency automated harvesting. The specific steps are as follows:

[0150] 1. Neighbor Classification

[0151] The radius and center position of the mushroom are calculated from the size and coordinates of the detection box in the perception algorithm. The radius can be estimated from the average of the length and width of the detection box, and the center position can be set as the center of the detection box. The relative distance between the center positions of the mushrooms is then used to determine the center position. Classify the surrounding mushrooms. Define the mushroom being considered for harvesting as the target mushroom, with a radius of... central position The mushrooms surrounding the target mushroom are its neighboring mushrooms, with a radius of... central position The neighboring mushroom has the following definition.

[0152] ① Neighboring mushrooms: Mushrooms that intersect or are tangent to the target mushroom.

[0153]

[0154] ② Next-neighbor mushrooms: within the interference range

[0155]

[0156] definition To determine the horizontal displacement of the mushroom when its root is bent, it can be set to... ,in This is an empirical coefficient.

[0157] ③ Irrelevant mushrooms: beyond the range of interference

[0158]

[0159] 2. Clustering

[0160] Based on the neighbor classification results above, the principle of graph connectivity is used to cluster mushrooms. Each mushroom is considered a node in the graph; if two mushrooms are neighbors or second-neighbors of each other, an edge is established between them. This constructs a neighbor relationship graph. Breadth-first search (BFS) or depth-first search (DFS) algorithms are used to find all connected subgraphs. Each connected subgraph is a mushroom cluster, representing a group of adjacent mushrooms. Mushrooms within each cluster share the same cluster ID. The system maintains a member list, radius, center location, and harvestability status for each cluster. Clustering facilitates subsequent harvesting planning calculations on a cluster-by-cluster basis, enabling parallel processing and accelerating processing efficiency.

[0161] 3. Optimization of the direction of the bend

[0162] To achieve non-destructive harvesting, the system calculates the optimal root-breaking direction for each target mushroom, avoiding collisions with adjacent mushrooms during harvesting. The root-breaking direction can be determined by calculating neighbor relationships and geometric relationships to establish a safe harvesting direction, such as the root-breaking direction calculated based on the internal common tangent theorem in patent CN115034466 A.

[0163] This paper presents a method for calculating the repulsive resultant force based on distance weights. For the target mushroom and all its neighboring mushrooms (including the neighborhood and the next neighborhood), the influence weight of each neighbor on the target mushroom is calculated.

[0164]

[0165] This weighted model allows nearby neighbors to have a greater influence on the root-breaking direction decision. Each neighboring mushroom is considered to exert a "repulsive force" on the target mushroom, with the direction pointing from the neighbor to the target.

[0166]

[0167] in Mushrooms from neighboring regions Pointing to the target mushroom The vector. The optimal root direction is determined by... The orientation is determined.

[0168] Because this system uses a harvesting strategy that involves taking a picture of a fixed scanning point and then capturing all mushrooms below that point, it's unknown whether mushrooms exist outside the field of view at edge locations. If the algorithm assumes there are no mushrooms outside the field of view and uses outward-facing fern root movements, it might damage other mushrooms. Therefore, during calculation, the area outside the field of view... At a distance of [distance], place uniformly [place objects] with a diameter of [diameter]. The virtual mushroom is used to calculate the resultant force direction, ensuring that the direction of the break is determined by the visible area. The calculation formula is as follows:

[0169]

[0170] 4. Removability assessment

[0171] For each mushroom, the system assesses its harvestability under the current environment. The assessment criteria include:

[0172] Current harvesting task indicators: The current harvesting task requirements, such as mushroom size and type. Simultaneously, diseased mushrooms are identified and dealt with according to the current settings.

[0173] Neighbor interference: Whether there is interference from other mushrooms in the direction calculated by the root-breaking optimization algorithm is determined by whether the root-breaking direction line segment intersects with other mushrooms.

[0174] Directional feasibility: Check whether there are feasible directions for bending the root.

[0175] If the mushroom is not interfered with in at least one direction of root bending and is reachable by the robotic arm, and meets the requirements of the current picking task, it is marked as "currently available for picking"; otherwise, it is marked as "currently unavailable for picking".

[0176] Understandably, the pickability assessment here is the "current pickability assessment." Once mushrooms have been picked, the pickability needs to be reassessed.

[0177] 5. Sequential Programming

[0178] As for the picking order, the clusters are first sorted, and the centroid position of each cluster is calculated. The mushrooms with the centroid located in the center of the field of view are picked first.

[0179] For each mushroom cluster, a convex hull peeling and layer-by-layer harvesting method is employed. A two-dimensional convex hull algorithm is used to calculate the outer contours of all currently harvestable mushrooms, and the mushrooms at the vertices of the convex hull are selected as the harvesting targets for this batch. For the mushroom point set in the target cluster area... Calculate the two-dimensional convex hull of the current point set. Mark the vertices of the convex hull. This represents the current harvesting layer. For each harvesting layer of mushrooms, the harvesting order defined in the harvestability assessment is used, prioritizing the harvesting of the mushrooms that best meet the requirements. If multiple mushrooms meet the requirements, they are harvested in a clockwise order based on the angle of the convex hull vertex relative to the cluster center.

[0180] The harvesting order employs a dynamic programming strategy. During the harvesting process at each scan point, scanning operations can be skipped to ensure harvesting speed. Pressure sensor readings confirm successful harvesting of the current mushroom. Upon each successful action, the system immediately updates the environmental model: removing the coordinates of harvested mushrooms, recalculating the neighbor relationships and harvestability status of affected mushrooms, recalculating the harvesting convex hull, and sorting the mushrooms at the current convex hull vertex harvesting layer. If the current cluster is harvested, the system automatically switches to the next cluster.

[0181] Through real-time environmental updates and sequence replanning, the system can adapt to dynamic changes during the harvesting process, ensuring operational continuity and reliability in complex and dense environments.

[0182] Sorting vision system

[0183] To improve sorting accuracy and efficiency, this system employs a three-level visual perception and hierarchical decision-making mechanism, simulating a "funnel-like" layer-by-layer screening process to ensure that only mushrooms meeting the standards enter the final sorting stage.

[0184] Level 1: Depth vision sensing at the end of the harvesting arm 310 – preliminary screening and posture assessment.

[0185] The harvesting arm camera performs real-time detection and preliminary classification of mushrooms before harvesting. Based on the perception algorithm described in Section 2, the harvesting arm camera captures images and outputs the mushroom bounding boxes and their corresponding disease attributes (e.g., normal, yellow spot disease, mold, insect pests). If a severely diseased or deformed mushroom is detected, the system directly marks it as "abnormal." Depending on the harvesting settings, the system chooses to discard the mushroom in the abnormal basket using the harvesting arm, or not harvest it at all.

[0186] Level 2: First HD camera 451 – Secondary verification and placement planning before handover.

[0187] The first high-definition camera is installed below the active cutter assembly, with its optical axis facing the gripper's receiving position. It is used to take a second picture of the mushroom before it is received by the gripper, verifying the mushroom's shape and integrity. It determines whether the mushroom's posture is stable, whether the sides of the cap are intact, and whether there are any yellow spots or lesions. If any abnormalities are found, the picking arm directly discards it into the abnormality basket, or places it on the gripping mechanism, where a collaborative processing robotic arm directly discards it.

[0188] Level 3: Second HD camera 452 - Ultimate quality analysis after root cutting.

[0189] The second high-definition camera is located next to the first camera, with its optical axis pointing vertically upwards, directly at the bottom of the mushroom after root cutting. It is used to immediately photograph the cross-section of the mushroom's bottom and the condition of its stem after root cutting. The camera analyzes the final quality indicators such as the smoothness of the cut surface, the integrity of the stem, and the presence of any hidden diseases. If the cut surface is uneven, has hidden diseases, or does not meet the morphological standards, it is marked as a "substandard product" and discarded into the abnormality basket. If it meets the A or B grade standards, based on the comprehensive score from the three cameras, a collaborative processing robotic arm precisely places it into the corresponding container in the collection device 500.

[0190] The second and third level models can employ lightweight classification or object detection models, achieving millisecond-level inference. This visual sorting system uses a funnel-shaped, step-by-step screening process to avoid misjudgments by a single visual method, thus improving sorting reliability. From harvesting to root cutting and sorting, the entire process is under closed-loop visual monitoring, ensuring that only qualified mushrooms enter the collection stage. The number of cameras, model type, and weight parameters can all be adjusted according to actual production needs to adapt to different mushroom species and grading standards.

[0191] The unified visual perception based on deep learning, the planning based on geometric rules (such as convex hull peeling and neighbor evaluation), and the sorting logic based on three-level cameras described above together constitute the 'expert experience-driven module' of this system. This module can not only independently, efficiently, and reliably execute harvesting tasks, ensuring the basic performance of the system; more importantly, as a structured data acquisition and annotation engine, it synchronously generates high-quality, labeled (such as images, action sequences, success rates, and quality levels) multimodal training data during task execution, providing a solid data foundation for the training and optimization of the embodied large model below.

[0192] The second part, intelligent decision-making systems based on embodied large models, will be introduced below.

[0193] The core of this system's intelligent decision-making is deployed on the central controller 600, constructing a complete data-driven reinforcement learning agent. This agent operates within a closed loop of "perception-decision-execution-feedback-optimization," as shown in the attached diagram. Its aim is to autonomously learn and continuously approach the optimal harvesting strategy through constant interaction with the harvesting environment.

[0194] Its core processes and components are summarized below:

[0195] 1. State Awareness and Decision Generation: The "brain" of the intelligent agent is a pre-trained embodied large model. It receives visual input from various sensors (such as images and point clouds) and task instructions in natural language (i.e., "picking settings"). It can also receive other auxiliary information, such as the mushroom detection box coordinates and physical coordinates output by the unified visual perception model. Based on this, the model outputs the current decision action (such as the robotic arm's trajectory and end-effector operation instructions), and can also simultaneously output predictions of future image states (such as changes in the scene after picking). This embodied large model is not limited to a single visual language action architecture; it can also be a world model or other advanced model variants that can handle multimodal inputs, output action sequences, and future image states, such as the world model. Its core is to establish a mapping from perception to action.

[0196] 2. Action Execution and Environment Interaction: The actions output by the model include, but are not limited to, actual joint angles, end-effector positions, movement speeds, and atomized motion skills, such as returning to the placement site, going to the first scan point, and navigating to the next site. These actions are executed in a real mushroom bed environment to change the environmental state (e.g., harvesting mushrooms) and obtain the execution results.

[0197] 3. Multi-level reward feedback: The system evaluates the consequences of each decision-making-execution cycle and generates quantitative reward signals through a built-in rule-based reward model and a human feedback mechanism.

[0198] (1) Rule-based rewards: mainly derived from the three-level visual evaluation system detailed in Part 1 above, which converts the compliance of harvesting actions, operational quality, and final output quality into structured, fine-grained reward values ​​in real time.

[0199] (2) Human feedback: As a sparse but high-order reward signal, it is provided by the operator after the task to calibrate the alignment of the model with complex reality standards and expert experience.

[0200] 4. Policy Optimization and Continuous Learning: The reward signals mentioned above, along with the corresponding state-action pairs, constitute the training data, which is used to optimize the decision-making model online or periodically (e.g., through reinforcement learning algorithms). By continuously maximizing the cumulative reward, the model can continuously evolve itself, enabling its decisions to continuously improve in dimensions such as efficiency, lossless rate, and sorting accuracy.

[0201] The following sections will elaborate on the key components that constitute this reinforcement learning agent, including the training of the embodied large model, the rule-based multi-level reward design, and the human-in-the-loop optimization mechanism.

[0202] Training and initialization of embodied large models

[0203] This system employs a phased reinforcement learning training paradigm to construct and optimize the embodied large model, specifically comprising the following two core phases:

[0204] Expert imitation pre-training phase:

[0205] The goal of this stage is to utilize the high-quality data accumulated by the expert data collection system in the first part above to provide the model with a high-performance initial strategy, thus avoiding the inefficiency and risk of reinforcement learning starting from scratch.

[0206] The training data system is comprehensive, containing two key types of supervision signals:

[0207] (1) Behavioral clone data: that is, the precise alignment data of "visual observation-task instruction-expert action sequence", which enables the model to initially master "how to perform harvesting actions according to the scene and instructions" through supervised learning.

[0208] (2) Course learning data: Intermediate layer perception and planning information generated during the operation of the expert system, including: ① Perception priors: Outputs from the unified visual perception model, such as the bounding box, segmentation mask, key point (best grasping point) coordinates, and disease attribute classification of the mushroom. This information can serve as initial attention guidance or intermediate prediction targets for the model to understand the scene. ② Planning knowledge: Outputs from the expert rule planning module, such as the "picking availability" assessment results of the mushroom, the picking order of dynamic programming, and the calculated optimal "root bending direction". This information is injected into the model training as a stronger inductive bias, helping it to understand the intrinsic relationship between scene structure, task objectives, and decision logic more quickly, thereby learning a deeper level of expert strategies on "why it is executed this way".

[0209] Based on the aforementioned multi-dimensional data, deep learning methods, including imitation learning, behavior cloning, and video prediction, are used to train the model end-to-end or in stages. Ultimately, a large embodied model is obtained that can deeply understand natural language task instructions, accurately interpret complex visual scenes, and directly output efficient and safe action sequences consistent with task objectives.

[0210] After deployment, the model can autonomously generate and execute a full set of robot actions, from mushroom recognition, picking point selection, motion planning to collaborative handover, based on real-time visual input and preset "picking settings," achieving a higher level of autonomous decision-making and generalization capabilities.

[0211] Online reinforcement learning optimization phase

[0212] After the pre-trained model is deployed, the system enters the online reinforcement learning phase. In this phase, the model acts as an agent, interacting with the environment in real time, and its training objective shifts from "imitating experts" to "maximizing cumulative rewards." The specific process is as follows:

[0213] (1) Exploration and utilization: Based on the learned expert strategy, the model explores strategies by introducing randomness (such as noise), tries new actions to discover potentially better harvesting methods; at the same time, it utilizes known effective actions based on the current optimal strategy most of the time.

[0214] (2) Reward-driven optimization: The state-action-reward-new state data generated by each interaction is stored in the experience replay pool. The system periodically uses reinforcement learning algorithms (such as Proximal Policy Optimization, Soft Actor-Critic SAC, etc.) to update the model policy gradient based on this data and the aforementioned multi-level reward feedback. The core objective of optimization is to enable the model to learn to select action sequences that can obtain higher level three visual rewards and more positive human feedback.

[0215] By combining these two stages, the system achieves a complete reinforcement learning training paradigm that starts quickly from expert experience and continues to evolve through environmental interaction, ensuring the continuous improvement of decision-making intelligence.

[0216] The following is a detailed description of the rule-based reward model and human feedback:

[0217] Rule-based reward model

[0218] To continuously train and optimize the embodied large model, this system constructs a rule-based multi-level reward feedback mechanism during runtime. This mechanism transforms the inherent three-level visual detection and pressure sensor detection stages in the harvesting process into structured reward signals that can be used to evaluate and optimize the embodied large model's action decisions. Each detection corresponds to a specific decision quality evaluation dimension:

[0219] 1. Harvesting Compliance Reward (Level 1 Assessment): This reward is based on image analysis from the depth vision sensor 310 at the end of the harvesting arm. When the embodied model outputs a harvesting action based on the current visual scene, the system compares the mushroom intended to be harvested with the Level 1 visual detection model's judgment of the scene. Assessment criteria include: whether the mushroom meets the size and maturity standards in the "Harvesting Settings," and whether it belongs to a disease type that should be skipped. Simultaneously, combined with pressure sensor readings in the silicone suction cup 320's air path, it can objectively determine whether the action successfully grasped the mushroom. This level of reward guides the model to learn "to harvest the correct target at the correct time and in the correct posture."

[0220] 2. Operational Quality Bonus (Second-Level Evaluation): This bonus is based on images captured by the first high-definition camera 451 before handover. It primarily evaluates the actual execution quality of the grasping action generated by the embodied large model's decision-making process, specifically including: whether the mushroom is adsorbed or held in the exact center of the cap, whether the cap is squeezed or damaged during the operation, and whether any surface lesions not initially identified are found at close range. This level of bonus is used to fine-tune the model's grasping pose planning and motion force control to maximize the probability of non-destructive operation.

[0221] 3. Final Quality Bonus (Third-Level Evaluation): This bonus is based on the analysis of images of the mushroom base after root cutting using a second high-definition camera (452). Its core objective is to evaluate the final output quality of the entire harvesting-processing workflow, inspired by embodied large-scale model decisions. Key indicators include: the smoothness of the root cutting surface and whether improper harvesting force caused hidden damage to the stipe. This level of bonus is directly related to the final grade of the product, driving the model to learn and optimize its initial harvesting strategy from a longer-term process chain perspective.

[0222] By quantifying the above three levels of visual detection results into hierarchical reward signals and feeding them into the training process of the embodied large model, the system can guide the model to explore and approach the globally optimal strategy of "high harvest rate, low damage rate, and excellent final grade" through reinforcement learning without relying on pre-programmed specific rules.

[0223] Human-in-the-loop reinforcement learning optimization

[0224] To further calibrate the gap between model decisions and the complex and ever-changing realities of production and expert experience, the system introduces a human-in-the-loop reinforcement learning mechanism. After the harvesting task is completed, picking and packing workers or operators can comprehensively evaluate the overall quality of the batch of mushrooms (such as the proportion of Grade A mushrooms and the extent of damage). This evaluation, as a sparse but high-value human feedback signal, is used to fine-tune the deployed embodied model, ensuring that its harvesting standards and sorting preferences continuously align with human experience, thus achieving adaptive evolution of the system.

[0225] Therefore, the central controller 600 of the present invention does not execute pre-programmed fixed rules, but runs an intelligent large-scale model decision-making system that can understand tasks, perceive the environment, plan actions, and continuously learn and optimize through visual feedback and human feedback.

[0226] Workflow

[0227] The core logic of this invention's system for achieving fully automated harvesting is based on the intelligent decision-making system (i.e., the embodied large model) deployed in the central controller 600, which coordinates and schedules all hardware modules to complete a closed loop from task understanding to high-quality output. Its overall workflow is as follows:

[0228] 1. Task Understanding and Global Planning: After system initialization, the central controller receives and parses the high-level natural language task instruction "Harvesting Settings". Based on this instruction and the global mushroom house map, the intelligent decision-making system performs task decomposition and macro-planning, generating a global movement and harvesting strategy that includes the target area sequence and priority. Guided by the SLAM navigation system, the mobile chassis 100 autonomously moves according to this strategy and accurately stops at the working position in front of the first target bed frame.

[0229] 2. Scene Perception and Real-Time Decision-Making: The operating platform is adjusted to the target planting layer height via the lifting mechanism 200. Subsequently, the intelligent decision-making system enters a real-time decision-making loop. The system receives real-time multimodal perception data (images, depth information) from sensors such as the vision sensor 102 and the depth vision sensor 310 at the end of the harvesting arm, and combines it with the current task context. The embodied large model processes this information in an integrated manner and directly outputs the optimal sequence of robot action instructions, including:

[0230] (1) Picking decision: Determine which mushrooms in the current field of vision meet the standards, and plan the specific picking points, picking order (considering dense avoidance) and non-destructive grabbing (fern root) direction.

[0231] (2) Motion generation: Generate the motion trajectory of the SCARA arm 300 from its current position to the target picking point and then to the collaborative handover point.

[0232] (3) Collaborative scheduling: Plan the receiving, cutting, sorting and delivery actions of the robotic arm 400 to ensure precise synchronization with the picking arm and form an efficient production line.

[0233] 3. Collaborative Execution and Closed-Loop Feedback: Planned action instructions are synchronously issued to each executing mechanism. The harvesting arm uses a silicone suction cup 320 to perform high-speed, non-destructive gripping and transfer the mushrooms to a fixed handover point; the collaborative processing robotic arm then receives the mushrooms, performs precise root cutting and online sorting based on three-level vision, and places the mushrooms into the corresponding containers of the collection device 500. The entire execution process also serves as a data collection and evaluation stage:

[0234] (1) Data recording: The system continuously records “status (image / instruction)-action-result” data pairs to enrich the expert dataset.

[0235] (2) Reward generation: The three-level vision system and pressure sensor evaluate the results of each step in real time and convert them into reward signals for model optimization.

[0236] (3) Status update: After each successful harvest, the virtual model of the actual environment of the mushroom bed obtained by scanning is updated in real time, providing the latest scene information for the next round of decision-making.

[0237] 4. Closed-loop and autonomous transfer of regional tasks: The system monitors the status of the collection containers through the 504 weighing sensor and prompts for basket replacement when necessary. When the intelligent system determines that the harvesting task in the current work area (such as a scanning point or a bed frame segment) has been satisfactorily completed (for example, all pickable mushrooms have been harvested or the preset conditions have been met), the central controller will automatically plan the next target location, move the chassis autonomously, and repeat the process from "scene perception and real-time decision-making" to "collaborative execution and closed-loop feedback" until all global harvesting tasks are completed.

[0238] 5. Continuous learning and system evolution: During task breaks or in the background, the system uses data collected during runtime (including rule rewards and manual feedback entered later) to periodically optimize and fine-tune the intelligent decision-making model through reinforcement learning algorithms, so that its harvesting strategy continues to evolve towards a more efficient, less lossy, and more production-oriented direction.

[0239] Through the above process, the system achieves full-stack intelligence, from high-level task instruction parsing to low-level action execution, and then to data-driven self-optimization. The reinforcement learning closed loop of "perception-decision-execution-feedback" runs throughout the entire process, enabling the robot not only to complete the entire process of "finding, picking, cutting, and separating" like a skilled worker, but also to have the ability to adapt and continuously improve beyond fixed rules. Under the premise of ensuring an ultra-low damage rate and perfect root cutting quality, the system maximizes its overall operational efficiency and long-term value.

[0240] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A collaborative mushroom harvesting humanoid robot system, characterized in that, include: Vision system, central controller, mobile chassis, lifting mechanism, operating platform, and picking SCARA arm, collaborative processing robotic arm, rotary cutting assembly, and collection device mounted on the operating platform; The central controller coordinates and controls the working status of other modules based on the information collected by the vision system. The mobile chassis and the lifting mechanism are used to adjust the position of the operating platform; The SCARA harvesting arm is used to harvest mushrooms and hand them over to the collaborative processing robotic arm; The collaborative processing robotic arm is used to grip the mushroom and work with the rotary cutting component to cut the root; The central controller determines the mushroom quality classification based on the information collected by the vision system, and controls the movement of the picking SCARA arm and / or the collaborative processing robotic arm to place the mushrooms into the corresponding category containers of the collection device.

2. The collaborative mushroom harvesting humanoid robot system according to claim 1, characterized in that, The top of the lifting mechanism is equipped with a lidar, which can collect environmental information to assist in obstacle avoidance.

3. The collaborative mushroom harvesting humanoid robot system according to claim 1, characterized in that, The harvesting SCARA arm is a three-axis or four-axis SCARA arm, with degrees of freedom for adjustment in the height direction and in the horizontal plane. The end of the harvesting SCARA arm is equipped with a silicone suction cup for adsorbing mushrooms.

4. The collaborative mushroom harvesting humanoid robot system according to claim 3, characterized in that, The silicone suction cup is funnel-shaped, including a tubular handle and a conical suction part. The tubular handle is used to be installed at the end of the harvesting SCARA arm, and the outer surface of the tubular handle is provided with a flexible reinforcement structure.

5. A collaborative mushroom harvesting humanoid robot system according to claim 3, characterized in that, The harvesting SCARA arm is also equipped with a lighting module, which can provide illumination during the harvesting process.

6. The collaborative mushroom harvesting humanoid robot system according to claim 1, characterized in that, The collaborative processing robotic arm includes a lifting module, a rotary table, and a two-finger gripper. The lifting module adjusts the height of the rotary table and the two-finger gripper, and the rotary table adjusts the orientation of the two-finger gripper.

7. A collaborative mushroom harvesting humanoid robot system according to claim 6, characterized in that, The two-finger gripper can grip and release mushrooms. The part of the two-finger gripper that grips the mushroom body is bowl-shaped, and its bottom is provided with a hole for the mushroom root to pass through.

8. A collaborative mushroom harvesting humanoid robot system according to claim 1, characterized in that, The system includes two or more sets of SCARA harvesting arms and collaborative processing robotic arms; the same set of SCARA harvesting arms and collaborative processing robotic arms hand over mushrooms at a fixed position, or the same set of SCARA harvesting arms and collaborative processing robotic arms adaptively selects a position to hand over mushrooms according to the rhythm.

9. A collaborative mushroom harvesting humanoid robot system according to claim 1, characterized in that, The rotary cutting assembly includes a drive motor and a cutting blade; After the collaborative processing robotic arm holds the mushroom to be cut at the cutting point, the drive motor drives the cutter to rotate and cut off the mushroom root.

10. A collaborative mushroom harvesting humanoid robot system according to claim 1, characterized in that, The collection device includes at least three containers, which are distributed on the same layer or different layers; each container is equipped with a weighing sensor at the bottom, which can measure the weight of the mushrooms in the container.

11. A collaborative mushroom harvesting humanoid robot system according to claim 1, characterized in that, The vision system includes: A visual sensor is used to collect video data of the mushroom bed area to determine the current harvesting status. A depth vision camera mounted on the harvesting SCARA arm is used to acquire three-dimensional information about the mushroom. A first high-definition camera is used to capture images of the mushrooms harvested by the SCARA arm. A second high-definition camera is used to capture images of the mushrooms after they have been cut and are held by the collaborative processing robotic arm.

12. A collaborative mushroom harvesting humanoid robot system according to claim 11, characterized in that, The central controller controls the harvesting SCARA arm to harvest mushrooms based on the three-dimensional information of the mushrooms collected by the vision sensor; the central controller also identifies abnormal mushrooms based on the mushroom image information collected by the vision sensor, and controls the harvesting SCARA arm to harvest the abnormal mushrooms and put them into the waste container.

13. A collaborative mushroom harvesting humanoid robot system according to claim 11, characterized in that, The central controller identifies abnormal mushrooms based on images captured by the first high-definition camera and controls the SCARA harvesting arm to place the abnormal mushrooms into a waste container, or controls the SCARA harvesting arm to transfer the abnormal mushrooms to the collaborative processing robotic arm, which then places the abnormal mushrooms into a waste container.

14. A collaborative mushroom harvesting humanoid robot system according to claim 11, characterized in that, The central controller identifies abnormal mushrooms based on images captured by the second high-definition camera, and the collaborative processing robotic arm places the abnormal mushrooms into a waste container. If no abnormal mushrooms are identified, the visual data collected by the depth vision camera, the first high-definition camera, and the second high-definition camera are combined to classify the quality of the mushrooms and place them into the corresponding category of containers in the collection device.

Images