Fruit picking method, device, apparatus, and medium

Abstract

The application relates to the technical field of fruit picking, and specifically provides a fruit picking method, device, equipment and medium, and aims to solve the problem that an existing scheme cannot express real physical interaction, thereby failing to verify the stability of a picking strategy in a virtual environment. To this end, the method provided by the application comprises the following steps: obtaining a plant model comprising a fruit rigid body, a fruit stem discrete rigid body segment and a stem discrete rigid body segment, the fruit rigid body being connected with the fruit stem discrete rigid body segment through a breakable joint; controlling an end effector to perform a picking action and calculating a picking parameter at the breakable joint; comparing the picking parameter with a preset picking parameter threshold value, if the picking parameter is greater than the preset picking parameter threshold value, the breakable joint is broken, a broken position marking point is generated, and a picking result is generated; based on the application, real physical interactions such as bending, swinging, pulling and separation in the picking process can be simulated, and the stability of the picking strategy can be verified in the virtual environment.

Description

Technical Field

[0001] This application relates to the field of fruit harvesting technology, specifically to a fruit harvesting method, apparatus, equipment, and medium. Background Technology

[0002] Current solutions for fruit harvesting typically involve building simulation environments for target detection, pose estimation, and path planning to simulate real-world harvesting scenarios, and then applying the simulation results to real-world harvesting scenarios.

[0003] While these solutions can be used for visual inspection or simple grasping demonstrations, in real-world harvesting scenarios, robots need to have complex interactions with fruits, stems, stalks, leaves, soil, and surrounding plants. When fruits are grasped, pulled, or twisted, the connecting stems will deform, swing, and experience constraint forces. Simulation environments cannot represent this physical interaction, making it impossible to assess whether the harvesting strategy is stable, whether it will accidentally touch surrounding leaves, whether it will break non-target branches, or whether it will cause fruits to pass through the mold or fly out abnormally.

[0004] Therefore, existing solutions cannot represent real physical interactions, making it impossible to verify the stability of harvesting strategies in a virtual environment.

[0005] Accordingly, there is a need in this field for a new fruit harvesting method to address the aforementioned problems. Summary of the Invention

[0006] In order to overcome the above-mentioned shortcomings, this application is proposed to solve or at least partially solve the technical problem that existing solutions cannot express real physical interactions, resulting in the inability to verify the stability of harvesting strategies in a virtual environment.

[0007] In a first aspect, a method for harvesting fruit is provided, the method comprising:

[0008] A fruit harvesting simulation scenario is constructed, and the fruit plants in the simulation scenario are modeled to obtain a plant model. The plant model includes at least one fruit rigid body, at least one fruit stalk discrete rigid body segment, and at least one stem discrete rigid body segment. Each fruit rigid body is sequentially connected to at least one fruit stalk discrete rigid body segment and at least one stem discrete rigid body segment. Adjacent discrete rigid body segments are connected by joints. The fruit rigid body is connected to the fruit stalk discrete rigid body segment closest to the fruit rigid body by a breakable joint.

[0009] The end effector of the simulated picking robot is controlled to approach the target fruit, contact and clamp the target fruit or the fruit stalk connected to the target fruit, and perform the picking action.

[0010] When performing the picking action, the picking parameters at the breakable joint are calculated;

[0011] The picking parameters are compared with a preset picking parameter threshold to obtain a comparison result. If the comparison result is that the picking parameters are greater than the preset picking parameter threshold, the fracture joint breaks, a fracture location marker is generated, and a picking result is generated. The picking result is used to guide the real robot to perform the fruit picking task.

[0012] In one technical solution of the above-mentioned fruit harvesting method, the step of constructing a fruit harvesting simulation scenario and modeling the fruit plants in the simulation scenario to obtain a plant model includes:

[0013] A fruit-picking simulation scenario is constructed. For each fruit plant in the simulation scenario, the fruit is constructed as a rigid body, the pedicel is constructed as at least one discrete rigid body segment of the pedicel connected in sequence, the stem is constructed as at least one discrete rigid body segment of the stem connected in sequence, and a collision body is constructed outside each discrete rigid body segment of the pedicel and each discrete rigid body segment of the stem to enclose the discrete rigid body segment.

[0014] A joint chain is configured for a fruit rigid body, at least one fruit stalk discrete rigid body segment and at least one stem discrete rigid body segment connected in sequence, wherein a joint connection is configured between adjacent discrete rigid body segments and a breakable joint connection is configured between the fruit rigid body and the fruit stalk discrete rigid body segment closest to the fruit rigid body.

[0015] After configuration, a plant model is obtained.

[0016] In one technical solution of the above-mentioned fruit picking method, the end effector of the controlled simulated picking robot approaches the target fruit, including:

[0017] Obtain the visual information of the simulated harvesting robot or the spatial annotation information in the fruit harvesting simulation scene;

[0018] The target fruit is located based on the visual information or the spatial labeling information, and a picking route is planned.

[0019] The end effector of the simulated picking robot is controlled to approach the target fruit based on the picking route. During the approach to the target fruit, the simulated picking robot is detected in real time whether it comes into contact with the collision object, and the picking route is adjusted accordingly.

[0020] In one technical solution of the above-mentioned fruit picking method, the joint and the fractured joint are D6 Joints or spring dampers, and the D6 Joints or spring dampers have translational degrees of freedom, rotational degrees of freedom, and adjustable stiffness and damping.

[0021] In one technical solution of the above-mentioned fruit harvesting method, the preset harvesting parameter threshold includes a joint fracture force threshold or a joint fracture moment threshold, the harvesting parameter includes a joint fracture force or a joint fracture moment, and if the comparison result is that the harvesting parameter is greater than the preset parameter threshold, the fractured joint breaks, including:

[0022] If the comparison result shows that either the joint fracture force or the joint fracture torque is greater than the corresponding preset threshold value, the fractureable joint breaks.

[0023] In one technical solution of the above-mentioned fruit harvesting method, the step of generating the breakage location marker and generating the harvesting result includes:

[0024] The fracture location marker is generated at the spatial location of the fracture joint, and the fracture time, fracture object, fracture location, picking parameters at the time of fracture, and end effector attitude are recorded.

[0025] Acquire the fruit detachment status, end effector clamping stability, and collision anomalies;

[0026] The picking result is generated based on the fruit detachment state, fracture object, fracture location, end effector posture, end effector clamping stability, and collision anomaly. The picking result includes whether the fruit picking was successful, fracture location coordinates, fruit ID, end effector picking action type, and whether there is clipping, abnormal collision, abnormal fracture, or abnormal movement.

[0027] In one technical solution of the above-mentioned fruit harvesting method, the method further includes:

[0028] If the comparison result is that the picking parameter is less than or equal to the preset picking parameter threshold, the picking parameters and picking action of the end effector of the simulated picking robot when performing the picking action are adjusted.

[0029] And / or,

[0030] When the harvesting result is a fruit harvesting failure, obtain the reason for the failure;

[0031] Based on the reasons for the failure, the end effector of the simulated harvesting robot was adjusted to perform harvesting parameters and harvesting actions.

[0032] In a second aspect, a fruit-harvesting apparatus is provided, the apparatus comprising:

[0033] A simulation scene construction module is used to construct a fruit picking simulation scene and model the fruit plants in the simulation scene to obtain a plant model. The plant model includes at least one fruit rigid body, at least one fruit stalk discrete rigid body segment and at least one stem discrete rigid body segment. Each fruit rigid body is sequentially connected to at least one fruit stalk discrete rigid body segment and at least one stem discrete rigid body segment. Adjacent discrete rigid body segments are connected by joints. The fruit rigid body is connected to the fruit stalk discrete rigid body segment closest to the fruit rigid body by a breakable joint.

[0034] The control module is used to control the end effector of the simulated picking robot to approach the target fruit, contact and clamp the target fruit or the fruit stalk connected to the target fruit, and perform the picking action.

[0035] The picking parameter calculation module is used to calculate the picking parameters at the breakable joint when performing the picking action;

[0036] The harvesting result generation module is used to compare the harvesting parameters with a preset harvesting parameter threshold to obtain a comparison result. If the comparison result is that the harvesting parameters are greater than the preset harvesting parameter threshold, the fracture joint breaks, a fracture location marker is generated, and a harvesting result is generated. The harvesting result is used to guide the real robot to perform the fruit harvesting task.

[0037] In a third aspect, an electronic device is provided, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, which, when executed by the at least one processor, implements the method described in any of the above-described fruit harvesting methods.

[0038] In a fourth aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, the program codes being adapted to be loaded and run by a processor to perform the method described in any of the above-described technical solutions of the fruit harvesting method.

[0039] The above-described technical solutions of this application have at least one or more of the following beneficial effects:

[0040] In implementing the fruit harvesting method provided in this application, a fruit harvesting simulation scenario is constructed, and the fruit plants in the simulation scenario are modeled to obtain a plant model. The plant model includes at least one fruit rigid body, at least one fruit stalk discrete rigid body segment, and at least one stem discrete rigid body segment. Each fruit rigid body is sequentially connected to at least one fruit stalk discrete rigid body segment and at least one stem discrete rigid body segment. Adjacent discrete rigid body segments are connected by joints, and the fruit rigid body is connected to the fruit stalk discrete rigid body segment closest to the fruit rigid body by a breakable joint. The end effector of the simulated harvesting robot is controlled to approach the target fruit. The robot contacts and grasps the target fruit or the fruit stalk connected to the target fruit to perform a picking action. During the picking action, the picking parameters at the breakable joint are calculated. The picking parameters are compared with a preset picking parameter threshold to obtain a comparison result. If the comparison result is that the picking parameters are greater than the preset picking parameter threshold, the breakable joint breaks, a breakage location marker is generated, and a picking result is generated. The picking result is used to guide the real robot to perform fruit picking tasks. Based on this application, it is possible to simulate real physical interactions such as bending, swinging, pulling, and detachment during the picking process, and the stability of the picking strategy can be verified in a virtual environment. Attached Figure Description

[0041] The disclosure of this application will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Wherein:

[0042] Figure 1 This is a schematic flowchart of the main steps of a fruit harvesting method according to an embodiment of this application;

[0043] Figure 2 This is a schematic diagram of the main structure of a plant model according to an embodiment of this application;

[0044] Figure 3 This is a schematic diagram illustrating the generation of fracture location markers according to an embodiment of this application;

[0045] Figure 4 This is a schematic diagram of a fruit picking simulation scene according to an embodiment of this application;

[0046] Figure 5 This is a schematic diagram of the main structure of an electronic device according to an embodiment of this application.

[0047] Figure label:

[0048] 21: Rigid body of fruit; 22: Discrete rigid body segment of fruit stalk; 23: Discrete rigid body segment of stem; 24: Collision body; 25: Joint chain; 26: Fragile joint; 27: Joint; 41: End effector; 42: Non-target leaf; 43: Target fruit; 51: Memory; 52: Processor. Detailed Implementation

[0049] Some embodiments of this application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of this application and are not intended to limit the scope of protection of this application.

[0050] In the description of this application, "module" and "processor" can include hardware, software, or a combination of both. A module can include hardware circuitry, various suitable sensors, communication ports, memory, and may also include software components, such as program code, or a combination of software and hardware. A processor can be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor can be implemented in software, in hardware, or a combination of both. Computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular terms "a" or "this" can also include plural forms.

[0051] In the embodiments of this application, it should be noted that, in this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0052] Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0053] In the description of the embodiments of this application, the words "example" or "for example" are used to indicate exemplification, illustration, or description. Any embodiment or design described as "example" or "for example" in the embodiments of this application is not to be construed as being more preferred or having more advantages than another embodiment or design. The use of the words "example" or "for example" is intended to present relative concepts in a clear manner.

[0054] Here we will first explain some of the terms used in this application.

[0055] USD: refers to a 3D scene description format used to express scene hierarchy, geometry, materials, physical properties, and reference relationships.

[0056] IsaacSim is a robot simulation platform based on Omniverse (NVIDIA's 3D collaboration and simulation platform), used to build physical simulation scenes, robot interaction tasks, and synthetic data environments.

[0057] Flexible physical modeling refers to the method of discretizing a continuous flexible object into multiple rigid body segments and simulating bending, swinging, pulling and breaking behavior through joints and constraints.

[0058] Plant model: refers to a data structure used for physical simulation, including at least one rigid body of fruit, at least one discrete rigid body segment of fruit stalk, and at least one discrete rigid body segment of stem. The rigid body and rigid body segments are connected by joints and fractured joints to simulate the physical characteristics and dynamic response of real plants.

[0059] Rigid body: refers to an object in simulation that has mass, collision, velocity and force response.

[0060] Collider: refers to the geometric range in which an object participates in contact, collision, and blocking in a physical simulation.

[0061] D6 Joint: A six-degree-of-freedom joint constraint that allows setting motion restrictions, stiffness, and damping on each of the six degrees of freedom (three translation axes and three rotation axes). It can be used to accurately express the restricted motion between flexible stems, fruit connection points, or branch segments, thereby simulating complex flexible connection characteristics. It is the most structurally complex joint type among the PhysX standard joints. By default, its behavior is equivalent to a fixed joint, that is, a rigid constraint coordinate system of two rigid bodies. However, developers can unlock each degree of freedom individually, allowing arbitrary combinations of rotation and translational motion along the X, Y, and Z axes. PhysX is a physics engine developed by NVIDIA.

[0062] A breakable joint is a physical joint with a preset breakage threshold. During simulation, when the force or torque applied to the joint exceeds its preset joint breakage force threshold or joint breakage torque threshold, the connection of the joint will be automatically released to simulate the real process of the fruit stem being broken off by physical force.

[0063] Harvesting parameters refer to the physical quantities that are calculated in real time during the simulated harvesting process to determine whether a breakable joint has broken. Specifically, they may include the joint breaking force or joint breaking torque applied to the breakable joint. These parameters are the direct physical basis for determining whether the harvesting action is successful.

[0064] Joint fracture force threshold: refers to the maximum linear force that a fractured joint can withstand before it breaks; when the tensile or shear force on the joint exceeds the preset joint fracture force threshold, the joint will automatically break, which is used to simulate the process of fruit stalks or stems being broken due to direct pulling.

[0065] Joint fracture torque threshold: refers to the maximum torsional torque that a fracture joint can withstand before it breaks; when the torque on the joint exceeds the preset joint fracture torque threshold, the joint will automatically break; used to simulate the process of fruit stems being twisted off during harvesting through rotation, torsion and other actions.

[0066] Fracture location marker: This refers to a data marker point generated at the three-dimensional spatial location where the fracture occurs when a fractured joint breaks. This marker point is typically used to record contextual information related to the fracture event, such as fracture time, fracture object, fracture location, acquisition parameters at the time of fracture, and end effector attitude, providing data support for subsequent analysis and strategy optimization.

[0067] See appendix Figure 1 , Figure 1 This is a schematic flowchart illustrating the main steps of a fruit harvesting method according to an embodiment of this application. Figure 1 As shown, the fruit harvesting method in this application embodiment mainly includes the following steps S101 to S104.

[0068] Step S101: Construct a fruit picking simulation scene and model the fruit plants in the simulation scene to obtain a plant model. The plant model includes at least one fruit rigid body, at least one fruit stalk discrete rigid body segment and at least one stem discrete rigid body segment. Each fruit rigid body is connected to at least one fruit stalk discrete rigid body segment and at least one stem discrete rigid body segment in sequence. Adjacent discrete rigid body segments are connected by joints. The fruit rigid body is connected to the fruit stalk discrete rigid body segment closest to the fruit rigid body by a breakable joint.

[0069] Step S102: Control the end effector of the simulated picking robot to approach the target fruit, contact and clamp the target fruit or the fruit stem connected to the target fruit, and perform the picking action.

[0070] Step S103: When performing the picking action, calculate the picking parameters at the fracture joint;

[0071] Step S104: Compare the picking parameters with the preset picking parameter threshold to obtain the comparison result. If the comparison result is that the picking parameters are greater than the preset picking parameter threshold, the joint can be broken, the break position mark point is generated, and the picking result is generated. The picking result is used to guide the real robot to perform the fruit picking task.

[0072] Based on the methods described in steps S101 to S104 above, a plant model with high physical fidelity is constructed, and the physical interaction process between the robot and the plant model is simulated, thereby achieving an effective evaluation of the harvesting strategy; the success or failure of harvesting is transformed from a simple script-triggered event into a dynamic process driven by real physical forces, thereby simulating real physical interactions such as bending, swinging, pulling, and detachment during the harvesting process.

[0073] Specifically, firstly, a plant model reflecting flexibility was constructed. This model simulates the realistic physical interactions of the plant caused by forces during the robot's harvesting action, such as bending, swaying, and pulling, enhancing the realism of the simulation environment. Secondly, by calculating the harvesting parameters at the breakable joint in real time and comparing them with preset thresholds, the fruit will only detach when the force or torque applied by the robot is sufficient to break the breakable joint. This approach directly links the harvesting results to the real physical interaction process, enabling reliable verification and evaluation of the effectiveness and stability of different harvesting strategies (such as different pulling forces, angles, or torsional movements) in the virtual environment. Finally, when the breakable joint breaks, the system generates a fracture location marker and a harvesting result. This data not only accurately reflects the reasons for successful or failed harvesting but can also be used to generate high-quality training data labels, helping to optimize the robot's harvesting action strategy and providing precise data support for guiding robot operations in the real world.

[0074] The following describes an embodiment of the fruit harvesting method provided in this application, specifically step S101.

[0075] In one embodiment of step S101 above, the fruit plant and its surrounding environment are organized into an interactive USD simulation scene on IsaacSim, that is, a fruit picking simulation scene is constructed, and the fruit plant in the simulation scene is modeled in a flexible physical way. The continuous flexible objects such as fruit stalks and stems are discretized into multiple rigid body segments, and bending, swinging, pulling and breaking behaviors are simulated through joints and constraints.

[0076] In one embodiment of step S101 above, a fruit harvesting simulation scenario is constructed, and the fruit plants in the simulation scenario are modeled to obtain a plant model, including:

[0077] Construct a fruit picking simulation scenario. For each fruit plant in the simulation scenario, construct the fruit as a rigid body, construct the fruit stalk as at least one discrete rigid body segment of the fruit stalk connected in sequence, construct the stem as at least one discrete rigid body segment of the stem connected in sequence, and construct a collider that wraps around each discrete rigid body segment of the fruit stalk and each discrete rigid body segment of the stem.

[0078] A joint chain is configured for a fruit rigid body, at least one fruit stalk discrete rigid body segment and at least one stem discrete rigid body segment connected in sequence, wherein a joint connection is configured between adjacent discrete rigid body segments and a breakable joint connection is configured between the fruit rigid body and the fruit stalk discrete rigid body segment closest to the fruit rigid body.

[0079] After configuration, a plant model is obtained.

[0080] See appendix Figure 2 , Figure 2 This is a schematic diagram of the main structure of a plant model according to an embodiment of this application. Figure 2 As shown, taking a strawberry plant as an example, the plant model in this application embodiment mainly includes a fruit rigid body 21, n discrete rigid body segments of the fruit stalk 22, m discrete rigid body segments of the stem 23, a collision body 24, a joint chain 25, a breakable joint 26, and a joint 27.

[0081] Specifically, a fruit-harvesting simulation scenario is constructed. For each fruit plant in the simulation scenario, the fruit is constructed as a rigid body 21, the stalk is constructed as at least one discrete rigid body segment 22 connected in sequence, and the stem is constructed as at least one discrete rigid body segment 23 connected in sequence. This allows the continuous flexible stalk and stem to simulate behaviors such as bending and torsion through the relative motion between multiple discrete rigid body segments. A collider 24 is constructed outside each discrete rigid body segment 22 and each discrete rigid body segment 23, which defines the geometric edges of the rigid body segment in the physical simulation world. The boundary is used to participate in collision detection; similarly, other objects in the simulation scene (such as non-target leaves) will also be assigned corresponding colliders; a joint chain 25 is configured for the sequentially connected fruit rigid body 21, at least one fruit stem discrete rigid body segment 22 and at least one stem discrete rigid body segment 23, wherein a joint 27 is configured to connect adjacent discrete rigid body segments, and a breakable joint 26 is configured to connect the fruit rigid body 21 and the fruit stem discrete rigid body segment 22 closest to the fruit rigid body 21. These joints allow limited relative rotation between the rigid body segments, so that the entire joint chain can simulate the bending and swaying of the stem.

[0082] Based on the methods described above, a plant model reflecting flexibility is constructed by discretizing the fruit stalks and stems of the plant into multiple rigid segments and connecting them using joints and fractured joints. This model can simulate realistic physical interactions such as bending, swinging, and pulling, enhancing the realism of the simulation environment. Simultaneously, collision bodies are constructed around the rigid segments or leaves. When the end effector of the simulated harvesting robot approaches the target fruit, it may touch non-target stems or leaves. Due to the presence of these collision bodies, such contact can be accurately detected, improving the physical realism of the simulation. This allows for the assessment of unexpected collisions that may occur in complex and crowded environments, thereby verifying the robustness and safety of the harvesting path planning.

[0083] In one embodiment of step S101 above, the joint and the fractureable joint are D6 joints or spring dampers, the joint chain is a D6 joint chain, and the D6 joint or spring damper has translational degrees of freedom, rotational degrees of freedom, and adjustable stiffness and damping; the translational degrees of freedom include three translational degrees of freedom (translation along the X, Y, and Z axes), the rotational degrees of freedom include three rotational degrees of freedom (rotation about the X, Y, and Z axes), and each degree of freedom can be independently locked, restricted, or set to free motion; such as Figure 2 As shown, joint stiffness includes linear constraints along / around the joint axis ( , , (unit N / m) and angle constraints ( , , (Unit: N·m / rad) Joint stiffness determines the strength of a joint to resist deformation under stress; joint damping includes linear constraints along / around the joint axis ( , , (unit: N·s / m) and angular constraints ( , , (Unit: N·m·s / rad), (x, y, z) is a local coordinate system, and the joint damping determines the rate of motion decay.

[0084] Based on the methods described above, by adjusting the translational degrees of freedom, rotational degrees of freedom, and adjustable stiffness and damping, the plant model can exhibit bending, torsion, and vibration characteristics very similar to those of a real fruit stalk or stem. The use of configurable joints such as D6 Joints or spring dampers enhances the physical fidelity of the plant model. Compared to using simple joints with fixed properties, this application can more flexibly and accurately simulate the unique mechanical properties of fruit plants of different species and growth stages, thus making the simulation results closer to the real physical world and providing a more reliable basis for verifying harvesting strategies.

[0085] The following describes an embodiment of the fruit harvesting method provided in this application, specifically step S102.

[0086] In one embodiment of step S102 above, controlling the end effector of the simulated harvesting robot to approach the target fruit includes:

[0087] Obtain visual information from the simulated fruit-picking robot or spatial annotation information from the fruit-picking simulation scene;

[0088] Locate the target fruit based on visual or spatial labeling information, and plan the picking route;

[0089] The end effector of the simulated picking robot, controlled by the picking route, approaches the target fruit. During the approach, the robot detects in real time whether it comes into contact with a collision object and adjusts the picking route accordingly.

[0090] Specifically, the visual information can be visual information collected by a virtual camera device on the simulated picking robot, and the spatial labeling information can be pre-set spatial labeling information in the simulation scene (such as the precise three-dimensional coordinates of the target fruit). Based on this information, the target fruit can be located, and a picking route to the target fruit can be generated using a motion planning algorithm. During the movement of the simulated picking robot along the picking route, it is detected in real time whether the end effector comes into contact with other objects in the environment (such as the various colliders 24 in the plant model). Once an unexpected collision that is about to occur or has already occurred is detected, the picking route can be adjusted immediately, for example, by replanning a path to avoid obstacles.

[0091] The method described in the above steps enables the simulated robot to have environmental perception and obstacle avoidance capabilities similar to those of a real robot; the effectiveness of robot navigation and obstacle avoidance algorithms can be fully verified and iterated during the simulation stage, thereby improving the success rate and safety of the harvesting strategy finally deployed on the real robot.

[0092] The following describes an embodiment of the fruit harvesting method provided in this application, specifically step S103.

[0093] In one embodiment of step S103 above, the picking parameters include joint fracture force or joint fracture moment. When performing the picking action, the picking parameters calculated in real time include the joint fracture force and joint fracture moment applied to the fractureable joint.

[0094] The following describes an embodiment of the fruit harvesting method provided in this application, specifically step S104.

[0095] In one embodiment of step S104 above, the preset harvesting parameter threshold includes a joint fracture force threshold or a joint fracture moment threshold. If the comparison result shows that the harvesting parameter is greater than the preset harvesting parameter threshold, the fractured joint can fracture, including:

[0096] If the comparison result shows that either the joint fracture force or the joint fracture moment is greater than the corresponding preset threshold, the joint can be broken.

[0097] Specifically, during the comparison, the system checks whether either the joint fracture force or the joint fracture torque exceeds its corresponding preset threshold. If either the joint fracture force or the joint fracture torque exceeds the preset threshold, the fractured joint breaks. For example, if the preset joint fracture force threshold is 5N and the joint fracture torque threshold is 2N·m, and the robot applies a pulling force of 5.1N to the fractured joint during the pulling process, the joint will break even if the torque is only 0.5N·m. Conversely, if the robot applies a torque of 2.2N·m to the fractured joint through torsion, the joint will also break even if the pulling force is only 1N.

[0098] The method described in the above steps can comprehensively simulate diverse harvesting methods in the real world. Whether relying on pure pulling force, torsional torque, or a combination of both, as long as one of them reaches the critical point of destruction, the fruit can be triggered to detach. It can comprehensively evaluate and compare the effectiveness of different harvesting actions (i.e., the types of harvesting actions of the robot's end effector, such as pulling, dragging, and twisting). Whether the fruit is picked depends on the actual pulling force, torque, and action path applied by the robot, rather than simple script triggering, providing a more refined physical basis for the robot to select the optimal harvesting strategy.

[0099] In one embodiment of step S104 above, generating fracture location markers and generating harvesting results includes:

[0100] Generate fracture location markers on the spatial location of the fracture joint, and record the fracture time, fracture object, fracture location, picking parameters at the time of fracture, and end effector attitude.

[0101] Acquire the fruit detachment status, end effector clamping stability, and collision anomalies;

[0102] The harvesting results are generated based on the fruit detachment status, fracture object, fracture location, end effector posture, end effector clamping stability, and collision anomalies. The harvesting results include whether the fruit harvesting was successful, fracture location coordinates, fruit ID, end effector harvesting action type, and whether there are any clipping, abnormal collisions, abnormal fractures, or abnormal movements.

[0103] Specifically, see the appendix. Figure 3 , Figure 3 This is a schematic diagram illustrating the generation of fracture location markers according to an embodiment of this application. For example... Figure 3 As shown, at the instant the fractured joint breaks, a fracture location marker is generated at the fracture site (using a proxy sphere model for location marking), and the fracture time, fractured object (e.g., the ID of a specific fruit), fracture location, picking parameters at the time of fracture (e.g., the final joint fracture force and joint fracture torque), and end effector attitude (position and orientation) are recorded. Simultaneously, states related to picking quality are acquired, such as the fruit's detachment state (e.g., whether it is stably held in the end effector, or is ejected or dropped), end effector holding stability (e.g., whether the fruit slips or wobbles unstably in the grippers), and collision anomalies (e.g., whether the end effector or fruit collides with other parts of the plant or the environment during picking). The process involves identifying instances of clipping, abnormal collisions, or abnormal breakage. Finally, based on the fruit's detachment state, the broken object, the breakage location, the end effector's posture, the end effector's gripping stability, and any collision anomalies, a harvesting result is generated. This result includes whether the fruit harvesting was successful (including successful and failed harvesting; failure could be due to the breakable joint not being broken or the fruit falling and not being caught), the breakage location coordinates, the fruit ID, the end effector's harvesting action type (e.g., pulling, rotating, lifting, or combined harvesting actions), and whether clipping, abnormal collisions, abnormal breakage, or motion anomalies occurred. The harvesting result can guide the real robot in performing fruit harvesting tasks and can also be used to further train and optimize the robot's movements in subsequent simulation scenarios.

[0104] Based on the methods described above, marker points are generated at the joint fracture location, which can be used to record picking breakpoints, generate training labels, analyze failure causes, and optimize robot actions. In simulation scenarios, the root cause of picking failure can be diagnosed, and high-quality training data and reward signals can be provided for machine learning-based picking strategies (such as reinforcement learning). At the same time, the impact of different picking actions on the picking success rate can be verified, thereby realizing the automated iteration and optimization of picking strategies.

[0105] In one embodiment of step S104 above, the method further includes:

[0106] If the comparison result shows that the picking parameters are less than or equal to the preset picking parameter threshold, adjust the picking parameters and picking actions of the end effector of the simulated picking robot when performing the picking action.

[0107] Specifically, if the calculated picking parameters are still less than or equal to the preset picking parameter threshold after a period of time, meaning the fruit has not been picked, the picking parameters and picking action of the end effector can be automatically adjusted; for example, by increasing the pulling force, changing the pulling angle, or switching from a lifting action to a twisting action.

[0108] In one embodiment of step S104 above, the method further includes:

[0109] When the harvesting result is a fruit harvesting failure, obtain the reason for the failure;

[0110] Based on the reasons for the failure, adjust the picking parameters and picking actions of the end effector of the simulated picking robot when performing the picking action.

[0111] Specifically, when the final harvesting result is a fruit harvesting failure, the system can first obtain and analyze the specific reasons for the failure (for example, the failure is due to a collision with a non-target leaf or the fruit falling after detachment from the leaf). Then, based on the reasons for the failure, the system can adjust the harvesting parameters and harvesting actions for the next harvesting attempt accordingly. For example, if the failure is due to a collision with a non-target leaf, the obstacle avoidance weight of the approach path planning algorithm is adjusted; if the failure is due to the fruit falling after detachment from the leaf, the clamping strategy or the movement trajectory after harvesting is adjusted.

[0112] Based on the methods described above, the robot's behavior can be dynamically and specifically adjusted according to real-time physical feedback and post-event result analysis. This makes it possible to develop and verify intelligent harvesting strategies with higher robustness and success rate in a simulation environment. Furthermore, the path, grasping posture, and harvesting actions can be verified in simulation before the real robot enters the greenhouse or farm for testing, reducing the cost of damaging fruits, colliding with plants, or debugging equipment.

[0113] In one application scenario according to an embodiment of this application, see appendix. Figure 4 , Figure 4 This is a schematic diagram of a fruit-picking simulation scenario according to one embodiment of this application. Figure 4 As shown, taking a strawberry plant as an example, the robot's end effector 41 avoids non-target leaves 42 and approaches the target fruit 43 to perform the picking action.

[0114] Another aspect of this application provides a fruit-harvesting device, the device comprising:

[0115] A simulation scene module is constructed to build a fruit picking simulation scene and to model the fruit plants in the simulation scene to obtain a plant model. The plant model includes at least one fruit rigid body, at least one fruit stalk discrete rigid body segment and at least one stem discrete rigid body segment. Each fruit rigid body is sequentially connected to at least one fruit stalk discrete rigid body segment and at least one stem discrete rigid body segment. Adjacent discrete rigid body segments are connected by joints. The fruit rigid body is connected to the fruit stalk discrete rigid body segment closest to the fruit rigid body by a breakable joint.

[0116] The control module is used to control the end effector of the simulated picking robot to approach the target fruit, contact and clamp the target fruit or the fruit stem connected to the target fruit, and perform the picking action.

[0117] The harvesting parameter calculation module is used to calculate the harvesting parameters at the breakable joints when performing the harvesting action;

[0118] The harvesting result generation module is used to compare the harvesting parameters with the preset harvesting parameter thresholds and obtain the comparison results. If the comparison results show that the harvesting parameters are greater than the preset harvesting parameter thresholds, the joint can be broken, the breakage position markers are generated, and the harvesting results are generated. The harvesting results are used to guide the real robot to perform fruit harvesting tasks.

[0119] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of this application, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in this application and therefore will also fall within the protection scope of this application.

[0120] Those skilled in the art will understand that all or part of the processes in the method of the above-described embodiment can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0121] Another aspect of this application provides a computer-readable storage medium.

[0122] In one embodiment of a computer-readable storage medium according to this application, the computer-readable storage medium can be configured to store a program for performing the fruit harvesting method of the above-described method embodiments. This program can be loaded and run by a processor to implement the above-described fruit harvesting method. For ease of explanation, only the parts related to the embodiments of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of this application. The computer-readable storage medium can be a storage device comprising various electronic devices. Optionally, in the embodiments of this application, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0123] Another aspect of this application provides an electronic device.

[0124] In one embodiment of an electronic device according to this application, the electronic device may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program that, when executed by the at least one processor, implements the methods described in any of the above embodiments. See Appendix Figure 5 , Figure 5 The example shows a memory 51 and a processor 52 connected via a bus communication connection.

[0125] In some embodiments of this application, the electronic device may further include at least one sensor for sensing information. The sensor is communicatively connected to any type of processor mentioned in this application. The processor communicates with the sensor to perform the methods described in any of the above embodiments. (The electronic device described in this application may be, but is not limited to, mobile phones, tablets, desktop computers, laptops, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc., and this application does not limit this.)

[0126] The technical solution of this application has been described above with reference to one embodiment shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. Without departing from the principles of this application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of this application.

Claims

1. A method for harvesting fruit, characterized in that, The method includes: A fruit harvesting simulation scenario is constructed, and the fruit plants in the simulation scenario are modeled to obtain a plant model. The plant model includes at least one fruit rigid body, at least one fruit stalk discrete rigid body segment, and at least one stem discrete rigid body segment. Each fruit rigid body is sequentially connected to at least one fruit stalk discrete rigid body segment and at least one stem discrete rigid body segment. Adjacent discrete rigid body segments are connected by joints. The fruit rigid body is connected to the fruit stalk discrete rigid body segment closest to the fruit rigid body by a breakable joint. The end effector of the simulated picking robot is controlled to approach the target fruit, contact and clamp the target fruit or the fruit stalk connected to the target fruit, and perform the picking action. When performing the picking action, the picking parameters at the breakable joint are calculated; The picking parameters are compared with a preset picking parameter threshold to obtain a comparison result. If the comparison result is that the picking parameters are greater than the preset picking parameter threshold, the fracture joint breaks, a fracture location marker is generated, and a picking result is generated. The picking result is used to guide the real robot to perform the fruit picking task.

2. The method according to claim 1, characterized in that, The construction of a fruit harvesting simulation scenario and the modeling of the fruit plants in the simulation scenario to obtain plant models include: A fruit-picking simulation scenario is constructed. For each fruit plant in the simulation scenario, the fruit is constructed as a rigid body, the pedicel is constructed as at least one discrete rigid body segment of the pedicel connected in sequence, the stem is constructed as at least one discrete rigid body segment of the stem connected in sequence, and a collision body is constructed outside each discrete rigid body segment of the pedicel and each discrete rigid body segment of the stem to enclose the discrete rigid body segment. A joint chain is configured for a fruit rigid body, at least one fruit stalk discrete rigid body segment and at least one stem discrete rigid body segment connected in sequence, wherein a joint connection is configured between adjacent discrete rigid body segments and a breakable joint connection is configured between the fruit rigid body and the fruit stalk discrete rigid body segment closest to the fruit rigid body. After configuration, a plant model is obtained.

3. The method according to claim 2, characterized in that, The end effector of the simulated harvesting robot approaches the target fruit, including: Obtain the visual information of the simulated harvesting robot or the spatial annotation information in the fruit harvesting simulation scene; The target fruit is located based on the visual information or the spatial labeling information, and a picking route is planned. The end effector of the simulated picking robot is controlled to approach the target fruit based on the picking route. During the approach to the target fruit, the simulated picking robot is detected in real time whether it comes into contact with the collision object, and the picking route is adjusted accordingly.

4. The method according to claim 1, characterized in that, The joint and the fractured joint are D6 joints or spring dampers, which have translational degrees of freedom, rotational degrees of freedom, and adjustable stiffness and damping.

5. The method according to claim 1, characterized in that, The preset harvesting parameter threshold includes a joint fracture force threshold or a joint fracture moment threshold. The harvesting parameter includes a joint fracture force or a joint fracture moment. If the comparison result shows that the harvesting parameter is greater than the preset parameter threshold, the fractured joint breaks, including: If the comparison result shows that either the joint fracture force or the joint fracture torque is greater than the corresponding preset threshold value, the fractureable joint breaks.

6. The method according to claim 1, characterized in that, The process of generating fracture location markers and generating harvesting results includes: The fracture location marker is generated at the spatial location of the fracture joint, and the fracture time, fracture object, fracture location, picking parameters at the time of fracture, and end effector attitude are recorded. Acquire the fruit detachment status, end effector clamping stability, and collision anomalies; The picking result is generated based on the fruit detachment state, fracture object, fracture location, end effector posture, end effector clamping stability, and collision anomaly. The picking result includes whether the fruit picking was successful, fracture location coordinates, fruit ID, end effector picking action type, and whether there is clipping, abnormal collision, abnormal fracture, or abnormal movement.

7. The method according to claim 6, characterized in that, The method further includes: If the comparison result is that the picking parameter is less than or equal to the preset picking parameter threshold, the picking parameters and picking action of the end effector of the simulated picking robot when performing the picking action are adjusted. And / or, When the harvesting result is a fruit harvesting failure, obtain the reason for the failure; Based on the reasons for the failure, the end effector of the simulated harvesting robot was adjusted to perform harvesting parameters and harvesting actions.

8. A fruit-harvesting device, characterized in that, The device includes: A simulation scene construction module is used to construct a fruit picking simulation scene and model the fruit plants in the simulation scene to obtain a plant model. The plant model includes at least one fruit rigid body, at least one fruit stalk discrete rigid body segment and at least one stem discrete rigid body segment. Each fruit rigid body is sequentially connected to at least one fruit stalk discrete rigid body segment and at least one stem discrete rigid body segment. Adjacent discrete rigid body segments are connected by joints. The fruit rigid body is connected to the fruit stalk discrete rigid body segment closest to the fruit rigid body by a breakable joint. The control module is used to control the end effector of the simulated picking robot to approach the target fruit, contact and clamp the target fruit or the fruit stalk connected to the target fruit, and perform the picking action. The picking parameter calculation module is used to calculate the picking parameters at the breakable joint when performing the picking action; The harvesting result generation module is used to compare the harvesting parameters with a preset harvesting parameter threshold to obtain a comparison result. If the comparison result is that the harvesting parameters are greater than the preset harvesting parameter threshold, the fracture joint breaks, a fracture location marker is generated, and a harvesting result is generated. The harvesting result is used to guide the real robot to perform the fruit harvesting task.

9. An electronic device, characterized in that, The electronic device includes at least one processor and a memory communicatively connected to the at least one processor; The memory stores a computer program, which, when executed by the at least one processor, implements the fruit harvesting method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by a processor to perform the fruit harvesting method according to any one of claims 1 to 7.

Images