Article retrieval method, apparatus, and robot

Abstract

This application relates to the field of warehousing and logistics technology, and discloses a method, apparatus, and robot for picking up items. The method includes: acquiring scene information corresponding to the target container during the process of the robot's picking and placing device picking up a target object from the target container; in response to a failure event of the picking and placing device picking up the target object, determining candidate pushing objects corresponding to the target object from multiple items in the target container; determining candidate pushing strategies corresponding to each candidate pushing object based on the scene information and the geometric features of each candidate pushing object; wherein, the candidate pushing strategy includes pushing parameters and a first score, the first score being used to indicate the execution priority of each candidate pushing strategy; determining a target pushing strategy from multiple candidate pushing strategies based on the first score in each candidate pushing strategy, and controlling the picking and placing device to execute the target pushing strategy to pick up the target object. This application can improve the accuracy of robot picking up items and the efficiency of task execution.

Description

Technical Field

[0001] This application relates to the field of warehousing and logistics technology, and in particular to a method, apparatus and robot for picking up and retrieving goods. Background Technology

[0002] In automated picking scenarios in warehousing, logistics, and industrial manufacturing, robots are typically equipped with robotic arms and end effectors (such as grippers or suction cups) to pick up target goods from containers such as bins and shelves. In actual working environments, goods are often densely stacked and randomly placed, and single picking actions often fail due to poor object posture, mutual obstruction, or limited picking space, affecting task execution efficiency. Summary of the Invention

[0003] To address the aforementioned problems, embodiments of this application provide a method, apparatus, and robot for retrieving items, which can improve the success rate of the robot in retrieving items and increase task execution efficiency. Specifically, embodiments of this application disclose the following technical solutions: A first aspect of this application provides a method for retrieving an object. The method includes: during the process of a robot's pick-and-place device retrieving a target object from a target container, acquiring scene information corresponding to the target container; wherein the scene information includes spatial distribution information of multiple items in the target container. In response to a failure event of the pick-and-place device to retrieve the target object, determining candidate pushing objects corresponding to the target object from among the multiple items in the target container. Based on the scene information and the geometric features of each candidate pushing object, determining candidate pushing strategies corresponding to each candidate pushing object; wherein the candidate pushing strategy includes pushing parameters and a first score, the pushing parameters including at least the candidate pushing object, a first pushing direction corresponding to the candidate pushing object, a first contact point, and a first pushing distance, and the first score indicating the execution priority of each candidate pushing strategy. Based on the first score in each candidate pushing strategy, determining a target pushing strategy from among the multiple candidate pushing strategies, and controlling the pick-and-place device to execute the target pushing strategy to retrieve the target object.

[0004] In some embodiments, the method further includes: responding to a failure event of the pick-and-place device to pick up a target object, determining at least one estimated pick-up pose corresponding to the target object based on scene information. The at least one estimated pick-up pose and the first point cloud data corresponding to the target container are input into a pick-up evaluation module in a preset evaluation model to obtain a feasibility score for each estimated pick-up pose; wherein the feasibility score is used to indicate the probability that the pick-and-place device can successfully pick up the target object from the target container based on the estimated pick-up pose. If the feasibility score corresponding to the target pick-up pose is greater than or equal to a preset feasibility threshold among the at least one estimated pick-up pose, the pick-and-place device is controlled to pick up the target object using the target pick-up pose.

[0005] In some embodiments, the method further includes: when the feasibility scores of at least one estimated pick-up pose are all less than a preset feasibility threshold, determining at least one estimated pushing action corresponding to the target object based on scene information; wherein each estimated pushing action includes at least an estimated pushing object corresponding to the target object, and a second pushing direction, a second contact point, and a second pushing distance corresponding to the estimated pushing object. Each estimated pushing action and the second point cloud data corresponding to the target container are input into the pushing evaluation module in a preset evaluation model to obtain an effectiveness score for each estimated pushing action; wherein the effectiveness score indicates the probability that the target object will become pickable after the pick-up / placement device executes the estimated pushing action. Based on the effectiveness scores of each pushing action, a target pushing action is determined from at least one estimated pushing action, and the pick-up / placement device is controlled to execute the target pushing action.

[0006] In some embodiments, after the control of the pick-up and place device to perform the target pushing action, the method further includes: obtaining updated scene information corresponding to the target container; and based on the updated scene information, re-outputting the feasibility score corresponding to the target object through the pick-up and evaluation module until the feasibility score of the target object exceeds a preset feasibility threshold.

[0007] In some embodiments, after determining the candidate pushing strategies corresponding to each candidate pushing object, the method further includes: inputting each candidate pushing strategy and the second point cloud data corresponding to the target container into the pushing evaluation module in a preset evaluation model to obtain a second score corresponding to each candidate pushing strategy; wherein, the second score is used to indicate the probability that the target object becomes a pickable state after the pick-and-place device executes the candidate pushing strategy. Based on the first score in each candidate pushing strategy, determining the target pushing strategy among multiple candidate pushing strategies includes: determining the target score of each candidate pushing strategy based on the first score and the second score; and determining the target pushing strategy among the candidate pushing strategies based on the target score of each candidate pushing strategy.

[0008] In some embodiments, after the above-described control pick-and-place device performs the target pushing action, the method further includes: determining result data of the pick-and-place device performing the target pushing action; adjusting a preset adjustable parameter based on the result data; wherein the preset adjustable parameter includes at least a preset pushing ratio factor for determining the first pushing distance.

[0009] In some embodiments, the method further includes: in response to a failure event of the pick-up and place device to pick up the target object, determining evaluation parameters corresponding to the current scenario; wherein the evaluation parameters include at least scenario complexity, geometric features of the target object, execution success rate of a preset evaluation model, and confidence level of the preset evaluation model; the execution success rate is used to indicate the probability that the pick-up and place device can successfully pick up the target object based on the preset evaluation model; the confidence level is used to indicate the probability that the feasibility score and / or validity output by the preset evaluation model is higher than a preset score threshold. Based on the evaluation parameters corresponding to the current scenario, determining that the current scenario satisfies a first constraint or a second constraint. If the current scenario satisfies the first constraint, performing the step of determining a candidate push object corresponding to the target object among multiple items in the target container. If the current scenario satisfies the second preset condition, performing the step of determining at least one estimated pick-up pose corresponding to the target object based on scenario information.

[0010] In some embodiments, the above-mentioned method of determining a candidate push object corresponding to the target object among multiple items in the target container in response to a failure event of the pick-and-place device failing to pick up the target object includes: determining failure information corresponding to the failure event in response to the failure event of the pick-and-place device failing to pick up the target object; wherein the failure information includes at least one of an error code and / or an obstruction item index. If the failure information indicates that the pick-and-place device has experienced a path collision and there is an obstruction item corresponding to the path collision, the obstruction item is determined as a candidate push object. If the failure information indicates that the pick-and-place device has not experienced a path collision or there is no obstruction item, a candidate push object corresponding to the target object is determined among multiple items based on scene information.

[0011] In some embodiments, determining the candidate pushing object corresponding to the target object among multiple items in the target container includes: determining the pose information of each item among the multiple items based on scene information, and determining the instability score corresponding to each item based on the size information and / or pose information of each item; determining the surrounding spatial information of the location of each item, and determining the perturbability score corresponding to each item based on the surrounding spatial information of the location of each item; identifying items whose instability score exceeds a first preset threshold, or whose perturbability score exceeds a second preset threshold, as candidate items, and determining the basic score of each candidate item based on the instability score and the perturbability score; and determining the candidate pushing object among the candidate items based on the basic score of each candidate item.

[0012] In some embodiments, the process of determining candidate push targets based on the baseline scores of each candidate item includes: determining an occlusion score for each candidate item based on the degree to which it occludes other items; determining a centering score for each candidate item based on the distance between each candidate item and the center position of the target container; determining a comprehensive score for each candidate item based on its baseline score, occlusion score, and centering score; and identifying at least one candidate item whose comprehensive score meets preset push conditions as a candidate push target.

[0013] In some embodiments, the method further includes: if the overall scores of all candidate items do not meet the preset promotion conditions, the candidate item with the highest overall score is determined as the candidate promotion target.

[0014] In some embodiments, when the pushing parameter in the candidate pushing strategy is a first pushing direction, determining the candidate pushing strategy corresponding to each candidate pushing object based on scene information and the geometric features of each candidate pushing object includes: determining the size of the surrounding space and the crowding degree of surrounding objects for each candidate pushing object based on scene information; and determining at least one first pushing direction for each candidate pushing object and a direction score corresponding to each first pushing direction based on the size of the surrounding space and the crowding degree of surrounding objects.

[0015] In some embodiments, when the pushing parameter in the candidate pushing strategy is the first contact point, determining the candidate pushing strategy corresponding to each candidate pushing object based on scene information and the geometric features of each candidate pushing object includes: determining the minimum distance between the surface of each candidate pushing object and the surface of the target container based on scene information; determining the position area of ​​each candidate pushing object in the target container based on the minimum distance between the surface of each candidate pushing object and the surface of the target container; determining the first contact point in the top area of ​​the candidate pushing object when the candidate pushing object is located in the edge area of ​​the target container; and determining the first contact point in the side area of ​​the candidate pushing object when the candidate pushing object is located in the middle area of ​​the target container.

[0016] In some embodiments, when the pushing parameter in the candidate pushing strategy is a first pushing distance, determining the candidate pushing strategy corresponding to each candidate pushing object based on scene information and the geometric features of each candidate pushing object includes: determining the height of the first contact point of each candidate pushing object, the vertical height of each candidate pushing object, and the horizontal dimension of each candidate pushing object based on the geometric features of each candidate pushing object; determining the moving distance of each candidate pushing object based on the height of the first contact point of each candidate pushing object, the vertical height of each candidate pushing object, and the horizontal dimension of each candidate pushing object; determining the preset minimum pushing distance as the first pushing distance when the moving distance is less than a preset minimum pushing distance; determining the maximum permitted pushing distance as the first pushing distance when the moving distance is greater than a preset maximum permitted pushing distance; and determining the moving distance as the first pushing distance when the moving distance is greater than the preset minimum pushing distance and less than the preset maximum permitted pushing distance.

[0017] In some embodiments, the above-mentioned moving distance satisfies formula (1): D1 = h × (W / H) × α. The above-mentioned first pushing distance satisfies the following formula (2): D2 = clamp (D1, Dmin, Dmax). Wherein, D1 is the moving distance corresponding to the candidate pushing object, h is the height of the first contact point of the candidate pushing object, W is the horizontal dimension of the candidate pushing object, H is the vertical height of the candidate pushing object, α is the preset pushing ratio factor; D2 is the first pushing distance of the candidate pushing object, Dmin is the preset minimum pushing distance, and Dmax is the maximum permissible distance.

[0018] In some embodiments, the method further includes: determining a first gap; wherein the first gap is a spatial gap along a first pushing direction, with the size of the end effector of the pick-and-place device as the corridor width; determining a second gap; wherein the second gap is a spatial gap along the first pushing direction and in a direction perpendicular to the first pushing direction, with the size of the candidate push object as the corridor width; and determining the smaller of the first gap and the second gap as the maximum permissible distance.

[0019] In some embodiments, determining the candidate pushing strategy corresponding to each candidate pushing object includes: determining the pushing parameters corresponding to the candidate pushing object based on the candidate pushing object, the first pushing direction corresponding to the candidate pushing object, the first contact point corresponding to the first pushing direction, and the first pushing distance corresponding to the first contact point; determining the first score in the candidate pushing strategy based on the comprehensive score corresponding to the candidate pushing object and the direction score of the first pushing direction corresponding to the candidate pushing object; and determining the candidate pushing strategy corresponding to the candidate pushing object based on the pushing parameters and the first score.

[0020] In some embodiments, determining the target pushing strategy from multiple candidate pushing strategies based on the first score of each candidate pushing strategy includes: performing feasibility verification on each candidate pushing strategy sequentially based on the first score of each candidate pushing strategy; and determining the first candidate pushing strategy that passes the feasibility verification as the target pushing strategy. The feasibility verification includes kinematic feasibility verification and / or collision safety verification. Kinematic feasibility verification includes at least: inverse kinematics solution of the path point sequence executing the candidate pushing strategy, joint space path interpolation collision detection, and singular configuration detection. Collision safety verification includes at least: verifying the rigid structure of the pick-and-place device using a first collision tolerance, verifying the contact pose of the pick-and-place device during the execution of the candidate pushing strategy using a second collision tolerance, and verifying each candidate pushing object using a third collision tolerance.

[0021] In some embodiments, the method further includes: if multiple candidate driving strategies fail the feasibility check, determining that there is no effective driving strategy, and instructing the output device to output an abnormal prompt message.

[0022] In some embodiments, the method further includes: performing kinematic feasibility verification and collision safety verification on each candidate push strategy in sequence based on the first score of each candidate push strategy; and, if multiple candidate push strategies fail the kinematic feasibility verification and collision safety verification, performing a step of determining at least one estimated take-up pose corresponding to the target object based on scene information.

[0023] A second aspect of this application provides an item retrieval device, including an acquisition module, a determination module, and a control module. The acquisition module is configured to: acquire scene information corresponding to a target container; wherein the scene information includes spatial distribution information of multiple items in the target container. The determination module is configured to: during the process of a robot's retrieval device retrieving a target object from the target container, in response to a failure event of the retrieval device retrieving the target object, determine candidate pushing objects corresponding to the target object among the multiple items in the target container; based on the scene information and the geometric features of each candidate pushing object, determine candidate pushing strategies corresponding to each candidate pushing object; wherein the candidate pushing strategy includes pushing parameters and a first score, the pushing parameters including at least the candidate pushing object, a first pushing direction corresponding to the candidate pushing object, a first contact point, and a first pushing distance, and the first score indicating the execution priority of each candidate pushing strategy; and based on the first score of each candidate pushing strategy, determine a target pushing strategy among the multiple candidate pushing strategies. The control module is configured to: control the retrieval device to push the candidate pushing objects corresponding to the target pushing strategy in the target container according to the target pushing strategy, in order to retrieve the target object.

[0024] A third aspect of this application provides a robot, including a control device and a pick-and-place device. The control device is configured to: acquire scene information corresponding to a target container; wherein the scene information includes spatial distribution information of multiple items in the target container; during the process of the robot's pick-and-place device retrieving a target object from the target container, in response to a failure event of the pick-and-place device retrieving the target object, determine candidate push objects corresponding to the target object among the multiple items in the target container; based on the scene information and the geometric features of each candidate push object, determine candidate push strategies corresponding to each candidate push object; wherein each candidate push strategy includes push parameters and a first score, the push parameters including at least the candidate push object, a first push direction corresponding to the candidate push object, a first contact point, and a first push distance, and the first score indicating the execution priority of each candidate push strategy; based on the first score of each candidate push strategy, determine a target push strategy among the multiple candidate push strategies, and generate a target execution instruction based on the target push strategy. The pick-and-place device is configured to: based on the target execution instruction, push the candidate push objects corresponding to the target push strategy in the target container according to the target push strategy to retrieve the target object.

[0025] A fourth aspect of this application provides an electronic device, including: a processor and a memory, the memory being used to store computer-executable instructions; the processor being used to read the instructions from the memory and execute the instructions to implement the item retrieval method described in the first aspect above.

[0026] A fifth aspect of this application provides a computer-readable storage medium storing computer program instructions, which, when read by a computer, execute the item retrieval method described in the first aspect.

[0027] A sixth aspect of this application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions that, when executed by a computer, cause the computer to perform the article-taking method described in the first aspect.

[0028] The seventh aspect of this application provides a computer program that, when executed by a processor, can implement the item retrieval method described in the first aspect.

[0029] The item-grabbing method, apparatus, and robot provided in this application, when the robot fails to grab a target object, determine candidate pushing objects corresponding to the target object in the target container based on scene information corresponding to the target container, and determine candidate pushing strategies corresponding to each candidate pushing object. Then, based on the first score of each candidate pushing strategy and a feasibility check, the optimal target pushing strategy can be selected from multiple candidate pushing strategies to control the picking and placing device to execute the target pushing strategy before grabbing the target object again. Compared with related technologies that involve repeated grabbing and manual processing after failure, the embodiments of this application can improve the accuracy and efficiency of item grabbing, thereby improving the robot's task execution efficiency. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A schematic diagram of a warehousing system provided in an embodiment of this application; Figure 2 This is a schematic diagram of a pick-and-place device provided in an embodiment of this application; Figure 3 A schematic diagram illustrating a method for retrieving an item according to an embodiment of this application; Figure 4 A schematic diagram illustrating another method for retrieving an item provided in an embodiment of this application; Figure 5 A schematic diagram illustrating yet another method for retrieving an item, as provided in an embodiment of this application; Figure 6 A schematic diagram illustrating another method for retrieving an item provided in an embodiment of this application; Figure 7 A schematic diagram of a preset evaluation model provided in an embodiment of this application; Figure 8 A schematic diagram illustrating another method for retrieving an item provided in an embodiment of this application; Figure 9 A schematic diagram illustrating another method for retrieving an item provided in an embodiment of this application; Figure 10 A schematic diagram illustrating another method for retrieving an item provided in an embodiment of this application; Figure 11 A schematic diagram illustrating another method for retrieving an item provided in an embodiment of this application; Figure 12 This is a schematic diagram of an item retrieval device provided in an embodiment of this application; Figure 13 A schematic diagram of a robot provided for an embodiment of this application; Figure 14 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0032] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, and to make the above-mentioned objectives, features and advantages of the embodiments of the present invention more apparent and understandable, the technical solutions in the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0033] Figure 1 This is a schematic diagram of a warehousing system provided as an embodiment of this application. Figure 1 As shown, the warehousing system 100 includes multiple carriers 10, robots 20, workstations 30, and pick-and-place devices 40.

[0034] In some examples, the carrier 10 can be a shelf, pallet, trolley, cage trolley, etc. The carrier 10 includes multiple storage locations (also called storage positions). The storage positions can be rectangular prism-shaped accommodating spaces, and the multiple storage positions on the carrier 10 can be neatly arranged along the length, width, and height directions of the carrier 10.

[0035] For example, the storage space on the carrier 10 is used to hold inventory objects such as goods or containers. The container can be a matching product specially designed for the carrier 10, or it can be a regular cargo box (also called a material box), or it can be the packaging of the goods (also called the original box). This application embodiment does not limit this. This application embodiment takes the inventory object placed on the carrier 10 as a container as an example for illustrative explanation.

[0036] For example, the warehousing system 100 includes multiple robots 20 and at least one workstation 30. The workstation 30 can pick the goods required for an order from the inventory container into the order container.

[0037] In some examples, robot 20 may include a handling robot 21, which can be used to handle goods, containers, or vehicles. For example, handling robot 21 may be an Automated Guided Vehicle (AGV). For example, handling robot 21 may transport inventory objects to workstation 30 for picking, or it may transport inventory objects to a corresponding destination to perform picking or sorting operations.

[0038] In some examples, robot 20 may also include picking robot 22, which can retrieve corresponding goods from a container. For example, picking robot 22 may be an embodied intelligent robot.

[0039] In some embodiments, the warehousing system 100 further includes a plurality of pick-and-place devices 40. The pick-and-place devices 40 may be independently installed or installed on the picking robot 22. When the pick-and-place devices 40 are installed on the picking robot 22, they may be in the form of robotic arms; this embodiment does not limit this.

[0040] Figure 2 This is a schematic diagram of a pick-and-place device provided in an embodiment of this application.

[0041] like Figure 2 As shown, the pick-and-place device 40 includes a base 41, a first connecting arm 44 connected to the base 41 via a first driving member 43, and an end effector 42.

[0042] In some examples, the end effector 42 can be a gripper, claw, or suction cup connected to the pick-and-place device 40 for picking up goods from a container. This application embodiment does not limit the specific structure of the end effector 42. For example, one end effector 42 or multiple end effectors 42 can be provided on the pick-and-place device 40. It should be noted that... Figure 2 The following is an example of a device 40 equipped with an end effector 42.

[0043] In some examples, the first driving member 43 can be a motor, which can directly or indirectly drive the first connecting arm 44 to move in three-dimensional space (or two-dimensional space).

[0044] In some examples, the pick-and-place device 40 further includes a second connecting arm 45 for connecting the first connecting arm 44 and the end effector 42, and the second connecting arm 45 is rotatable relative to the first connecting arm 44. For example, the first connecting arm 44 and the second connecting arm 45 can be rotated relative to each other by a second drive member 46.

[0045] like Figure 2 As shown, one end of the end effector 42 may also be equipped with a third drive member 47 and a fourth drive member 48 connected to the third drive member 47. The third drive member 47 can drive the end effector 42 to move vertically, and the fourth drive member 48 can drive the end effector 42 to rotate by a corresponding angle. For example, under the drive of the third drive member 47 and the fourth drive member 48, the end effector 42 can pick up the goods required for the order from the container, and transfer the picked-up goods to the target location by moving the first connecting arm 44 and the second connecting arm 45.

[0046] In some examples, during operations where the robot's pick-and-place device retrieves items from containers such as bins and shelves, the items are often densely stacked and haphazardly arranged. The pick-and-place device frequently fails to perform its retrieval action due to poor item positioning, mutual obstruction, or limited space, affecting the robot's item retrieval efficiency and reducing its overall task execution efficiency. It should be noted that this application embodiment uses the example of the pick-and-place device 40 being mounted on a robot (such as a picking robot) for illustrative purposes.

[0047] To address the aforementioned issues, this application provides an item retrieval method. This method employs a perturbation rule of pushing followed by retrieving to retrieve items. First, items meeting the pushing requirements are identified within the target container. Then, the corresponding pushing action for each item to be pushed is determined. After the retrieval device performs the pushing operation on the item to be pushed, the retrieveable state of the target object can be altered, increasing the success rate of retrieving the target object from the target container. Compared to related technologies that involve repeated retrieval after a failed attempt, this application's embodiment improves the robot's item retrieval efficiency and task execution efficiency.

[0048] For example, the item retrieval method provided in this application embodiment can be implemented by a controller. The controller can be a control device in a warehousing system or a control module in a robot; this application embodiment does not limit this.

[0049] In some examples, where the method of retrieving the item is implemented by a control device, the control device can be a server or a terminal device, or other device that deploys a Robot Management System (RMS). The terminal device can include at least one of a personal computer, laptop computer, smartphone, tablet computer, and portable wearable device; the server can include a standalone server or a server cluster consisting of multiple servers, and this application embodiment does not limit this.

[0050] In some examples, where the method of picking up the item is implemented by a control module in the robot, the robot can be the aforementioned picking robot 22, i.e., an embodied intelligent robot. The robot may include a control module and a pick-and-place device, which may be a robotic arm on the robot for picking up and placing items.

[0051] It should be noted that the following embodiments can be illustrated by taking an example where the item picking method is implemented by a robot and the picking and placing device is a robotic arm installed on the robot.

[0052] In some embodiments, the item retrieval method provided in this application can be implemented through three different schemes (such as Scheme 1, Scheme 2 and Scheme 3).

[0053] In some examples, Scheme 1 can be a deterministic method based on geometrical physical rules. Scheme 1 can generate candidate pushing strategies for each item based on geometrical physical rules, filter executable pushing strategies through kinematics and collision checks, and execute the optimal strategy after prioritization to obtain the perturbation planning scheme for the target object.

[0054] In some examples, Scheme 2 can be a model-based approach using point cloud deep learning. Scheme 2 can be based on a point cloud deep learning model, using a take-up evaluation module to determine whether the target object can be directly taken, and if it cannot be taken, using a push evaluation module to evaluate the effectiveness of the predicted push action, thus forming an evaluation-push-re-evaluation-take-up perturbation planning scheme.

[0055] In some examples, Scheme 3 can be a combination of Scheme 1 and Scheme 2. Scheme 3 can combine Scheme 1 and Scheme 2 to form a composite perturbation planning scheme. Scheme 3 includes, but is not limited to, three combination methods: a combination of candidate generation and model verification, a combination of model result-driven adaptive adjustment of rule parameters, and a combination of hierarchical invocation with each other as a fallback.

[0056] The specific implementation methods of the above three schemes will be explained in detail below with reference to the accompanying drawings.

[0057] Figure 3 This is a schematic diagram illustrating a method for retrieving an item according to an embodiment of the present invention. The following is in conjunction with... Figure 3 The specific implementation process of Scheme 1 (i.e., the deterministic method based on geometric and physical rules) is explained below. For example... Figure 3 As shown, the method for taking the item includes steps 310 to 350 as shown below.

[0058] Step 310: During the process of the robot's pick-up and place device taking the target object from the target container, obtain the scene information corresponding to the target container.

[0059] In some examples, the picking and placing device can be a mechanism on a robot used to perform item picking operations, including but not limited to a robotic arm and an end effector, which may include grippers, suction cups, etc. The target container can be a container that can store items, such as a bin or pallet in a warehousing scenario. The target container may contain multiple items, and the multiple items may include the target object to be picked up. The target object can be any item to be picked up in the target container, and this application embodiment does not limit this.

[0060] For example, the scene information corresponding to the target container may include the spatial distribution information of multiple items in the target container, such as three-dimensional spatial distribution information. For instance, the spatial distribution information may include the position coordinates, orientation angle, and size of each item in three-dimensional space, as well as the relative positional relationship and stacking relationship between the items.

[0061] In some examples, scene information may include point cloud data or depth images acquired by a data acquisition device. For example, the data acquisition device may be a vision sensor, such as a 3D vision sensor, a laser profile scanner, a binocular vision camera, or a time-of-flight (ToF) camera. The data acquisition device may be positioned above the target container or on the robot's pick-and-place device. This application does not limit the location or type of the data acquisition device.

[0062] In other examples, the scene information may also include attribute information of each item within the target container. For example, the type, size, and weight of the item. This attribute information can be obtained through instance segmentation and model matching of point cloud data, or it can be pre-acquired from the control device in the warehousing system; this embodiment of the application does not limit this approach.

[0063] Step 320: In response to a failure event of the pick-up and drop-off device to pick up the target object, determine the candidate push object corresponding to the target object among multiple items in the target container.

[0064] In some examples, step 320 is used to determine which items (i.e., candidate push objects) need to be pushed in order to retrieve the target object. That is, to determine which one or more items should be pushed to improve the conditions for retrieving the target object and make the success rate of retrieving the target object higher.

[0065] For example, a pick-and-place device may encounter a pick-and-place failure event when retrieving a target object from a target container. The causes of this failure include, but are not limited to: the pick-and-place device failing to detect an available pick-and-place point corresponding to the target object; or, a path collision occurring during the pick-and-place action; or, the pick-and-place device failing to find a feasible joint angle configuration (i.e., inverse kinematics unreachable) when performing inverse kinematics solution. This application embodiment does not limit the causes of the failure event.

[0066] For example, path collisions during the picking and placing process can include: the robot's robotic arm or end effector colliding with other items or the container wall within the target container during the movement path to pick up the target object, thus preventing it from reaching the target object.

[0067] In some examples, in densely stacked scenarios, the pick-and-place mechanism easily fails to retrieve the target object, and related techniques that simply repeat the same pick-up action will repeatedly fail. Therefore, it is necessary to change the state of the items in the target container. Pushing operations can effectively change the scene state. For example, by pushing an item, its own pose or its relative position to other items can be changed, thereby creating better conditions for retrieving the target object. Therefore, to implement pushing operations, one or more candidate push objects need to be identified from the multiple items in the target container; that is, items that, when pushed, may improve the retrieveability of the target object.

[0068] In some embodiments, determining the candidate push object in step 320 above may include: in response to a failure event of the pick-up and place device failing to pick up the target object, determining the failure information corresponding to the failure event; if the failure information indicates that the pick-up and place device has a path collision and there is an obstructing item corresponding to the path collision, determining the obstructing item as a candidate push object; otherwise, based on scene information, determining the candidate push object corresponding to the target object among multiple items.

[0069] For example, if the robot detects that the pick-and-place device has failed to pick up the target object, it can obtain the failure information corresponding to the failure event. The failure information may include an error code and / or an index of the obstructing item.

[0070] In some examples, the error code can be a predefined identifier for a failed pick-up attempt, with different error codes corresponding to different reasons for failure. Reasons for failure can include those mentioned above such as pick-up point not found, path collision, inverse kinematics unreachable, or other reasons. The obstructing item index is used to indicate the obstructing item that caused the current pick-up failure. Each item can be pre-configured with a unique index (ID).

[0071] For example, if the failure message indicates that the pick-up / placement device has experienced a path collision and there is an obstruction corresponding to the path collision, it indicates that the current pick-up failure is due to a path collision, and the obstacle causing the collision is an obstruction. The obstruction can be one item in the target container or multiple items; this embodiment does not limit this. If the obstruction is determined based on its index, it can be used as a candidate to be pushed, and a pushing operation can be subsequently performed on that candidate.

[0072] For example, when a robotic arm approaches a target object, its end effector collides with another item inside the target container. The collision detection module (such as a force sensor or a visual collision detection module) can record the index of the colliding item. After reading the failure information, the robot can determine that the colliding item is an obstruction and identify it as a candidate to be pushed.

[0073] For example, if the failure message does not indicate a path collision or the absence of obstructing objects, the candidate push object corresponding to the target object can be further determined from among multiple objects based on scene information.

[0074] In some examples, the robot can determine a comprehensive score for each item based on its geometric features (including but not limited to the size, shape, posture, and position of each item) and scene information (including but not limited to the size of the space around each item and the relative positional relationship between each item and other items), and then identify candidate objects to be pushed based on the comprehensive score. The comprehensive score is used to evaluate the likelihood or magnitude of the benefit of pushing the candidate object to improve the reachability of the target object.

[0075] In some examples, the comprehensive scoring factors for items include, but are not limited to: the size information of each item, the pose information of each item, and the surrounding space information of each item's location. For example, smaller items are easier to push and require less force; or, upright items are more likely to change their posture after being pushed than flat items, resulting in a more significant change in the scene state after being pushed; or, the more abundant the surrounding space, the less likely the item is to collide with other items when pushed, and the safer the pushing operation is.

[0076] In some examples, the robot can determine candidate items to push based on the overall score of each item. For example, one or more items with high overall scores can be identified as candidate items to push, or items with overall scores exceeding a preset threshold can be identified as candidate items to push.

[0077] It should be noted that the specific method for determining the candidate push object will be described in detail in subsequent embodiments (such as subsequent steps 321 to 324), and will not be repeated here.

[0078] The item retrieval method provided in this application identifies the item that caused the retrieval failure or the item that is worth pushing when the retrieval device fails to retrieve the target object, and determines these items as candidate push objects. By determining candidate push objects, the method avoids blindly pushing arbitrary items, thus improving the targeting and effectiveness of the pushing operation.

[0079] Step 330: Based on the scene information and the geometric features of each candidate push object, determine the candidate push strategy corresponding to each candidate push object.

[0080] In some examples, after determining which item to push (i.e., the candidate push object) in step 320, step 330 can further determine how to push it. Step 330 can generate one or more specific pushing action plans (i.e., candidate pushing strategies) for each candidate push object, and evaluate and score each candidate pushing strategy in order to select the optimal plan in subsequent steps.

[0081] For example, for each candidate push object, one or more candidate push strategies can be generated. Each candidate push strategy can correspond to one candidate push object, and one candidate push object can correspond to one or more candidate push strategies. The candidate push strategy can include push parameters and a first score. The push parameters include at least: the candidate push object, the first push direction corresponding to the candidate push object, the first contact point, and the first push distance.

[0082] In some examples, the first pushing direction is the direction in which the pick-and-place device applies a pushing force to the candidate object; that is, in which direction the end effector of the pick-and-place device specifically pushes the candidate object. A candidate object can correspond to at least one pushing direction, and different pushing directions can affect the pushing effect. For example, pushing towards an open area can safely move the candidate object away, while pushing towards a crowded area may cause the candidate object to collide with other objects; or, pushing towards the target object may push the candidate object towards the target object, exacerbating obstruction and hindering the retrieval of the target object.

[0083] In some examples, the first contact point is the location where the end effector of the pick-and-place device comes into contact with the candidate object to be pushed. A candidate object to be pushed can correspond to at least one contact point, and different contact points can affect the pushing effect and safety. For example, a contact point that is too high may cause the object to tip over, a pushing position that is too low may prevent the object from tipping over, and an improper pushing position may cause the end effector to slip during the pushing process.

[0084] In some examples, the first pushing distance is the distance the end effector of the pick-and-place device moves along the pushing direction during the pushing process. The pushing distance also affects the pushing effect and safety; for example, if the pushing distance is too short, the candidate object to be pushed may not undergo the expected pose change, and if the pushing distance is too long, a collision may occur.

[0085] For example, for each candidate pushing object, its corresponding first pushing direction, first contact point, and first pushing distance are determined, thereby obtaining the pushing parameters of the pushing operation corresponding to each candidate pushing object. Since the pushing parameters are different, the disturbance effect achieved by the corresponding pushing operation is different, and the expected effect that each pushing operation may achieve can be represented by a first score.

[0086] In some examples, a candidate push strategy can correspond to a first score. The higher the first score, the better the expected effect of the push operation corresponding to the candidate push strategy is, and therefore it can be executed first. For example, the first score is related to the selection of the first push direction, the first contact point, and the first push distance.

[0087] It should be noted that the specific method for determining the candidate push strategy will be described in detail in subsequent embodiments (such as subsequent steps 331 to 334), and will not be repeated here.

[0088] The item-taking method provided in this application can generate one or more candidate pushing strategies for each candidate pushing object. Each candidate pushing strategy includes the pushing parameters required for the pushing operation (i.e., which one to push, which direction to push, where to push, and how far to push), and generates a first score reflecting the quality of each candidate pushing strategy, providing data support for the selection of subsequent pushing strategies.

[0089] Step 340: Based on the first score among the candidate push strategies, determine the target push strategy among the multiple candidate push strategies.

[0090] In some examples, step 340 may select the optimal or most suitable strategy from the multiple candidate push strategies generated in step 330 as the final strategy to be executed (i.e., the target push strategy). The selection criterion for the target push strategy is a comprehensive consideration of the success probability of pushing the strategy and the effect after pushing it. The first score can provide a basis for the selection of the target push strategy; the higher the first score, the better the overall expected effect of the candidate push strategy, and the more likely it is to be given priority for execution.

[0091] For example, multiple candidate push strategies can be ranked according to their first scores. For instance, the ranking can be in descending order, with the candidate push strategy with the highest first score ranked first and the candidate push strategy with the lowest score ranked last.

[0092] In some examples, the top-ranked candidate push strategy can be identified as the target push strategy.

[0093] In other examples, a first score can be used to filter candidates before determining the target-driven strategy, and the feasibility of the initially selected candidate strategies can be verified to ensure that the determined target-driven strategy is physically executable.

[0094] The following section explains the feasibility verification process for candidate promotion strategies.

[0095] In some embodiments, step 340 may include: performing feasibility checks on each candidate push strategy in sequence based on the first score of each candidate push strategy, and determining the first candidate push strategy that passes the feasibility check as the target push strategy.

[0096] For example, the multiple candidate push strategies determined in step 330 can be sorted in descending order from high to low, and the top preset number of candidate push strategies in the sorted list can be selected as the strategies to be verified for feasibility verification. The preset number can be set according to requirements; it can be a fixed number or a percentage of the total number of candidate push strategies.

[0097] For example, feasibility verification includes kinematic feasibility verification and / or collision safety verification. For instance, kinematic feasibility verification can be performed on the strategy to be verified, or collision safety verification can be performed on the strategy to be verified; or, both kinematic feasibility verification and collision safety verification can be performed on the strategy to be verified.

[0098] For example, kinematic feasibility verification is used to verify whether the pick-and-place device can physically execute the candidate push strategy. Kinematic feasibility verification includes at least: inverse kinematics solution of the path point sequence for executing the candidate push strategy, joint space path interpolation collision detection, and singular configuration detection.

[0099] In some examples, the inverse kinematics (IK) solution can be performed first on the path point sequence of the candidate propulsion strategy. The path point sequence includes the approach pose, pre-contact pose, contact pose, and endpoint pose. The IK solution can be performed sequentially on each pose in the path point sequence. The IK solution of the next pose is used as the initial value of the IK solution of the previous pose.

[0100] In some examples, singular configuration detection can also be performed during the inverse kinematics solution process described above. A singular configuration refers to a configuration where a joint of the robotic arm is at a specific angle, causing the robotic arm to lose degrees of freedom in a certain direction or its motion speed to tend towards infinity. When a singular configuration is detected, it can be determined that the current candidate propulsion strategy has failed the kinematic feasibility check.

[0101] In some examples, joint space path interpolation collision detection can also be performed during the inverse kinematics solution process described above. For instance, as the robotic arm moves from the starting point to the ending point of a path point sequence, intermediate points are densely inserted in the joint space, and collision detection is performed on each intermediate point. If the robotic arm at any intermediate point collides with the target container, other objects within the target container, or any obstacle, it can be determined that the current candidate pushing strategy has failed the kinematic feasibility check.

[0102] For example, the collision safety check is used to verify whether safety can be guaranteed when executing the candidate pushing strategy. The collision safety check adopts a graded collision threshold check method. The collision safety check includes at least: checking the rigid structure of the pick-and-place device using a first collision tolerance, checking the contact pose of the pick-and-place device during the execution of the candidate pushing strategy using a second collision tolerance, and checking each candidate pushing object using a third collision tolerance.

[0103] In some examples, the first collision tolerance is less than the second collision tolerance, and the second collision tolerance is less than the third collision tolerance.

[0104] For example, a first collision tolerance is used to verify the rigid structure of the pick-and-place device (such as the metal links and joints of a robotic arm). This first collision tolerance can be a zero-penetration hard constraint, meaning that the rigid structure is absolutely not allowed to penetrate any object. A second collision tolerance is used to verify the contact pose (including the pre-contact pose, contact pose, and end pose) during the execution of the candidate pushing strategy by the pick-and-place device. This second collision tolerance can be a soft constraint that allows for slight penetration, used to absorb measurement errors from the vision sensor and control errors from the robotic arm. A third collision tolerance is used to verify the deformable items in the target container (such as soft-packaged goods, sponges, rubber products, etc.). This third collision tolerance can be a deformable soft constraint that allows for a larger amount of penetration, used to simulate the contact deformation of flexible tools such as suction cups.

[0105] In some examples, among the selected strategies to be verified, some strategies may pass the feasibility check, while others may not. The strategies to be verified can be checked sequentially according to their first score, from highest to lowest, until the first strategy passes the feasibility check. This strategy is then identified as a target-driven strategy (e.g., an early stopping mechanism). The target-driven strategy can be the first, any one in the middle, or even the last strategy among the strategies to be verified; this embodiment does not limit this.

[0106] In some embodiments, if multiple candidate driving strategies fail the feasibility check, it is determined that there is no valid driving strategy, and the output device is instructed to output an abnormal prompt message.

[0107] For example, if all the selected strategies to be verified pass the feasibility check, then among the multiple candidate pushing strategies, excluding the strategy to be verified, the remaining candidate strategies are sequentially tested for feasibility according to their first score. If any candidate pushing strategy passes the feasibility check, then the first strategy to pass the feasibility check is determined as the target strategy. If all candidate pushing strategies fail the kinematic feasibility check and the collision safety check, then it is determined that the current push operation cannot improve the detachable state of the target object. In this case, a corresponding prompt message can be output, such as a prompt that there is no effective pushing strategy.

[0108] In some examples, the output device can be set on the robot or in other locations in the warehousing system. The output device can be a display device, audio device, interactive device, etc. This application embodiment does not limit the setting location and type of the output device.

[0109] The item retrieval method provided in this application can select a target pushing strategy that has both high expected effect and physical feasibility from multiple candidate pushing strategies through a first score and feasibility verification, thus ensuring the safety and effectiveness of the final pushing action.

[0110] Step 350: Control the pick-and-place device to execute the target pushing strategy to pick up the target object.

[0111] In some examples, after determining the target pushing strategy through step 340 above, step 350 can execute the pushing operation based on the pushing parameters in the target pushing strategy. That is, step 350 can transform the aforementioned planned target pushing strategy into the physical action of the picking and placing device, and change the pose or position of the candidate pushing object through the actual pushing operation, thereby creating feasible conditions for picking up the target object, and ultimately achieving the successful picking up of the target object.

[0112] For example, after determining the target pushing strategy, a corresponding control command can be generated based on the target pushing parameters in the target pushing strategy, and the control command can be sent to the pick-and-place device. The target pushing parameters include at least the target pushing object, the first target pushing direction, the first target contact point, and the first target pushing distance.

[0113] In some examples, after receiving a control command, the pick-and-place device performs the following operations according to the control command: the joints of the robotic arm move to move the end effector from its current position to the position of the target first contact point on the target push object; after reaching the target first contact point, the end effector applies a thrust along the target first push direction, and the thrust is continuously applied until the end effector moves the target first push distance along the target first push direction, and then the end effector stops applying force.

[0114] In some examples, after the pushing operation is completed, the reachability of the target object within the target container is improved. For instance, items that were originally obscuring the target object are removed, exposing it; or the target object's posture changes (e.g., from upright to lying flat), making it easier for the pick-and-place device to approach and retrieve it; or the space around the target object increases, providing sufficient operating space for the pick-and-place device. Therefore, after the pushing operation is completed, the robot can continue to control the pick-and-place device to attempt to retrieve the target object again. Since the pushing operation has improved the reachability of the target object, the pick-and-place device can now successfully retrieve the target object from the target container.

[0115] The item retrieval method provided in this application, when the robot fails to retrieve the target object, identifies candidate pushing objects corresponding to the target object in the target container based on scene information corresponding to the target container, and determines candidate pushing strategies for each candidate pushing object. Then, based on the first score of each candidate pushing strategy and a feasibility check, the optimal target pushing strategy is selected from multiple candidate pushing strategies. This allows the retrieval device to execute the target pushing strategy and retrieve the target object again. Compared to related technologies that involve repeated retrieval followed by manual processing after retrieval failure, this application embodiment can improve the accuracy and efficiency of item retrieval, thereby improving the robot's task execution efficiency.

[0116] The process of determining the candidate targets corresponding to the target target in step 320 above will be explained in detail below.

[0117] Figure 4 This is a schematic diagram illustrating another method for retrieving an item provided in an embodiment of this application. For example... Figure 4 As shown, step 320 above may include steps 321 to 324 as shown below.

[0118] Step 321: Based on scene information, determine the pose information of each item among multiple items, and based on the size information and / or pose information of each item, determine the instability score corresponding to each item.

[0119] In some examples, the pose information of each item can be extracted from the scene information. This pose information includes the item's three-dimensional position coordinates and orientation angles within the target container, i.e., how the item is positioned in space. The size information of each item can be obtained from the storage system. Size information can include the item's height and horizontal dimensions. The item's height H indicates its maximum vertical dimension (vertical height), and its horizontal dimension W indicates its maximum width or diameter on a horizontal plane (maximum horizontal dimension).

[0120] For example, an instability score is used to characterize whether an object is prone to tipping over or changing position in its current pose. The instability score can be determined based on the object's size information and / or pose information. For instance, the instability score can be determined based on the object's size information; alternatively, it can be determined based on the object's pose information; or it can be determined based on both the object's size information and pose information. For example, the overturning critical angle of an item can be determined based on its size information, and an instability score can be determined based on the overturning critical angle. The instability score is inversely proportional to the overturning critical angle; the smaller the overturning critical angle, the higher the instability score.

[0121] In some examples, the tangent of the tipping-off critical angle θ is tan(θ) = W / H, where W is the maximum horizontal dimension of the object and H is the vertical height of the object. For example, for a tall, slender object placed vertically, its tipping-off critical angle is small (i.e., a small W / H value), meaning it can tip over with a relatively small pushing force, thus exhibiting higher instability. Conversely, for a flat object placed horizontally, its tipping-off critical angle is large (a large W / H value), requiring a larger pushing force to tip over, thus exhibiting lower instability.

[0122] For example, an instability score can be determined based on the pose information of an object. This instability score can be corrected based on the angle between the object's central axis and the vertical direction. A larger angle results in a larger correction factor and a higher instability score. For instance, when an object is tilted, its central axis deviates from the vertical direction by a certain angle, indicating that the object is already partially unstable and may overturn even with a small external force; therefore, its instability score should be significantly increased.

[0123] In some examples, the instability score can be set between 0 and 1, where 0 represents perfectly stable and 1 represents extremely unstable. A higher instability score indicates a more unstable item, and therefore a greater likelihood of it being selected as a candidate for promotion.

[0124] Step 322: Determine the surrounding spatial information of each item's location, and based on the surrounding spatial information of each item's location, determine the perturbability score corresponding to each item.

[0125] For example, the surrounding spatial information includes the space occupancy within a preset range around the item, and the perturbability score is used to characterize whether the item is easily touched and pushed by the pick-and-place device. The perturbability score of the item can be determined based on the surrounding spatial information.

[0126] In some examples, a preset area (such as a circular or rectangular region) can be defined on a horizontal plane with the object's center of mass as the center. The proportion of space occupied by other objects within this preset area, excluding the current object, can be calculated, and the object's perturbability score can be determined based on this proportion. For example, the object's perturbability score is inversely proportional to the proportion of space occupied; the lower the proportion, the more open the space around the object, making it easier for the retrieval and placement device to approach and apply force, resulting in a higher perturbability score.

[0127] In other examples, a minimum distance between the item and the target container wall can be determined, and the item's perturbability score can be determined based on this minimum distance. For example, the item's perturbability score is proportional to this minimum distance; the larger the minimum distance, the more open the space around the item, making it easier for the pick-and-place device to approach the item and apply a push, resulting in a higher perturbability score.

[0128] In other examples, the perturbability score of an item can be determined based on a weighted sum of the aforementioned space occupancy ratio and minimum distance. For instance, the lower the space occupancy ratio of an item, the greater the minimum distance between the item and the target container wall, and the more open the item is in an open area, the higher its perturbability score.

[0129] In some examples, the perturbability score can be set between 0 and 1, where 0 represents completely unperturbable and 1 represents highly perturbable. A higher perturbability score indicates that the item is more easily perturbed and therefore more likely to be selected as a candidate for push.

[0130] Step 323: Items whose instability score exceeds the first preset threshold or whose perturbability score exceeds the second preset threshold are identified as candidate items, and the basic score of each candidate item is determined based on the instability score and the perturbability score.

[0131] For example, instability score or perturbability score can be used as a first screening criterion to filter candidate items from multiple items. Items with instability scores exceeding a first preset threshold can be identified as candidate items, or items with perturbability scores exceeding a second preset threshold can be identified as candidate items. That is, as long as an item meets either the instability or perturbability criteria, the item can be included in the candidate set, which may include candidate items filtered by the first screening criterion.

[0132] For example, the first preset threshold and the second preset threshold can be fixed values ​​preset according to requirements, or they can be adaptively adjusted according to the distribution of items in the target container. This application embodiment does not limit this.

[0133] For example, after determining the candidate set, a baseline score for each candidate item can be determined based on its instability and perturbability scores. This baseline score can be used as a quantitative assessment to evaluate whether a candidate item is worth promoting. A higher baseline score indicates a greater expected benefit for the item as a potential candidate.

[0134] For example, after determining the instability score and the perturbability score, the base score for each item can be determined by a weighted sum of the instability score and the perturbability score.

[0135] In some examples, the base score P1 = w1 × instability score + w2 × perturbability score. Here, w1 and w2 are preset weighting coefficients, and w1 + w2 = 1. For example, w1 can be equal to w2, or w1 can be less than w2, or w1 can be greater than w2.

[0136] It should be noted that the weighting coefficients w1 and w2 can be adjusted according to the application scenario. For example, in scenarios where the scene state needs to be changed quickly, the weight of the instability score can be increased (i.e., prioritizing pushing items that are easy to tip over); in scenarios where precise pushing is required while ensuring safety, the weight of the perturbability score can be increased (i.e., prioritizing pushing items that are easy to approach).

[0137] In other examples, the base score may also incorporate the instability score and perturbability in other ways, such as taking the maximum value or product of the instability score and the perturbability score. This application does not limit this approach.

[0138] Step 324: Based on the basic scores of each candidate item, determine the candidate items to be promoted.

[0139] In some examples, after determining the baseline score of each candidate item, candidate items can be selected for promotion based on their baseline scores. For instance, candidate items with baseline scores exceeding a preset baseline score threshold can be identified as candidate items for promotion.

[0140] In other examples, to further enhance the expected benefits of the push operation, candidate items can be further filtered to identify those that meet a second filtering criterion. This second filtering criterion relates to occlusion and centering conditions.

[0141] Figure 5 This is a schematic diagram illustrating yet another method for retrieving an item, as provided in an embodiment of this application. Figure 5 As shown, step 324 above includes steps 3241 to 3244 as shown below.

[0142] Step 3241: Determine the occlusion score of each candidate item based on the degree of occlusion of other items by each candidate item.

[0143] For example, occlusion level can be used to characterize how much a candidate item occludes other items. A higher occlusion level indicates that the item occludes more other items, or covers more other items. Pushing an item with a high occlusion level can expose more occluded items, thus creating more opportunities for the target object to be taken.

[0144] In one example, the vertical projection coverage of the point cloud of each candidate item can be determined, and the proportion of other item point clouds covered within that coverage area can be determined. This coverage proportion represents the percentage of other items whose total surface area is covered by the candidate item. This coverage proportion is proportional to the occlusion score; a higher coverage proportion indicates that the candidate item occludes more other items, resulting in a higher occlusion score and a greater likelihood that the candidate item will be selected as a potential push target.

[0145] Step 3242: Determine the centering score for each candidate item based on the distance between each candidate item and the center of the target container.

[0146] For example, the centering score can be used to characterize the positional advantage of a candidate item within the target container. Items located near the center of the target container typically have more surrounding space, giving the handling device more room to maneuver and making them less likely to be pushed against the container walls and jammed. Therefore, items closer to the center of the container have a higher pushing priority.

[0147] In some examples, the centering score is determined based on the center distance between each candidate item and the center of the target container. The centering score can be inversely proportional to the center distance between the candidate item and the center of the target container; the closer the candidate item is to the center, the higher the centering score; the farther the candidate item is from the center (i.e., closer to the edge or corner), the lower the centering score.

[0148] In some examples, the center position of the target container and the centroid position of the candidate item can be obtained, and the horizontal distance between the center position of the target container and the centroid position of the candidate item can be determined as the center distance.

[0149] For example, the centering score can be set between 0 and 1, with a value of 1 when the item is in the center of the target container and a value of 0 when it is in the farthest corner of the target container. The higher the centering score, the easier the item is to push, and the greater the likelihood that the candidate item will be selected as a potential push target.

[0150] Step 3243: Determine the comprehensive score of each candidate item based on its base score, occlusion score, and centering score.

[0151] In some examples, the overall score can be a comprehensive quantitative assessment of the overall benefit of pushing a candidate item. The higher the overall score, the greater the expected overall benefit of pushing the item. The overall score takes into account the item's own pushability (i.e., the base score), the exposure effect on other items after pushing (i.e., the occlusion score), and the safety of the operational space (i.e., the centering score).

[0152] For example, the overall score of a candidate item can be determined based on a weighted sum of the base score, occlusion score, and centering score.

[0153] In some examples, the overall score P2 = w3 × base score + w4 × occlusion score + w5 × centering score, where w3, w4 and w5 are preset weighting coefficients, and w3 + w4 + w5 = 1.

[0154] It should be noted that the weighting coefficients w3, w4, and w5 can be preset or adjusted according to the needs of the application scenario. For example, in scenarios where items are heavily stacked and have complex occlusion relationships in the target container, the weight of the occlusion score can be increased to prioritize key occluders that can expose other items. In scenarios where items are relatively sparse in the target container, but some items are in an unstable state, the weight of the base score can be increased to prioritize unstable items that are easily tipped over. In scenarios where the target container is small and the operation space is limited, the weight of the centering score can be increased to prioritize items with more operation space.

[0155] For example, the superimposed score corresponding to the candidate item can be determined based on the weighted sum of the occlusion score and the centering score, and the comprehensive score of the candidate item can be determined based on the weighted sum of the base score and the superimposed score.

[0156] In some examples, the overlay score P3 = w6 × occlusion score + w7 × centering score, where w6 + w7 = 1. The overall score P2 = w8 × base score P1 + w9 × overlay score P3, where w8 + w9 = 1.

[0157] Step 3244: Identify at least one candidate item whose comprehensive score meets the preset promotion conditions as a candidate promotion target.

[0158] In some examples, meeting the preset push conditions may include meeting a preset comprehensive score threshold, that is, all candidate items whose comprehensive score exceeds the preset comprehensive score threshold can be identified as candidate push objects.

[0159] In other examples, if the top N items (e.g., N is an integer) meet the criteria of having the highest overall score, then the candidate items can be sorted from highest to lowest overall score, and the top N items with the highest overall scores can be selected as the candidates to be promoted.

[0160] In some embodiments, if the overall score of each candidate item does not meet the preset promotion conditions, the candidate item with the highest overall score is determined as the candidate to be promoted.

[0161] In some scenarios, the pushing value of all items in the target container is low. For example, when all items are in a stable, flat state and there is no obvious occlusion relationship between them, the instability score, perturbability score, and occlusion score of each item may be very low, resulting in no item having a comprehensive score that meets the preset pushing conditions. In this case, a fallback mechanism can be set up to identify the candidate item with the highest comprehensive score as the candidate pushing object, thereby ensuring that at least one candidate pushing object can be identified in any scenario, thus guaranteeing the robustness of the method.

[0162] The item-picking method provided in this application can comprehensively evaluate the pushing value of each item based on its size, pose, surrounding space, occlusion level, and location information, and determine at least one candidate pushing object. Furthermore, this application also establishes a fallback mechanism to ensure that at least one candidate pushing object can be determined in any scenario, avoiding situations where the pushing operation cannot be performed due to the score not meeting the preset pushing conditions, thus improving the system's robustness and scenario adaptability.

[0163] The process of determining the candidate promotion strategy corresponding to each candidate promotion target in step 330 above will be explained in detail below.

[0164] Figure 6 This is a schematic diagram illustrating another method for retrieving an item provided in an embodiment of this application. For example... Figure 6 As shown, step 330 above includes steps 331 to 334 as shown below.

[0165] Step 331: Based on the size of the space around each candidate pushing object and the crowding of surrounding items, determine at least one first pushing direction for each candidate pushing object, and the direction score corresponding to each first pushing direction.

[0166] In some examples, after identifying the candidate object to be pushed in step 320, step 331 is used to determine the optimal direction to push the candidate object. The choice of pushing direction can affect the safety and effectiveness of the pushing operation. For example, pushing into an open area can avoid collisions, while pushing into a crowded area may cause the candidate object to collide with other objects or get stuck.

[0167] For example, the size of the space around each candidate pushing object and the crowding of surrounding objects can be determined based on scene information. By evaluating the size of the space and the crowding in each direction, the optimal pushing direction, i.e., the first pushing direction, can be selected.

[0168] In some examples, the direction can be divided into multiple sectors within the horizontal plane where the candidate push object is located, with the center direction of each sector serving as a candidate push direction. For each sector, the crowding level of neighboring objects within that sector is calculated. For example, starting from the centroid of the candidate push object, a fan-shaped or rectangular search area is constructed by extending a preset length along the center direction of the sector. The number of point clouds of other objects or the proportion of their occupied space within this search area is then calculated. A higher proportion indicates a higher level of crowding in that direction, resulting in a lower direction score for that push direction.

[0169] In some examples, the gap size of the candidate push object in each direction can also be determined. The gap size includes the distance between the candidate push object and the nearest object in each direction. The smaller this distance, the smaller the gap size in that direction, and the lower the direction score for that push direction.

[0170] For example, the candidate pushing directions for each sector can be sorted based on the congestion level of each sector and the gap size of the candidate pushing objects in each direction. The lower the congestion level and the larger the gap, the higher the ranking and the higher the direction score, indicating that the direction is more suitable for pushing.

[0171] In some examples, one or more directions can be determined as the first driving direction based on the sorting results described above. For example, the direction ranked first in the sort can be selected as the first driving direction, or the directions ranked several positions above the first driving direction can be selected as the first driving direction.

[0172] It should be noted that, in addition to determining the first pushing direction based on the size of the surrounding space and the crowding of surrounding objects, additional candidate pushing directions can be inserted for special cases. For example, when the occlusion degree of the candidate pushing object exceeds a preset occlusion threshold, a direction away from the center of the occluding object can be inserted as a candidate pushing direction. When the shape of the candidate pushing object meets a preset elongated condition, a direction perpendicular to the long axis of the candidate pushing object can be inserted as a candidate pushing direction.

[0173] The item retrieval method provided in this application can evaluate the advantages and disadvantages of each direction based on the space size and crowding level in each direction, and insert optimizations for special cases, thereby determining the first pushing direction corresponding to each candidate pushing object. The first pushing direction can ensure that the subsequent pushing operation is both safe and effective.

[0174] Step 332: Based on the minimum distance between the surface of each candidate push object and the surface of the target container, determine the position region of each candidate push object in the target container. Determine the first contact point based on the position region of each candidate push object in the target container.

[0175] In some examples, after determining the candidate push object in step 320, step 332 is used to determine the optimal position to push the candidate push object. The location of the contact point (i.e., the push position) can affect the effectiveness of the push operation. For example, different contact points result in different motion patterns (such as translation, tilting, rotation, etc.) of the candidate push object. By selecting an appropriate contact point for the candidate push object, different push strategies can be adopted for different scenarios, improving the effectiveness of the push operation.

[0176] For example, the minimum distance between the surface of each candidate push object and the surface of the target container can be determined based on scene information.

[0177] In some examples, the minimum distance can be determined by calculating the distance between the point cloud of the candidate push object and the point cloud of the inner wall of the target container. For example, the minimum distance between the point clouds can be determined as the minimum distance between the candidate push object and the target container.

[0178] In some examples, the location region of a candidate push object within the target container can be determined based on the minimum distance. When the minimum distance is less than a preset edge threshold, the candidate push object can be determined to be located in the edge region (i.e., close to the container wall); when the minimum distance is greater than or equal to the preset edge threshold, the candidate push object can be determined to be located in the middle region.

[0179] For example, when the candidate object to be pushed is located in the edge region of the target container, the first contact point is determined to be located in the top region of the candidate object to be pushed. The top region is the area on or near the upper surface of the candidate object to be pushed. In this case, the contact point corresponds to a top-sliding strategy, that is, pressure can be applied from the top to cause the candidate object to slide away from the edge region along the container wall under pressure, avoiding the item being pushed against the wall and getting stuck due to lateral pushing.

[0180] For example, when the candidate push object is located in the middle region of the target container, the first contact point is determined to be located in the side region of the candidate push object. The side region is the side of the candidate push object and is located in the lower-middle region of the candidate push object. In this case, the contact point corresponds to a side-push overturning strategy, that is, a horizontal thrust can be applied from the lower-middle part of the side to form an effective overturning moment, causing the candidate push object to rotate around its bottom edge and overturn, thereby significantly changing its posture.

[0181] In some examples, after determining that the first contact point is located in the side region, its specific position on the side can be further determined. For example, the horizontal coordinate of the first contact point can be set near the geometric center of the side of the candidate pushing object to ensure uniform force application. Alternatively, the horizontal coordinate of the contact point can be set on the side of the candidate pushing object closer to the first pushing direction.

[0182] It should be noted that after determining the first pushing direction through step 331, the first contact point corresponding to the first pushing direction can be determined. For example, the first pushing direction may correspond to at least one first contact point, but this embodiment of the application does not limit this.

[0183] The item retrieval method provided in this application can adaptively determine a suitable first contact point based on the position area (edge ​​or middle) of the candidate push object in the target container, thereby adopting different pushing strategies for different spatial constraints and improving the effectiveness and safety of the pushing operation.

[0184] Step 333: Based on the height of the first contact point of each candidate pushing object, the vertical height of each candidate pushing object, and its horizontal dimension, determine the moving distance of each candidate pushing object. Based on the moving distance of each candidate pushing object, determine the first pushing distance.

[0185] In some examples, after determining the first pushing direction in step 321 and the first contact point in step 322, step 333 is used to determine the optimal distance to push the candidate object along the first pushing direction from the first contact point. The pushing distance can affect the effectiveness and safety of the pushing operation. A pushing distance that is too short may prevent the candidate object from undergoing the expected pose change, resulting in an ineffective push; a pushing distance that is too long may lead to a collision or the object falling. Therefore, it is necessary to determine a first pushing distance that is both effective and safe for completing the pushing operation.

[0186] It should be noted that after determining the first contact point in step 321, the first pushing distance corresponding to each first contact point can be determined. The first pushing distances corresponding to different first contact points may be different or the same.

[0187] For example, the height h of the first contact point of each candidate push object, the vertical height H and the horizontal dimension W of each candidate push object can be determined based on the geometric characteristics of each candidate push object.

[0188] In some examples, the height h of the first contact point can be the vertical distance of the first contact point relative to the bottom of the candidate push object. The vertical height H can be the total height of the candidate push object in the vertical direction, which can be determined by the distribution range of the point cloud of the candidate push object in the vertical direction. The horizontal dimension W can refer to the maximum dimension of the candidate push object in the horizontal direction.

[0189] In some embodiments, the travel distance satisfies the following formula (1).

[0190] D1 = h × (W / H) × α; Formula (1); In formula (1), D1 is the moving distance of the candidate pushing object, h is the height of the first contact point of the candidate pushing object, W is the horizontal dimension of the candidate pushing object, H is the vertical height of the candidate pushing object, and α is the preset pushing ratio factor.

[0191] In some examples, the higher the height h of the first contact point, the greater the overturning moment generated when pushing the candidate object, and the greater the horizontal displacement required to achieve complete overturning. Therefore, the movement distance D1 is proportional to the height h of the first contact point.

[0192] In some examples, W / H is the aspect ratio of the object (i.e., the ratio of its horizontal dimension to its vertical height), and the tangent of the tipping-over critical angle is tan(θ) = W / H. For example, for a tall, slender, upright object, its tipping-over critical angle is small, requiring only a small horizontal displacement to tip it over; for a flat, lying object, its tipping-over critical angle is large, requiring a larger horizontal displacement to change its posture. Therefore, the movement distance D1 is proportional to W / H, meaning the smaller W / H is, the smaller the movement distance.

[0193] In some examples, α is a preset push ratio factor. α can be set according to requirements and can be greater than 1 or less than 1. This application embodiment does not limit this. α can be used as a safety factor to ensure that the candidate push object can reliably overturn, avoiding deviations between theoretical and actual values ​​caused by factors such as friction and surface unevenness.

[0194] For example, after determining the moving distance, a first pushing distance can be determined based on the relationship between the moving distance and a preset minimum pushing distance and a preset maximum permitted pushing distance. Specifically, if the moving distance is less than the preset minimum pushing distance, the preset minimum pushing distance is determined as the first pushing distance; if the moving distance is greater than the preset maximum permitted pushing distance, the maximum permitted pushing distance is determined as the first pushing distance; and if the moving distance is greater than the preset minimum pushing distance but less than the preset maximum permitted pushing distance, the moving distance is determined as the first pushing distance.

[0195] In some embodiments, the first pushing distance satisfies the following formula (2).

[0196] D2 = clamp(D1, Dmin, Dmax); Formula (2); In formula (2), D2 is the first pushing distance of the candidate pushing object, Dmin is the preset minimum pushing distance, and Dmax is the maximum permitted distance.

[0197] In some examples, the preset minimum pushing distance Dmin can be a preset constant to prevent the calculated movement distance from being too small, thus rendering the pushing action invalid. When the movement distance is less than the preset minimum pushing distance Dmin, even if the pushing operation is performed, the candidate object may not actually move effectively due to factors such as inertia and friction, resulting in an invalid pushing operation. Therefore, in this case, the first pushing distance D2 can be determined as the preset minimum pushing distance Dmin.

[0198] In some examples, the maximum permissible distance Dmax is used to indicate the maximum distance an item can move within the target container, and can be dynamically determined based on the physical space constraints within the target container. When the moving distance exceeds the maximum permissible distance Dmax, it indicates that the moving distance exceeds the maximum safe distance allowed by the physical space, and performing a pushing operation according to the moving distance would pose a safety hazard. Therefore, in this case, the first pushing distance D2 can be determined as the maximum permissible distance Dmax.

[0199] In some examples, when Dmin ≤ D1 ≤ Dmax, it indicates that the moving distance satisfies both the lower limit requirement of effectiveness and the upper limit requirement of safety. Therefore, this moving distance D1 can be determined as the first pushing distance D2.

[0200] In some embodiments, a first gap is determined; a second gap is determined; and the smaller of the first gap and the second gap is determined as the maximum permissible distance.

[0201] For example, the first gap is a spatial gap along the first pushing direction, defined by the size of the end effector of the pick-and-place device as the width of the corridor. The first gap can be the minimum spatial passage required for the end effector of the pick-and-place device to move along the pushing direction, or the maximum distance that the end effector can safely pass through during its movement along the pushing direction, to ensure that the end effector does not collide with surrounding objects.

[0202] For example, the second gap is a spatial gap along the first pushing direction and in the direction perpendicular to the first pushing direction, with the size of the candidate pushing object as the width of the corridor. The second gap can be the minimum spatial passage required for the candidate pushing object to be pushed, or the maximum distance that the candidate pushing object can safely pass through during its movement along the pushing direction, to ensure that the candidate pushing object will not collide with surrounding objects during its movement.

[0203] In some examples, the smaller of the first and second gaps can be determined as the maximum permissible distance Dmax. By selecting the smaller of the first and second gaps, it can be ensured that the pick-and-place device will not be damaged due to interference between the end effector and surrounding objects during the pushing process, nor will the pushing failure or damage to the object be caused by collision between the candidate object and surrounding objects.

[0204] In some examples, if either the first gap or the second gap is less than a preset minimum gap threshold, the first pushing direction can be determined to be infeasible and discarded.

[0205] It should be noted that the first pushing distance determined in step 333 can correspond to the first pushing direction determined in step 331. That is, the first pushing distances corresponding to different first pushing directions can be the same or different, and a corresponding first pushing distance can be determined for each first pushing direction.

[0206] Step 334: Based on the candidate pushing object, the first pushing direction corresponding to the candidate pushing object, the first contact point corresponding to the first pushing direction, and the first pushing distance corresponding to the first contact point, determine the pushing parameters corresponding to the candidate pushing object. Based on the comprehensive score corresponding to the candidate pushing object and the direction score of the first pushing direction corresponding to the candidate pushing object, determine the first score in the candidate pushing strategy. Based on the pushing parameters and the pushing object, determine the candidate pushing strategy.

[0207] In some examples, steps S331 to S333 above can determine a complete candidate pushing strategy for each candidate pushing object. This candidate pushing strategy can include the following pushing parameters: candidate pushing object, first pushing direction, first contact point, and first pushing distance. Specifically, step S331 determines the direction of pushing, step S332 determines the position to push, and step S333 determines the distance to push.

[0208] For example, after determining the driving parameters corresponding to the candidate driving strategy, the first score of the candidate driving strategy can be further determined based on the comprehensive score corresponding to the candidate driving object and the directional score of the driving direction corresponding to the candidate object.

[0209] In some examples, the first score can be determined based on a weighted sum of the overall score and the directional score.

[0210] In some examples, the first score = w10 × overall score + w11 × directional score. Here, w10 and w11 are preset weighting coefficients, and w10 + w11 = 1. It should be noted that the weighting coefficients w10 and w11 can be adjusted according to the needs of the application scenario. For example, in scenarios with severe item stacking and complex occlusion relationships, the weight of the overall score can be increased to prioritize items with higher overall returns; in scenarios with limited operational space and high collision risk, the weight of the directional score can be increased to prioritize safer directions.

[0211] In other examples, the first score can incorporate more scoring factors. For instance, factors such as whether the location of the first contact point is conducive to force application, whether the first pushing distance matches the size of the candidate pushing object, and whether the selection of the end effector matches the candidate pushing object can be included in the calculation of the first score. This application does not specifically limit the factors influencing the first score.

[0212] For example, the first score indicates the probability that the target object can be successfully picked up by the pick-and-place device after it performs a push on the candidate push object based on the push parameters. The higher the probability of successful picking up, the higher the execution priority of the candidate push strategy. In other words, the first score is a prediction of the probability that the target object can be successfully picked up after executing the candidate push strategy.

[0213] The following section provides a detailed explanation of the implementation process of Scheme 2 (i.e., the deterministic method based on geometric and physical rules).

[0214] In some examples, Scenario 2 can achieve the retrieval of target objects in the target container using a pre-trained network model (such as a pre-defined evaluation model).

[0215] Figure 7 This is a schematic diagram of a preset evaluation model provided in an embodiment of this application. Figure 7 As shown, the preset evaluation model 700 may include a take-up evaluation module 710 and a push evaluation module 720.

[0216] In some examples, the picking evaluation module 710 can be a trained neural network model whose input can be point cloud data (such as referred to as first point cloud data) and candidate picking poses, and whose output can be a feasibility score of the candidate picking poses. The feasibility score indicates the probability of successful picking of the target object by the picking device using the candidate picking poses. For example, a higher feasibility score indicates a higher success rate in picking up the target object.

[0217] In some examples, the push evaluation module 720 can be a trained neural network model that takes point cloud data (such as second point cloud data) as input and a predicted push action as output, and the effectiveness score of the predicted push action as output. The effectiveness score indicates the probability that the target object will become reachable after the pick-and-place device performs the predicted push action. For example, a higher effectiveness score indicates a higher probability that the target object will become reachable.

[0218] It should be noted that, as Figure 7 As shown, retrieving the output of evaluation module 710 will affect the specific execution of evaluation module 720. The following section will discuss this further. Figure 8 The specific execution process of the preset evaluation model is explained.

[0219] Figure 8 This is a schematic diagram illustrating another method for retrieving an item provided in an embodiment of this application. For example... Figure 8 As shown, the method includes steps 810 to 870 as shown below.

[0220] Step 810: During the process of the robot's pick-up and place-down device taking the target object from the target container, obtain the scene information corresponding to the target container.

[0221] It should be noted that step 810 is similar to the implementation process of step 310 in the above embodiment, and will not be repeated here to avoid repetition.

[0222] Step 820: In response to a failure event of the pick-up and place device to pick up the target object, determine at least one estimated pick-up pose corresponding to the target object based on the scene information.

[0223] For example, the candidate pick-up pose can be the position and orientation information of the end effector of the pick-up and place-up device when performing a pick-up operation on the target object in three-dimensional space. The position information can be the coordinates (x, y, z) of the end effector in three-dimensional space, such as the spatial location of the pick-up point. The orientation information can be the orientation angle of the end effector.

[0224] In some examples, candidate pick-up poses can be generated based on scene information. For instance, the surface point cloud of the target object can be uniformly sampled, and the sampled points can be used as the positions of candidate pick-up points. For each candidate pick-up point position, multiple candidate pick-up directions are generated. The positions of the candidate pick-up points and each direction vector are combined to form multiple candidate pick-up poses.

[0225] It should be noted that the number of candidate pick-up poses can be configured according to actual needs, and this application embodiment does not limit this. For example, the number of candidate pick-up poses can cover as many different positions and pick-up directions on the surface of the target object as possible.

[0226] Step 830: Input at least one estimated pick-up pose and the first point cloud data corresponding to the target container into the pick-up evaluation module in the preset evaluation model to obtain the feasibility score of each estimated pick-up pose.

[0227] In some examples, the first point cloud data may include a local point cloud extracted from the scene point cloud of the target container, centered on the pick-up point of the candidate pick-up pose. For example, the local point cloud may contain geometric information near the candidate pick-up point, including surface curvature, normal vector direction, edge features, etc., which is the basis for determining the feasibility of pick-up.

[0228] For example, candidate pick-up poses are used together with the first point cloud data (i.e., local point cloud) as input to the pick-up evaluation module. The candidate pick-up poses determine the direction from which the end effector approaches the target object, and different candidate pick-up poses correspond to different local geometric feature distributions.

[0229] In some examples, the pick-up evaluation module can output a feasibility score for each estimated pick-up pose. The feasibility score indicates the probability that the pick-up device can successfully pick up the target object from the target container based on the estimated pick-up pose. For example, the feasibility score can be a continuous value between 0 and 1 output by the pick-up evaluation module; a higher feasibility score indicates a higher success rate for that candidate pick-up pose.

[0230] Step 840: If, in at least one of the estimated pick-up poses, the feasibility score corresponding to the target pick-up pose is greater than or equal to a preset feasibility threshold, the pick-up and place-up device is controlled to pick up the target object through the target pick-up pose.

[0231] For example, the feasibility scores of all candidate pick-up poses are iterated to determine whether there are any candidate pick-up poses whose feasibility scores exceed a preset feasibility threshold. The preset feasibility threshold can be a pre-set value. For instance, if the preset feasibility threshold is set to 0.7, and the success rate of the current candidate pick-up pose exceeds 70%, it indicates that the current candidate pick-up position is reliable enough and can be directly executed.

[0232] In some examples, if there are multiple candidate pick-up poses with feasibility scores equal to or equal to a preset feasibility threshold, the candidate pick-up pose with the highest feasibility score can be selected as the target pick-up pose. If there is only one candidate pick-up pose with a feasibility score equal to or equal to the preset feasibility threshold, then that candidate pick-up pose can be determined as the target pick-up pose.

[0233] In some examples, after determining the target picking pose, corresponding control commands can be generated based on the target picking pose to control the picking and placing device to move to the position and posture of the target picking pose, and to perform a picking operation on the target object to remove the target object from the target container.

[0234] Step 850: If the feasibility score of at least one estimated grasping pose is less than the preset feasibility threshold, determine at least one estimated pushing action corresponding to the target object based on the scene information.

[0235] For example, if the feasibility score of each estimated picking pose output by the picking evaluation module is less than the preset feasibility threshold, it indicates that the picking device cannot successfully pick up the target object in the current scenario of the target container. Therefore, a pushing operation needs to be performed first.

[0236] For example, at least one estimated pushing action corresponding to the target object can be determined based on scene information. The estimated pushing action is used to indicate a complete pushing operation plan, and the estimated pushing action includes at least: an estimated pushing object corresponding to the target object, and a second pushing direction, a second contact point, and a second pushing distance corresponding to the estimated pushing object.

[0237] In some examples, the process of determining the estimated pushing object can be similar to the process of determining candidate pushing objects in steps 321 to 324 of the above embodiments. To avoid repetition, it will not be described again here. Alternatively, multiple items that are close to the target object can be selected from the target container as estimated pushing objects.

[0238] In some examples, the determination of the second pushing direction, the second contact point, and the second pushing distance corresponding to each preset pushing object can be similar to the process of determining the first pushing direction, the first contact point, and the first pushing distance in steps 331, 332, and 333 of the above embodiments. To avoid repetition, it will not be described again here.

[0239] In other examples, the second pushing direction, the second contact point, and the second pushing distance can also be determined by randomly sampling each estimated pushing object. For example, multiple directions can be randomly sampled within the horizontal plane where the candidate pushing object is located as the second pushing direction. Multiple points can be uniformly sampled on the surface of the candidate pushing object as candidate contact points. Multiple preset candidate distances can be used as the second pushing distance.

[0240] For example, each estimated pushing object is combined with its corresponding second pushing direction, second contact point and second pushing distance to form at least one estimated pushing action corresponding to each preset pushing object.

[0241] Step 860: Input the second point cloud data corresponding to each predicted driving action and the target container into the driving evaluation module in the preset evaluation model to obtain the effectiveness score of each predicted driving action.

[0242] In some examples, the second point cloud data may include point clouds of each estimated pushing object within the target container and point clouds surrounding each estimated pushing object. The second point cloud data and at least one estimated pushing action generated in step 850 are input into the pushing evaluation module.

[0243] For example, the push evaluation module can output an effectiveness score corresponding to each preset push action. The effectiveness score indicates the probability that the target object becomes graspable after the pick-and-place device performs the estimated push action.

[0244] For example, the effectiveness score can be a continuous value between 0 and 1 from the output of the evaluation module. The higher the effectiveness score, the greater the likelihood that the feasibility score of the target object will be significantly improved after the predicted action is taken, that is, the greater the likelihood that the target object will become available.

[0245] In some examples, the effectiveness score in step 820 is different from the feasibility score in step 830. The feasibility score is mainly used to assess the success probability of directly taking the target object in the current state, while the effectiveness score is mainly used to assess the degree of improvement in the success probability when taking the target object after performing the pushing action.

[0246] Step 870: Based on the effectiveness score of each estimated pushing action, determine the target pushing action among at least one estimated pushing action, and control the pick-and-place device to execute the target pushing action.

[0247] For example, the effectiveness scores of all estimated pushing actions are iterated, and the estimated pushing action with the highest effectiveness score is selected as the target pushing action. After determining the target pushing action, a corresponding control command is generated based on the pushing parameters of the target pushing action (estimated pushing object, second pushing direction, second contact point, and second pushing distance) to control the pick-and-place device to execute the target pushing action to push the estimated object.

[0248] In some embodiments, after step 870 above, the method may further include: obtaining updated scenario information corresponding to the target container; and, based on the updated scenario information, re-outputting the feasibility score of the target object through the evaluation module until the feasibility score of the target object exceeds a preset feasibility threshold.

[0249] For example, if the distribution of items within the target container changes after the push action is performed, it is necessary to reassess whether the push action successfully improved the accessibility of the target object. If the feasibility score of the target object exceeds a preset feasibility threshold after the push, the push operation is considered successful, and the object can be retrieved.

[0250] In some examples, after a target-driven action, the feasibility score of the target object can be reassessed using the evaluation module. If the feasibility score of the target object still does not exceed the preset feasibility threshold, it indicates that the execution of the target-driven action has not been effective in improving the target object, and new driving actions need to be executed.

[0251] It should be noted that the process of re-outputting the feasibility score of the target object by retrieving the evaluation module based on the updated scenario information is similar to the implementation of steps 820 to 830 in the above embodiments. To avoid repetition, it will not be described again here. That is to say, steps 820 to 870 can be repeated until the feasibility score corresponding to the target object exceeds the preset feasibility threshold.

[0252] For example, if the number of iterations exceeds the preset maximum number of iterations and the feasibility score corresponding to the target object still does not exceed the preset feasibility threshold, the iterative calculation can be stopped and the output device can be instructed to output an abnormal prompt message.

[0253] In some examples, if the feasibility score of the target object exceeds the preset feasibility threshold after the target-driven action is performed, it indicates that the execution of the target-driven action has a significant improvement effect on the target object, and the target object can be obtained based on the above step 840.

[0254] The item retrieval method provided in this application embodiment achieves item recognition, pushing, and retrieval through a pre-trained preset evaluation model. The retrieval evaluation module in the preset evaluation model scores the feasibility of candidate retrieval poses. When a highly feasible retrieval pose exists, the retrieval device is controlled to directly execute the retrieval action. When no highly feasible retrieval pose exists, the effectiveness of the predicted pushing action is evaluated by the pushing evaluation module in the preset evaluation model. The most effective pushing action is then selected for execution, and the scene state is continuously improved through a closed-loop iterative mechanism, thereby improving the accuracy and efficiency of the robot in retrieving items.

[0255] The following is a detailed explanation of the implementation process of Scheme 3 (a combination of Scheme 1 and Scheme 2).

[0256] In some examples, the combination of Scheme 1 and Scheme 2 includes, but is not limited to, the first combination method, the second combination method, and the third combination method. The specific implementation methods of these three combination methods are explained below with reference to the accompanying drawings.

[0257] Figure 9 This is a schematic diagram illustrating another method for retrieving an item provided in an embodiment of this application. The following is in conjunction with... Figure 9 The first composite method will be explained. For example... Figure 9 As shown, the method includes steps 910 to 950 as shown below.

[0258] Step 910: During the process of the robot's pick-up and place device taking the target object from the target container, obtain the scene information corresponding to the target container.

[0259] Step 920: In response to a failure event of the pick-up and drop-off device to pick up the target object, a candidate push object corresponding to the target object is determined among multiple items in the target container.

[0260] Step 930: Based on the scene information and the geometric features of each candidate push object, determine the candidate push strategy corresponding to each candidate push object.

[0261] It should be noted that steps 910 to 930 are similar to steps 310 to 330 in the above embodiments, and will not be described again here to avoid repetition. After determining the candidate driving strategy through step 930, the target driving strategy can be selected from the candidate driving strategies based on the candidate driving strategy and the preset evaluation model. The candidate driving strategy includes driving parameters and a first score.

[0262] Step 940: Input the third point cloud data corresponding to each candidate driving strategy and the target container into the driving evaluation module in the preset evaluation model to obtain the second score corresponding to each candidate driving strategy; determine the target score of each candidate driving strategy based on the first score and the second score; determine the target driving strategy among the candidate driving strategies based on the target score of each candidate driving strategy.

[0263] In some examples, step 930 can determine the candidate pushing strategy corresponding to each candidate pushing object. Each candidate pushing strategy can include complete pushing parameters (such as candidate pushing object, first pushing direction, first contact point, first pushing distance) and a first score. Each candidate pushing strategy can be used as input to the pushing evaluation module in the preset evaluation model.

[0264] In other examples, after determining the candidate driving strategies corresponding to each candidate driving object through step 930, a feasibility check can be performed on each candidate driving strategy, and all candidate driving strategies that pass the feasibility check can be used as inputs to the driving evaluation module in the preset evaluation model.

[0265] For example, candidate pushing strategies and third point cloud data (similar to the second point cloud data mentioned above) corresponding to the target container can be input into the pushing evaluation module, so that the pushing evaluation module outputs a second score corresponding to each candidate pushing strategy. The second score indicates the probability that the target object becomes available for handling after the pick-and-place device executes a candidate pushing strategy.

[0266] It should be noted that the process of determining the second score corresponding to the candidate driving strategy in step 940 is similar to the process of step 860 in the above embodiment. The candidate driving strategy in step 940 is equivalent to the predicted driving action in step 860, and the second score is equivalent to the effectiveness score.

[0267] For example, after obtaining the second score corresponding to each candidate push strategy through the push evaluation module, the target score of each candidate push strategy can be determined based on the second score and the first score of each candidate push strategy.

[0268] In some examples, the target score can be a weighted sum of the first and second scores. For example, target score = w12 × first score + w13 × second score. Here, w12 and w13 are preset weighting coefficients, and w12 + w13 = 1.

[0269] It should be noted that the weight coefficients w12 and w13 can be adjusted according to the actual application scenario, and this application embodiment does not limit this. For example, in scenarios with limited data accumulation and low model confidence, the weight of the first score can be increased, relying more on deterministic rules; in scenarios with abundant data accumulation and high model confidence, the weight of the second score can be increased, relying more on data-driven prediction.

[0270] In other examples, the target score may also adopt other fusion methods, such as taking the product of the first score and the second score, etc., which are not limited in this application embodiment.

[0271] For example, after determining the target score corresponding to each candidate driving strategy, a target driving strategy can be determined from the candidate driving strategies based on the target score of each candidate driving strategy. For instance, the candidate driving strategy with the highest target score can be used as the target driving strategy.

[0272] Step 950: Control the pick-and-place device to execute the target pushing strategy to pick up the target object.

[0273] It should be noted that step 950 is similar to the implementation process of step 350 in the above embodiment, and will not be repeated here to avoid repetition.

[0274] The item retrieval method provided in this application, specifically the first composite approach in Scheme 3, organically combines Scheme 1 (a deterministic method based on geometric physical rules) and Scheme 2 (a modeling method based on point cloud deep learning) to form a composite decision-making architecture of candidate generation + learning verification. The first composite approach first generates candidate driving strategies (such as candidate driving strategies that have undergone feasibility verification) using Scheme 1, ensuring that all candidate driving strategies are physically executable. Then, the driving evaluation module of Scheme 2 performs data-driven effectiveness prediction on the candidate driving strategies, evaluating the actual driving effect of each candidate driving strategy from empirical data. By integrating the two scoring methods, the final decision simultaneously possesses the physical feasibility guarantee of Scheme 1 and the data-driven effectiveness prediction capability of Scheme 2, achieving a balance between safety assurance and effect optimization.

[0275] Figure 10 A schematic diagram illustrating another method for retrieving an item provided in this application embodiment is shown below in conjunction with... Figure 10 The second composite method will be explained. For example... Figure 10 As shown, the method includes steps 1010 to 1080 as shown below.

[0276] Step 1010: During the process of the robot's pick-up and place device taking the target object from the target container, obtain the scene information corresponding to the target container.

[0277] Step 1020: In response to a failure event of the pick-up and place device to pick up the target object, determine at least one estimated pick-up pose corresponding to the target object based on scene information. Step 1030: Input at least one estimated pick-up pose and the first point cloud data corresponding to the target container into the pick-up evaluation module in the preset evaluation model to obtain the feasibility score of each estimated pick-up pose. Step 1040: If, in at least one of the estimated pick-up poses, the feasibility score corresponding to the target pick-up pose is greater than or equal to a preset feasibility threshold, the pick-up and place-up device is controlled to pick up the target object through the target pick-up pose. Step 1050: If the feasibility score of at least one estimated grasping pose is less than a preset feasibility threshold, determine at least one estimated pushing action corresponding to the target object based on scene information. Step 1060: Input the second point cloud data corresponding to each predicted driving action and the target container into the driving evaluation module in the preset evaluation model to obtain the effectiveness score of each predicted driving action. Step 1070: Based on the effectiveness score of each estimated pushing action, determine the target pushing action in at least one estimated pushing action, and control the pick-and-place device to execute the target pushing action.

[0278] It should be noted that steps 1010 to 1070 are similar to steps 810 to 870 in the above embodiments, and will not be described again here to avoid repetition. After determining the target pushing action through step 1070 and controlling the pick-up and place device to execute the target pushing action, the method further includes the following step 1080.

[0279] Step 1080: Determine the result data of the target pushing action performed by the pick-and-place device; based on the result data, adjust the preset adjustable parameters.

[0280] In some examples, after the pick-and-place device is controlled to perform the target pushing action in step 1070, the result data of the pushing operation can be determined. This result data includes, but is not limited to, the following information: whether the pushing operation was successfully executed, whether any abnormalities occurred during the pushing operation, the actual pose change of the candidate object after the pushing, the change in the feasibility score of the target object after the pushing, and whether the target object was ultimately successfully picked up.

[0281] For example, statistical analysis can be performed on the results of multiple pushing operations to extract patterns of pushing effectiveness, thereby adjusting the preset adjustable parameters involved in Scheme 1. These preset adjustable parameters include at least a preset pushing ratio factor used to determine the first pushing distance.

[0282] In some examples, when the preset adjustable parameters include a preset push ratio factor, the push effectiveness can be categorized into push success rates for different values ​​of the push ratio factor α. For instance, the push success rate is 75% when α=1.2, 82% when α=1.4, and 78% when α=1.6. Multiple results show that the optimal α value in this scenario is approximately 1.4. Therefore, the preset push ratio factor can be adjusted to 1.4. This facilitates subsequent calculations using the adjusted preset push ratio factor when implementing Scheme 1.

[0283] In some examples, the preset adjustable parameters may also include the first preset threshold (i.e., the instability score threshold), the second preset threshold (i.e., the perturbation score threshold), various weight coefficients (such as the weight coefficients of the base score, occlusion score, and centering score), and various collision tolerance thresholds, etc., as described in Scheme 1 above. This application does not limit these parameters.

[0284] The item retrieval method provided in this application embodiment, in Scheme 3, the second composite method can achieve adaptive optimization of some adjustable parameters in Scheme 1 through Scheme 2. The geometric-physical rules in Scheme 1 include multiple preset adjustable parameters (such as the push ratio factor α, various scoring thresholds, and various weight coefficients), the values ​​of which directly affect the generation quality of candidate push strategies. In Scheme 1, the adjustment of these parameters can rely on experience and repeated trials, which is inefficient and difficult to guarantee optimality. By feeding back the result data accumulated during the actual execution of Scheme 2 to the parameter adjustment stage of Scheme 1, the adjustable parameters can be continuously optimized with the accumulation of execution data, achieving the effect of data-driven rule evolution. This allows Scheme 1 to maintain determinism and interpretability while possessing the ability for continuous optimization.

[0285] Figure 11 A schematic diagram illustrating another method for retrieving an item provided in this application embodiment is shown below in conjunction with... Figure 11 The third composite method will be explained. For example... Figure 11 As shown, the method includes steps 1110 to 1130 as shown below.

[0286] Step 1110: In response to a failure event of the pick-up and place device to pick up the target object, determine the evaluation parameters corresponding to the current scenario.

[0287] For example, after a failure to retrieve the target object occurs, several evaluation parameters corresponding to the current scenario can be determined first. These evaluation parameters may include at least: scenario complexity, geometric features of the target object, execution success rate of the preset evaluation model, and confidence level of the preset evaluation model. The evaluation parameters are used to determine whether to use Scheme 1 or Scheme 2 to retrieve the target object in the current scenario.

[0288] In some examples, scene complexity can serve as a quantitative assessment of the stacking and arrangement of items within a target container. For instance, scene complexity can be determined based on at least the following factors: the total number of items within the target container, the degree of stacking between items (e.g., the number of overlapping layers in the vertical direction), the degree of occlusion between items (e.g., the proportion of an item's area obscured by other items), and the irregularity of item shapes (e.g., the degree to which item shapes deviate from regular geometry). A weighted fusion of these factors yields a comprehensive score representing the scene complexity.

[0289] In some examples, the geometric features of the target object may include its shape, surface flatness, and size. The shape of the target object can be used to determine whether it is a regular or irregular shape; surface flatness is used to determine whether the target object is easy to contact and apply force to; and size is used to determine whether the target object is within the range of the pick-and-place device.

[0290] For example, the execution success rate of the preset evaluation model is used to indicate the probability that the pick-and-place device can successfully pick up the target object based on the preset evaluation model. For instance, the execution success rate can be calculated by counting the number of successes of the preset evaluation model (including the pick-up evaluation module and the push evaluation module) in the most recent N actual executions. For example, the execution success rate = number of successes / total number of executions.

[0291] For example, the confidence level of the preset evaluation model is used to indicate the probability that the feasibility score and / or validity score output by the preset evaluation model is higher than a preset score threshold. For example, the confidence level of the preset evaluation model can be determined by calculating the uncertainty of the model output score for each output of the preset evaluation model; when the uncertainty is high, it indicates that the preset evaluation model is not confident enough in the current prediction, and the confidence level is low.

[0292] Step 1120: Based on the evaluation parameters corresponding to the current scenario, determine whether the current scenario satisfies the first constraint condition or the second constraint condition.

[0293] In some examples, the first constraint is the condition for executing Option 1, and the second constraint is the condition for executing Option 2. After determining the evaluation parameters corresponding to the current scenario, it is further determined whether the current scenario satisfies the first constraint or the second constraint.

[0294] For example, satisfying the first constraint condition may include at least one of the following: the scene complexity does not exceed a preset complexity threshold, the geometric features of the target object meet the geometric specification conditions, the execution success rate of the preset evaluation model is lower than a preset success rate threshold, and the confidence level of the preset evaluation model is lower than a preset confidence threshold.

[0295] For example, if the current scenario does not satisfy the first constraint, it can be determined that the current scenario satisfies the second constraint.

[0296] In some examples, the scenario complexity can be compared with a preset complexity threshold. If the scenario complexity is less than the preset complexity threshold, it indicates that the current scenario is relatively simple and suitable for executing Option 1; otherwise, it indicates that the current scenario is relatively complex and Option 1 may not be able to effectively cover it, so Option 2 is more suitable.

[0297] In some examples, it can be determined whether the geometric features of the target object meet the geometric specification conditions. When the target object is a regular shape, has a flat surface, and is of moderate size, it indicates that the geometric features of the target object meet the geometric specification conditions. Therefore, the geometric physical rules of Scheme 1 can be effectively applied to the target object and can be executed. Otherwise, it indicates that the geometric shape of the target object is too complex or irregular, and the rule method of Scheme 1 may fail. Therefore, Scheme 2 is more suitable.

[0298] In some examples, the execution success rate of the preset evaluation model can be compared with the preset success rate threshold. If the execution success rate is lower than the preset success rate threshold, it indicates that the preset evaluation model is not performing well in the current or recent scenario, and should be reverted to Option 1 as a fallback; otherwise, Option 2 can be used.

[0299] In some examples, the confidence level of the preset evaluation model can be compared with the preset performance threshold. If the confidence level is lower than the preset confidence threshold, it indicates that the prediction reliability of the preset evaluation model in the current scenario is insufficient, and it should fall back to Scheme 1 as a fallback; otherwise, Scheme 2 can continue to be used.

[0300] In some examples, the judgment logic for determining whether the first constraint or the second preset condition is met based on the evaluation parameters may include: when the scene complexity does not exceed the preset complexity threshold and the geometric features of the target object meet the geometric specification conditions, it is determined that the first constraint condition is met; otherwise, the second constraint condition is met.

[0301] In other examples, the judgment logic for determining whether the first constraint or the second preset condition is met based on the evaluation parameters may also include: when the model execution success rate is lower than the preset success rate threshold, or when the model confidence is lower than the preset confidence threshold, the first constraint is determined to be met; otherwise, the second constraint is met.

[0302] For example, priorities can be set for the aforementioned evaluation parameters. If the model execution success rate and model confidence have higher priority than scene complexity and geometric features, then as long as the model execution success rate is determined to be lower than a preset success rate threshold, or the model confidence is determined to be lower than a preset confidence threshold, regardless of scene complexity and geometric features, it can be determined that the current scene meets the first constraint condition, and Scheme 1 can be executed as a safety guarantee. Conversely, if the model execution success rate is higher than a preset success rate threshold and the model confidence is higher than a preset confidence threshold, regardless of scene complexity and geometric features, it can be determined that the current scene meets the second constraint condition, i.e., Scheme 2 should be executed.

[0303] For example, in addition to the evaluation parameters mentioned above, satisfying the second constraint may also include: if none of the candidate driving strategies determined in Scheme 1 pass the subsequent feasibility verification, it can be determined that the current scenario satisfies the second constraint, and Scheme 2 will re-evaluate it.

[0304] Step 1130: Under the condition that the first constraint condition is met in the current scene, determine the candidate pushing object corresponding to the target object among the multiple items in the target container; determine the candidate pushing strategy corresponding to each candidate pushing object based on the scene information and the geometric features of each candidate pushing object; determine the target pushing strategy among the multiple candidate pushing strategies based on the first score in each candidate pushing strategy; control the pick-up and put-down device to execute the target pushing strategy to pick up the target object.

[0305] In some examples, after determining in step 1120 that the current scenario satisfies the first constraint condition, the steps of Scheme 1 can be executed to retrieve the target object. It should be noted that the specific implementation process of Scheme 1 has been described in detail in steps 310 to 350 of the above embodiments, and will not be repeated here to avoid repetition.

[0306] Step 1140: Under the condition that the current scene satisfies the second constraint, determine at least one estimated pick-up pose corresponding to the target object based on the scene information; input the at least one estimated pick-up pose and the first point cloud data corresponding to the target container into the pick-up evaluation module in the preset evaluation model to obtain the feasibility score of each estimated pick-up pose; if the feasibility score of the target pick-up pose in at least one estimated pick-up pose is greater than or equal to the preset feasibility threshold, control the pick-up and place device to pick up the target object through the target pick-up pose; if the feasibility scores of at least one estimated pick-up pose are all less than the preset feasibility threshold, determine at least one estimated pushing action corresponding to the target object based on the scene information; input each estimated pushing action and the second point cloud data corresponding to the target container into the pushing evaluation module in the preset evaluation model to obtain the effectiveness score of each estimated pushing action; based on the effectiveness score of each estimated pushing action, determine the target pushing action among at least one estimated pushing action, and control the pick-up and place device to execute the target pushing action.

[0307] In some examples, after determining in step 1120 that the current scene meets the second constraint condition, the steps of Scheme Two can be executed to retrieve the target object. It should be noted that the specific implementation process of Scheme Two has been described in detail in steps 810 to 870 of the above embodiments, and will not be repeated here to avoid repetition.

[0308] The item retrieval method provided in this application, through a hierarchical invocation and a third composite approach with mutual fallback, can dynamically select the optimal execution scheme based on scene characteristics and model state. When the scene is regular, geometrically simple, and the model is reliable, the efficient Scheme 1 is adopted; when the scene is complex, geometrically irregular, and the model is reliable, the more generalizable Scheme 2 is adopted. Furthermore, when the model is unreliable, it also falls back to Scheme 1 to ensure a safety lower bound; when the rule-based solution has no solution, Scheme 2 attempts to overcome the obstacle. This third composite approach ensures that the robot can operate stably in various scenarios, preventing safety risks due to model misjudgments and avoiding unsolvable shutdowns due to rule limitations, thereby further improving the system's robustness and scene adaptability.

[0309] Figure 12 This is a schematic diagram of an item-retrieving device provided in an embodiment of this application. Figure 12 As shown, the item retrieval device 1200 includes an acquisition module, a determination module, and a control module. Wherein: The acquisition module 1210 is configured to acquire scene information corresponding to the target container; wherein, the scene information includes the spatial distribution information of multiple items in the target container.

[0310] The determination module 1220 is configured to: during the process of the robot's pick-up and place-down device picking up a target object from a target container, in response to a failure event of the pick-up and place-down device picking up the target object, determine the candidate pushing object corresponding to the target object among multiple items in the target container; based on scene information and the geometric features of each candidate pushing object, determine the candidate pushing strategy corresponding to each candidate pushing object; wherein, the candidate pushing strategy includes pushing parameters and a first score, the pushing parameters include at least the candidate pushing object, the first pushing direction corresponding to the candidate pushing object, the first contact point and the first pushing distance, and the first score is used to indicate the execution priority of each candidate pushing strategy; and based on the first score of each candidate pushing strategy, determine the target pushing strategy among multiple candidate pushing strategies.

[0311] The control module 1230 is configured to control the pick-and-place device to push the candidate push object corresponding to the target push strategy in the target container in order to pick up the target object.

[0312] In some embodiments, the determining module 1220 is further configured to: in response to a failure event of the pick-and-place device to pick up the target object, determine at least one estimated pick-up pose corresponding to the target object based on scene information. The at least one estimated pick-up pose and the first point cloud data corresponding to the target container are input to the pick-up evaluation module in a preset evaluation model to obtain a feasibility score for each estimated pick-up pose; wherein the feasibility score is used to indicate the probability that the pick-and-place device can successfully pick up the target object from the target container based on the estimated pick-up pose. The control module 1230 is further configured to: if the feasibility score corresponding to the target pick-up pose in at least one of the estimated pick-up poses is greater than or equal to a preset feasibility threshold, control the pick-and-place device to pick up the target object using the target pick-up pose.

[0313] In some embodiments, the determining module 1220 is further configured to: determine at least one estimated pushing action corresponding to the target object based on scene information when the feasibility scores of at least one estimated pick-up pose are all less than a preset feasibility threshold; wherein each estimated pushing action includes at least an estimated pushing object corresponding to the target object, and a second pushing direction, a second contact point, and a second pushing distance corresponding to the estimated pushing object. Each estimated pushing action and the second point cloud data corresponding to the target container are input to the pushing evaluation module in a preset evaluation model to obtain an effectiveness score for each estimated pushing action; wherein the effectiveness score indicates the probability that the target object will become pickable after the pick-up / placement device performs the estimated pushing action. Based on the effectiveness scores of each estimated pushing action, a target pushing action is determined from at least one estimated pushing action. The control module 1230 is further configured to: control the pick-up / placement device to perform the target pushing action.

[0314] In some embodiments, the acquisition module 1210 is further configured to: acquire updated scenario information corresponding to the target container. The determination module 1220 is further configured to: based on the updated scenario information, re-output the feasibility score corresponding to the target object through the evaluation module until the feasibility score of the target object exceeds a preset feasibility threshold.

[0315] In some embodiments, the determining module 1220 is further configured to: input the second point cloud data corresponding to each candidate pushing strategy and the target container into the pushing evaluation module in the preset evaluation model to obtain a second score corresponding to each candidate pushing strategy; wherein, the second score is used to indicate the probability that the target object becomes a pickable state after the pick-and-place device executes the candidate pushing strategy. Based on the first score in each candidate pushing strategy, a target pushing strategy is determined among multiple candidate pushing strategies, including: determining a target score for each candidate pushing strategy based on the first score and the second score; and determining a target pushing strategy among the candidate pushing strategies based on the target score of each candidate pushing strategy.

[0316] In some embodiments, the determining module 1220 is further configured to: determine result data of the pick-up and place device performing the target pushing action; and adjust a preset adjustable parameter based on the result data; wherein the preset adjustable parameter includes at least a preset pushing ratio factor for determining the first pushing distance.

[0317] In some embodiments, the determining module 1220 is further configured to: in response to a failure event of the pick-up and place device to pick up the target object, determine evaluation parameters corresponding to the current scenario; wherein the evaluation parameters include at least scenario complexity, geometric features of the target object, execution success rate of a preset evaluation model, and confidence level of the preset evaluation model; the execution success rate is used to indicate the probability that the pick-up and place device can successfully pick up the target object based on the preset evaluation model; the confidence level is used to indicate the probability that the feasibility score and / or validity output by the preset evaluation model is higher than a preset score threshold. Based on the evaluation parameters corresponding to the current scenario, determine that the current scenario satisfies a first constraint condition or a second constraint condition. If the current scenario satisfies the first constraint condition, perform the step of determining a candidate push object corresponding to the target object among multiple items in the target container. If the current scenario satisfies the second preset condition, perform the step of determining at least one estimated pick-up pose corresponding to the target object based on scenario information.

[0318] In some embodiments, the determining module 1220 is further configured to: determine failure information corresponding to the failure event in response to a failure event of the pick-up and place device to pick up the target object; wherein the failure information includes at least one of an error code and / or an obstruction item index. If the failure information indicates that the pick-up and place device has experienced a path collision and there is an obstruction item corresponding to the path collision, the obstruction item is determined as a candidate push object. If the failure information indicates that the pick-up and place device has not experienced a path collision or there is no obstruction item, a candidate push object corresponding to the target object is determined from multiple items based on scene information.

[0319] In some embodiments, the determining module 1220 is configured to: determine the pose information of each item among a plurality of items based on scene information, and determine the instability score corresponding to each item based on the size information and / or pose information of each item; determine the surrounding spatial information of the location of each item, and determine the perturbability score corresponding to each item based on the surrounding spatial information of the location of each item; determine items whose instability score exceeds a first preset threshold, or whose perturbability score exceeds a second preset threshold, as candidate items, and determine the basic score of each candidate item based on the instability score and the perturbability score; and determine candidate pushing objects among the candidate items based on the basic scores of each candidate item.

[0320] In some embodiments, the determining module 1220 is configured to: determine an occlusion score for each candidate item based on the degree of occlusion of other items by each candidate item; determine a centering score for each candidate item based on the distance between each candidate item and the center position of the target container; determine a comprehensive score for each candidate item based on its base score, occlusion score, and centering score; and determine at least one candidate item whose comprehensive score meets a preset pushing condition as a candidate pushing object.

[0321] In some embodiments, the determining module 1220 is further configured to: determine the candidate item with the highest comprehensive score as the candidate to be promoted when the comprehensive scores of each candidate item do not meet the preset promotion conditions.

[0322] In some embodiments, when the pushing parameter in the candidate pushing strategy is a first pushing direction, the determining module 1220 is configured to: determine the size of the surrounding space and the crowding of surrounding items for each candidate pushing object based on scene information; and determine at least one first pushing direction for each candidate pushing object and a direction score for each first pushing direction based on the size of the surrounding space and the crowding of surrounding items.

[0323] In some embodiments, when the pushing parameter in the candidate pushing strategy is the first contact point, the determining module 1220 is configured to: determine the minimum distance between the surface of each candidate pushing object and the surface of the target container based on scene information; determine the position area of ​​each candidate pushing object in the target container based on the minimum distance between the surface of each candidate pushing object and the surface of the target container; determine the first contact point in the top area of ​​the candidate pushing object when the candidate pushing object is located in the edge area of ​​the target container; and determine the first contact point in the side area of ​​the candidate pushing object when the candidate pushing object is located in the middle area of ​​the target container.

[0324] In some embodiments, when the pushing parameter in the candidate pushing strategy is a first pushing distance, the determining module 1220 is configured to: determine the height of the first contact point of each candidate pushing object, the vertical height of each candidate pushing object, and the horizontal dimension of each candidate pushing object based on the geometric features of each candidate pushing object; determine the moving distance of each candidate pushing object based on the height of the first contact point of each candidate pushing object, the vertical height of each candidate pushing object, and the horizontal dimension of each candidate pushing object; determine the preset minimum pushing distance as the first pushing distance when the moving distance is less than a preset minimum pushing distance; determine the maximum permitted pushing distance as the first pushing distance when the moving distance is greater than a preset maximum permitted pushing distance; and determine the moving distance as the first pushing distance when the moving distance is greater than the preset minimum pushing distance and less than the preset maximum permitted pushing distance.

[0325] In some embodiments, the above-mentioned moving distance satisfies formula (1): D1 = h × (W / H) × α. The above-mentioned first pushing distance satisfies the following formula (2): D2 = clamp (D1, Dmin, Dmax). Wherein, D1 is the moving distance corresponding to the candidate pushing object, h is the height of the first contact point of the candidate pushing object, W is the horizontal dimension of the candidate pushing object, H is the vertical height of the candidate pushing object, α is the preset pushing ratio factor; D2 is the first pushing distance of the candidate pushing object, Dmin is the preset minimum pushing distance, and Dmax is the maximum permissible distance.

[0326] In some embodiments, the determining module 1220 is further configured to: determine a first gap; wherein the first gap is a spatial gap along a first pushing direction, with the size of the end effector of the pick-and-place device as the corridor width; determine a second gap; wherein the second gap is a spatial gap along the first pushing direction and in a direction perpendicular to the first pushing direction, with the size of the candidate push object as the corridor width; and determine the smaller of the first gap and the second gap as the maximum permissible distance.

[0327] In some embodiments, the determining module 1220 is configured to: determine the pushing parameters corresponding to the candidate pushing object based on the candidate pushing object, the first pushing direction corresponding to the candidate pushing object, the first contact point corresponding to the first pushing direction, and the first pushing distance corresponding to the first contact point; determine the first score in the candidate pushing strategy based on the comprehensive score corresponding to the candidate pushing object and the directional score of the first pushing direction corresponding to the candidate pushing object; and determine the candidate pushing strategy corresponding to the candidate pushing object based on the pushing parameters and the first score.

[0328] In some embodiments, the determining module 1220 is configured to: perform feasibility checks on each candidate pushing strategy sequentially based on a first score for each candidate pushing strategy; and determine the first candidate pushing strategy that passes the feasibility check as the target pushing strategy. The feasibility check includes kinematic feasibility check and / or collision safety check; the kinematic feasibility check includes at least: inverse kinematics solution of the path point sequence executing the candidate pushing strategy, joint space path interpolation collision detection, and singular configuration detection. The collision safety check includes at least: checking the rigid structure of the pick-and-place device using a first collision tolerance, checking the contact pose of the pick-and-place device during the execution of the candidate pushing strategy using a second collision tolerance, and checking each candidate pushing object using a third collision tolerance.

[0329] In some embodiments, the determining module 1220 is further configured to: determine that there is no valid driving strategy if multiple candidate driving strategies fail the feasibility check, and instruct the output device to output an abnormal prompt message.

[0330] In some embodiments, the determining module 1220 is further configured to: perform kinematic feasibility verification and collision safety verification on each candidate pushing strategy in sequence based on the first score of each candidate pushing strategy; and, if multiple candidate pushing strategies fail the kinematic feasibility verification and collision safety verification, perform a step of determining at least one estimated take-up pose corresponding to the target object based on scene information.

[0331] Figure 13 This is a schematic diagram of a robot provided as an embodiment of this application. Figure 13 As shown, the robot 1300 includes a control device 1310 and a pick-and-place device 1320. Wherein: The control device 1310 is configured to: acquire scene information corresponding to the target container; wherein the scene information includes spatial distribution information of multiple items in the target container. During the process of the robot's pick-and-place device retrieving a target object from the target container, in response to a failure event of the pick-and-place device retrieving the target object, candidate pushing objects corresponding to the target object are determined among the multiple items in the target container. Based on the scene information and the geometric features of each candidate pushing object, candidate pushing strategies are determined for each candidate pushing object; wherein the candidate pushing strategy includes pushing parameters and a first score, the pushing parameters including at least the candidate pushing object, a first pushing direction corresponding to the candidate pushing object, a first contact point, and a first pushing distance, and the first score is used to indicate the execution priority of each candidate pushing strategy. Based on the first scores of each candidate pushing strategy, a target pushing strategy is determined among the multiple candidate pushing strategies, and a target execution instruction is generated based on the target pushing strategy.

[0332] The pick-and-place device 1320 is configured to: based on the target execution instruction, push the candidate push object corresponding to the target push strategy in the target container in accordance with the target push strategy to pick up the target object.

[0333] It should be noted that the item picking device and robot provided in this application embodiment are used to execute the corresponding item picking method provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0334] Figure 14 This is a schematic diagram of an electronic device provided in an embodiment of this application. In some embodiments, the electronic device includes one or more processors and a memory. The memory is configured to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the item retrieval method in the above embodiments.

[0335] like Figure 14As shown, the electronic device 1000 includes a processor 1001 and a memory 1002. Exemplarily, the electronic device 1000 may also include a communications interface 1003 and a communications bus 1004.

[0336] The processor 1001, memory 1002, and communication interface 1003 communicate with each other via communication bus 1004. Communication interface 1003 is used to communicate with other network elements such as clients or other servers.

[0337] In some embodiments, the processor 1001 is used to execute program 1005, specifically performing the relevant steps in the above-described embodiment of the item retrieval method. Specifically, program 1005 may include program code, which includes computer-executable instructions.

[0338] For example, processor 1001 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. Electronic device 1000 may include one or more processors, which may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0339] In some embodiments, memory 1002 is used to store program 1005. Memory 1002 may include high-speed RAM memory, and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0340] Specifically, program 1005 can be called by processor 1001 to cause electronic device 1000 to perform the operation of taking items.

[0341] This application provides a computer-readable storage medium storing at least one executable instruction. When the executable instruction is executed on an electronic device 1000, the electronic device 1000 performs the item retrieval method described in the above embodiment.

[0342] The executable instructions can be used to cause the electronic device 1000 to perform the operation of taking items.

[0343] For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0344] In some embodiments, this application provides a computer program product including a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions that, when executed by a computer, cause the computer to perform the article retrieval method described in any of the above embodiments.

[0345] In some embodiments, this application also provides a computer program that, when executed by a processor, can implement the item retrieval method described in any of the above embodiments.

[0346] The beneficial effects that the article-grabbing device, robot, electronic device, computer-readable storage medium, computer program product, and computer program provided in this application embodiment can achieve are similar to the beneficial effects of the article-grabbing method provided above, and will not be repeated here.

[0347] It should be noted that in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0348] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0349] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0350] For the purposes of this specification, "computer-readable medium" can mean any means that can contain, store, communicate, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0351] More specific examples of computer-readable media (a non-exhaustive list) include the following: electrical connections having one or more wires (electronic devices), portable computer disks (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM).

[0352] Furthermore, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory. It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof.

[0353] In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0354] The embodiments described above do not constitute a limitation on the scope of protection of this application.

Claims

1. A method for taking an item, characterized in that, The method includes: During the process of the robot's picking and placing device retrieving the target object from the target container, scene information corresponding to the target container is acquired; wherein, the scene information includes the spatial distribution information of multiple items in the target container; In response to a failure event of the pick-up and put-down device to pick up the target object, a candidate push object corresponding to the target object is determined from multiple items in the target container; Based on the scene information and the geometric features of each candidate pushing object, a candidate pushing strategy corresponding to each candidate pushing object is determined; wherein, the candidate pushing strategy includes pushing parameters and a first score, the pushing parameters include at least the candidate pushing object, the first pushing direction corresponding to the candidate pushing object, the first contact point and the first pushing distance, and the first score is used to indicate the execution priority of each candidate pushing strategy; Based on the first score of each of the candidate pushing strategies, a target pushing strategy is determined among the multiple candidate pushing strategies, and the pick-and-place device is controlled to execute the target pushing strategy to pick up the target object.

2. The method according to claim 1, characterized in that, The method further includes: In response to a failure event of the pick-and-place device to pick up the target object, at least one estimated pick-up pose corresponding to the target object is determined based on the scene information; The first point cloud data corresponding to the at least one estimated pick-up pose and the target container is input into the pick-up evaluation module in the preset evaluation model to obtain the feasibility score of each estimated pick-up pose; wherein, the feasibility score is used to indicate the probability that the pick-up and place device can successfully pick up the target object from the target container based on the estimated pick-up pose; If, among the at least one estimated retrieval pose, there exists a feasibility score greater than or equal to a preset feasibility threshold corresponding to the target retrieval pose, the retrieval and placement device is controlled to retrieve the target object through the target retrieval pose.

3. The method according to claim 2, characterized in that, The method further includes: If the feasibility scores of at least one estimated grasping pose are all less than the preset feasibility threshold, at least one estimated pushing action corresponding to the target object is determined based on the scene information; wherein each estimated pushing action includes at least an estimated pushing object corresponding to the target object, and a second pushing direction, a second contact point, and a second pushing distance corresponding to the estimated pushing object; The estimated pushing action and the second point cloud data corresponding to the target container are input into the pushing evaluation module in the preset evaluation model to obtain the effectiveness score of each estimated pushing action; wherein, the effectiveness score is used to indicate the probability that the target object becomes a pickable state after the pick-and-place device performs the estimated pushing action; Based on the effectiveness score of each of the estimated pushing actions, a target pushing action is determined among the at least one estimated pushing action, and the pick-and-place device is controlled to execute the target pushing action.

4. The method according to claim 3, characterized in that, After controlling the pick-and-place device to perform the target pushing action, the method further includes: Obtain the updated scenario information corresponding to the target container; Based on the updated scenario information, the feasibility score corresponding to the target object is re-output through the acquisition and evaluation module until the feasibility score of the target object exceeds the preset feasibility threshold.

5. The method according to claim 3, characterized in that, After determining the candidate push strategy corresponding to each of the candidate push objects, the method further includes: The third point cloud data corresponding to each candidate pushing strategy and the target container is input into the pushing evaluation module in the preset evaluation model to obtain a second score corresponding to each candidate pushing strategy; wherein, the second score is used to indicate the probability that the target object becomes a pickable state after the pick-and-place device executes the candidate pushing strategy; The step of determining the target push strategy from multiple candidate push strategies based on the first score of each of the candidate push strategies includes: Based on the first score and the second score, the target score for each candidate promotion strategy is determined; Based on the target scores of each candidate driving strategy, the target driving strategy is determined from the candidate driving strategies.

6. The method according to claim 3, characterized in that, After the method controls the pick-and-place device to perform the target pushing action, the method further includes: Determine the result data of the pick-and-place device performing the target pushing action; Based on the result data, the preset adjustable parameters are adjusted; wherein, the preset adjustable parameters include at least a preset push ratio factor for determining the first push distance.

7. The method according to claim 3, characterized in that, The method further includes: In response to a failure event of the pick-and-place device to retrieve the target object, evaluation parameters corresponding to the current scene are determined; wherein, the evaluation parameters include at least scene complexity, geometric features of the target object, execution success rate of the preset evaluation model, and confidence level of the preset evaluation model; the execution success rate is used to indicate the probability that the pick-and-place device can successfully retrieve the target object based on the preset evaluation model; the confidence level is used to indicate the probability that the feasibility score and / or effectiveness output by the preset evaluation model is higher than a preset score threshold; Based on the evaluation parameters corresponding to the current scenario, it is determined that the current scenario satisfies the first constraint condition or the second constraint condition; If the current scenario satisfies the first constraint condition, the step of determining the candidate push object corresponding to the target object among the multiple items in the target container is executed; If the current scene meets the second preset condition, the step of determining at least one estimated retrieval pose corresponding to the target object based on the scene information is executed.

8. The method according to claim 1, characterized in that, In response to a failure event of the pick-up and drop-off device to pick up the target object, determining a candidate push object corresponding to the target object from among multiple items in the target container includes: In response to a failure event of the pick-and-place device to pick up the target object, failure information corresponding to the failure event is determined; wherein, the failure information includes an error code and / or an index of the obstructing item; If the failure information indicates that the pick-and-place device has experienced a path collision and there is an obstructing object corresponding to the path collision, the obstructing object is identified as the candidate pushing object; If the failure information indicates that the pick-up and place device has not experienced a path collision or that there are no obstructing items, a candidate push object corresponding to the target object is determined from among the multiple items based on the scene information.

9. The method according to claim 1, characterized in that, Determining candidate push objects corresponding to the target object from among multiple items in the target container, including: Based on the scene information, determine the pose information of each of the multiple items, and based on the size information and / or pose information of each item, determine the instability score corresponding to each item; Determine the surrounding spatial information of the location of each item, and based on the surrounding spatial information of the location of each item, determine the perturbability score corresponding to each item; Items whose instability score exceeds a first preset threshold, or whose perturbability score exceeds a second preset threshold, are identified as candidate items, and a base score for each candidate item is determined based on the instability score and the perturbability score. Based on the baseline scores of each candidate item, the candidate promotion target is determined among the candidate items.

10. The method according to claim 9, characterized in that, The process of determining the candidate promotion target from among the candidate items based on their base scores includes: Based on the degree of occlusion of each candidate item on other items, an occlusion score is determined for each candidate item. Based on the distance between each candidate item and the center position of the target container, the centering score corresponding to each candidate item is determined; Based on the base score, the occlusion score, and the centering score of each candidate item, a comprehensive score for each candidate item is determined. At least one candidate item whose comprehensive score meets the preset promotion conditions is identified as the candidate promotion object.

11. The method according to claim 10, characterized in that, The method further includes: If none of the candidate items meet the preset promotion conditions in terms of overall score, the candidate item with the highest overall score will be determined as the candidate to be promoted.

12. The method according to claim 1, characterized in that, When the pushing parameter in the candidate pushing strategy is the first pushing direction, determining the candidate pushing strategy corresponding to each candidate pushing object based on the scene information and the geometric features of each candidate pushing object includes: Based on the scene information, determine the size of the space around each candidate pushing object and the crowding of surrounding items; Based on the size of the space surrounding each candidate pushing object and the crowding of surrounding objects, at least one first pushing direction corresponding to each candidate pushing object is determined, as well as the direction score corresponding to each first pushing direction.

13. The method according to claim 12, characterized in that, When the pushing parameter in the candidate pushing strategy is the first contact point, determining the candidate pushing strategy corresponding to each candidate pushing object based on the scene information and the geometric features of each candidate pushing object includes: Based on the scene information, determine the minimum distance between the surface of each candidate pushing object and the surface of the target container; Based on the minimum distance between the surface of each candidate pushing object and the surface of the target container, the position region of each candidate pushing object in the target container is determined; If the candidate push object is located in the edge region of the target container, it is determined that the first contact point is located in the top region of the candidate push object; If the candidate push object is located in the middle region of the target container, the first contact point is determined to be located in the side region of the candidate push object.

14. The method according to claim 13, characterized in that, When the pushing parameter in the candidate pushing strategy is the first pushing distance, determining the candidate pushing strategy corresponding to each candidate pushing object based on the scene information and the geometric features of each candidate pushing object includes: Based on the geometric features of each candidate pushing object, the height of the first contact point of each candidate pushing object, the vertical height of each candidate pushing object, and the horizontal dimension are determined. The moving distance of each candidate pushing object is determined based on the height of the first contact point of each candidate pushing object, the vertical height of each candidate pushing object, and the horizontal dimension of each candidate pushing object; If the moving distance is less than the preset minimum pushing distance, the preset minimum pushing distance is determined as the first pushing distance; If the moving distance is greater than the preset maximum permissible pushing distance, the maximum permissible pushing distance is determined as the first pushing distance; If the moving distance is greater than the preset minimum pushing distance and less than the preset maximum permissible pushing distance, the moving distance is determined as the first pushing distance.

15. The method according to claim 14, characterized in that, The distance traveled satisfies the following formula (1): D1 = h × (W / H) × α; Formula (1); The first pushing distance satisfies the following formula (2): D2 = clamp(D1, Dmin, Dmax); Formula (2); Wherein, D1 is the moving distance corresponding to the candidate pushing object, h is the height of the first contact point of the candidate pushing object, W is the horizontal dimension of the candidate pushing object, H is the vertical height of the candidate pushing object, and α is the preset pushing ratio factor; D2 is the first pushing distance of the candidate pushing object, Dmin is the preset minimum pushing distance, and Dmax is the maximum permitted distance.

16. The method according to claim 14, characterized in that, The method further includes: A first gap is determined; wherein the first gap is a spatial gap along the first pushing direction, with the size of the end effector of the pick-and-place device as the corridor width; Determine a second gap; wherein the second gap is a spatial gap determined along the first pushing direction and the direction perpendicular to the first pushing direction, with the size of the candidate pushing object as the width of the corridor; The smaller of the first gap and the second gap is determined as the maximum permissible distance.

17. The method according to claim 14, characterized in that, The determination of the candidate push strategy corresponding to each of the candidate push objects includes: Based on the candidate pushing object, the first pushing direction corresponding to the candidate pushing object, the first contact point corresponding to the first pushing direction, and the first pushing distance corresponding to the first contact point, the pushing parameters corresponding to the candidate pushing object are determined; Based on the comprehensive score corresponding to the candidate push object and the directional score of the first push direction corresponding to the candidate push object, the first score in the candidate push strategy is determined; Based on the driving parameters and the first score, a candidate driving strategy corresponding to the candidate driving object is determined.

18. The method according to claim 1, characterized in that, The determination of the target push strategy from multiple candidate push strategies based on the first scores of each candidate push strategy includes: Based on the first score of each candidate driving strategy, the feasibility of each candidate driving strategy is verified in turn. The first candidate driving strategy that passes the feasibility check is determined as the target driving strategy; The feasibility verification includes kinematic feasibility verification and / or collision safety verification; The kinematic feasibility verification includes at least: inverse kinematics solution of the path point sequence for executing the candidate propulsion strategy, joint space path interpolation collision detection, and singular configuration detection. The collision safety verification includes at least the following: verifying the rigid structure of the pick-up and place device using a first collision tolerance, verifying the contact pose of the pick-up and place device during the execution of the candidate pushing strategy using a second collision tolerance, and verifying each candidate pushing object using a third collision tolerance.

19. The method according to claim 18, characterized in that, The method further includes: If none of the candidate driving strategies pass the feasibility check, it is determined that there is no effective driving strategy, and the output device is instructed to output an abnormal prompt message.

20. The method according to claim 3, characterized in that, The method further includes: Based on the first score of each candidate propulsion strategy, kinematic feasibility and collision safety are verified sequentially for each candidate propulsion strategy. If none of the multiple candidate push strategies pass the kinematic feasibility check and the collision safety check, the step of determining at least one estimated take-up pose corresponding to the target object based on the scene information is executed.

21. An item-retrieving device, characterized in that, include: The acquisition module is configured to: acquire scene information corresponding to the target container; wherein, the scene information includes spatial distribution information of multiple items in the target container; The determination module is configured to: during the process of the robot's pick-and-place device retrieving a target object from the target container, in response to a failure event of the pick-and-place device retrieving the target object, determine candidate pushing objects corresponding to the target object from multiple items in the target container; based on the scene information and the geometric features of each candidate pushing object, determine candidate pushing strategies corresponding to each candidate pushing object; wherein, the candidate pushing strategy includes pushing parameters and a first score, the pushing parameters including at least the candidate pushing object, a first pushing direction corresponding to the candidate pushing object, a first contact point, and a first pushing distance, the first score being used to indicate the execution priority of each candidate pushing strategy; and based on the first score of each candidate pushing strategy, determine a target pushing strategy from multiple candidate pushing strategies. The control module is configured to control the pick-and-place device to push the candidate push object corresponding to the target push strategy in the target container according to the target push strategy, so as to pick up the target object.

22. A robot, characterized in that, include: The control device is configured as follows: Obtain scene information corresponding to the target container; wherein, the scene information includes spatial distribution information of multiple items in the target container; During the process of the robot's pick-up and place device retrieving the target object from the target container, in response to a failure event of the pick-up and place device retrieving the target object, a candidate push object corresponding to the target object is determined from multiple items in the target container; Based on the scene information and the geometric features of each candidate pushing object, a candidate pushing strategy corresponding to each candidate pushing object is determined; wherein, the candidate pushing strategy includes pushing parameters and a first score, the pushing parameters include at least the candidate pushing object, the first pushing direction corresponding to the candidate pushing object, the first contact point and the first pushing distance, and the first score is used to indicate the execution priority of each candidate pushing strategy; Based on the first score of each of the candidate push strategies, a target push strategy is determined among the multiple candidate push strategies, and a target execution instruction is generated based on the target push strategy; The pick-and-place device is configured to: based on the target execution instruction, push the candidate push object corresponding to the target push strategy in the target container according to the target push strategy, so as to pick up the target object.

23. An electronic device, characterized in that, include: One or more processors and memory; The memory is configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the item retrieval method according to any one of claims 1-20.

24. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method for taking an item according to any one of claims 1-20.

25. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the article-taking method according to any one of claims 1-20.

Images