Object grabbing method, device and system

Abstract

The invention provides an object grabbing method, device and system, and the method comprises the steps: determining a target gap of a target container based on the pose of an object, and determining the object located at the edge of the target gap as a to-be-grabbed object; acquiring a grabbing pose, a first obstacle avoidance pose and a second obstacle avoidance pose of the to-be-grabbed object; the grabbing posture is the actual posture of the to-be-grabbed object, the first obstacle avoidance posture is the posture located in a target gap, and the second obstacle avoidance posture is the posture located in an upper layer area of the to-be-grabbed object of a target container; and based on the grabbing posture, the first obstacle avoidance posture and the second obstacle avoidance posture, the mechanical arm is controlled to move to the grabbing posture and grab the to-be-grabbed object, the mechanical arm is controlled to move to the first obstacle avoidance posture from the grabbing posture, and the mechanical arm is controlled to move to the second obstacle avoidance posture from the first obstacle avoidance posture. According to the technical scheme, obstacle avoidance of object grabbing can be achieved.

Description

Technical Field

[0001] This application relates to the field of robot control, and in particular, to an object grasping method, device, and system. Background Art

[0002] A material box is a container for loading various objects (objects can also be referred to as workpieces or materials), and usually multiple layers of objects are loaded in the container. To improve automation and reduce labor costs, a robotic arm can be used for automatic grasping, that is, the robotic arm sequentially grasps each object in the container to improve the sorting efficiency.

[0003] However, since the intervals between the objects in the container are small, when using a robotic arm to grasp the objects in the container, the robotic arm may collide with other objects, causing damage to the other objects. In addition, when using a robotic arm to grasp the objects in the container, the robotic arm may also collide with the container, causing damage to the container. Summary of the Invention

[0004] This application provides an object grasping method. A plurality of objects are placed in a target container, and the method includes:

[0005] Determining a target gap of the target container based on the poses of the objects, and determining the objects located at the edge of the target gap as the objects to be grasped;

[0006] Obtaining a grasping pose, a first obstacle avoidance pose, and a second obstacle avoidance pose of the object to be grasped; wherein, the grasping pose is the actual pose of the object to be grasped, the first obstacle avoidance pose is the pose located in the target gap, and the second obstacle avoidance pose is the pose located in the upper area of the object to be grasped in the target container;

[0007] Controlling the robotic arm to move to the grasping pose and grasp the object to be grasped based on the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose, controlling the robotic arm to move from the grasping pose to the first obstacle avoidance pose, and controlling the robotic arm to move from the first obstacle avoidance pose to the second obstacle avoidance pose.

[0008] This application provides an object grasping device. A plurality of objects are placed in a target container, and the device includes:

[0009] A determining module, configured to determine a target gap of the target container based on the poses of the objects, and determine the objects located at the edge of the target gap as the objects to be grasped;

[0010] An acquisition module, configured to acquire the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose of the object to be grasped; wherein, the grasping pose is the actual pose of the object to be grasped, the first obstacle avoidance pose is the pose located in the target gap, and the second obstacle avoidance pose is the pose located in the upper area of the object to be grasped in the target container;

[0011] A control module, configured to control the robotic arm to move to the grasping pose and grasp the object to be grasped, control the robotic arm to move from the grasping pose to the first obstacle avoidance pose, and control the robotic arm to move from the first obstacle avoidance pose to the second obstacle avoidance pose based on the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose.

[0012] This application provides an object grasping system. Multiple objects are placed in a target container. The object grasping system includes a depth camera, a robotic arm, and a control device for the robotic arm, wherein:

[0013] The depth camera is configured to collect a depth image and send the depth image to the control device;

[0014] The control device is configured to obtain the pose of the object based on the depth image, determine the target gap of the target container based on the pose of the object, and determine the object located at the edge of the target gap as the object to be grasped; acquire the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose of the object to be grasped; wherein, the grasping pose is the actual pose of the object to be grasped, the first obstacle avoidance pose is the pose located in the target gap, and the second obstacle avoidance pose is the pose located in the upper area of the object to be grasped in the target container; and send a control instruction to the robotic arm, where the control instruction includes the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose;

[0015] The robotic arm is configured to, after receiving the control instruction, control the robotic arm to move to the grasping pose and grasp the object to be grasped, control the robotic arm to move from the grasping pose to the first obstacle avoidance pose, and control the robotic arm to move from the first obstacle avoidance pose to the second obstacle avoidance pose based on the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose.

[0016] This application provides an electronic device, including: a processor and a machine-readable storage medium. The machine-readable storage medium stores machine-executable instructions that can be executed by the processor; wherein, the processor is configured to execute the machine-executable instructions to implement the object grasping method in the above examples of this application.

[0017] This application provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, it implements the object grasping method in the above examples of this application.

[0018] The present application provides a machine-readable storage medium storing machine-executable instructions executable by a processor; wherein, the processor is configured to execute the machine-executable instructions to implement the object grasping method of the above example of the present application.

[0019] As can be seen from the above technical solutions, in the embodiments of the present application, the target gap of the target container is determined based on the pose of the object, and the object located at the edge of the target gap is determined as the object to be grasped, and the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose of the object to be grasped are obtained. Since the first obstacle avoidance pose is the pose located in the target gap and the second obstacle avoidance pose is the pose located in the upper area of the object to be grasped in the target container, therefore, by controlling the robotic arm to move from the grasping pose to the first obstacle avoidance pose and controlling the robotic arm to move from the first obstacle avoidance pose to the second obstacle avoidance pose, obstacle avoidance during object grasping can be achieved, and obstacle avoidance in all directions of the object is considered. Even if the interval between objects in the container is small, when using a robotic arm to grasp an object in the container, the robotic arm will not collide with other objects and will not cause damage to other objects. When using a robotic arm to grasp an object in the container, the robotic arm will not collide with the container and will not cause damage to the container. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a schematic flowchart of an object grasping method in an embodiment of the present application;

[0021] Figure 2 is a schematic flowchart of an object grasping method in an embodiment of the present application;

[0022] Figure 3A is a schematic diagram of the region where the intercepted detection ROI is located in an embodiment of the present application;

[0023] Figure 3B is a schematic diagram of three-dimensional points of a container model in an embodiment of the present application;

[0024] Figure 3C is a schematic diagram of three-dimensional points of an object model in an embodiment of the present application;

[0025] Figure 4A is a schematic diagram of multiple objects in a target container in an embodiment of the present application;

[0026] Figure 4B is a schematic diagram of the central pose of an object region in an embodiment of the present application;

[0027] Figure 4C is a comparison schematic diagram of the central pose of an object region and the central pose of a container in the present application;

[0028] Figure 4D Schematic diagram of an object located at the edge of a target gap in an embodiment of the present application;

[0029] Figure 4E Schematic diagram of the grasping sequence in an embodiment of the present application;

[0030] Figure 5A Schematic diagram of an object to be grasped being close to the edge in an embodiment of the present application;

[0031] Figure 5B Schematic diagram of there being no blank direction in any of the four directions of the target container in an embodiment of the present application;

[0032] Figure 5C Schematic diagram of the moving direction of an object in an embodiment of the present application;

[0033] Figure 6 Schematic diagram of the structure of an object grasping device in an embodiment of the present application;

[0034] Figure 7 Hardware structure diagram of an electronic device in an embodiment of the present application. Specific embodiments

[0035] In an embodiment of the present application, an object grasping method is proposed. This method can be applied to the control device of a robotic arm. The control device is used to control the robotic arm to grasp an object in a target container, and multiple objects are placed in the target container. Refer to Figure 1 As shown, it is a flowchart of this method. This method may include:

[0036] Step 101: Determine the target gap of the target container based on the pose of the object, and determine the object located at the edge of the target gap as the object to be grasped.

[0037] Step 102: Obtain the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose of the object to be grasped; wherein, the grasping pose is the actual pose of the object to be grasped, the first obstacle avoidance pose is the pose located in the target gap, and the second obstacle avoidance pose is the pose located in the upper area of the object to be grasped in the target container.

[0038] Step 103: Based on the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose, control the robotic arm to move to the grasping pose and grasp the object to be grasped, control the robotic arm to move from the grasping pose to the first obstacle avoidance pose, and control the robotic arm to move from the first obstacle avoidance pose to the second obstacle avoidance pose.

[0039] Exemplarily, when determining the target void of the target container based on the pose of an object, the object here can be each object to be grasped in the acquired image, that is, the target void of the target container is determined based on the poses of each object (such as multiple objects). Regarding the pose of the object, the pose of the object can be the pose of the object in the container coordinate system of the target container, or the pose in other coordinate systems, such as the pose in the camera coordinate system, the pose in the robot coordinate system, the position in the world coordinate system, etc. There is no restriction on this and it can be the pose in any coordinate system. Hereinafter, the pose of the object in the container coordinate system of the target container is taken as an example.

[0040] Exemplarily, in order to improve efficiency, the pose of the object can be the pose of the object in the container coordinate system of the target container. Before determining the target void of the target container based on the pose of the object, a depth image can also be acquired by a depth camera, and the pose of the target container in the camera coordinate system of the depth camera and the pose of the object in the camera coordinate system can be obtained based on this depth image. Based on the conversion relationship between the camera coordinate system and the robot coordinate system, the pose of the target container in the camera coordinate system is converted into the reference pose of the target container in the robot coordinate system, and the pose of the object in the camera coordinate system is converted into the pose of the object in the robot coordinate system. A container coordinate system is established with this reference pose (i.e., the pose of the target container in the robot coordinate system) as the origin, and the pose of the object in the robot coordinate system is converted into the pose of the object in the container coordinate system.

[0041] Exemplarily, if the depth image includes multiple three-dimensional points, obtaining the pose of the target container in the camera coordinate system of the depth camera and the pose of the object in the camera coordinate system based on this depth image can include, but is not limited to: obtaining the first type of three-dimensional points of the target container and the second type of three-dimensional points of the object from the multiple three-dimensional points; determining the pose of the target container in the camera coordinate system based on the first type of three-dimensional points, and determining the pose of the object in the camera coordinate system based on the second type of three-dimensional points of the object. Among them, the similarity between the first type of three-dimensional points and the three-dimensional points of the container model is greater than the first threshold, and the similarity between the second type of three-dimensional points of the object and the three-dimensional points of the object model is greater than the second threshold; among them, the three-dimensional points of the container model are determined based on the depth image corresponding to the sample container, and this depth image only includes the sample container, and the three-dimensional points in this depth image are the three-dimensional points of the container model; among them, the three-dimensional points of the object model are determined based on the depth image corresponding to the sample object, and this depth image only includes the sample object, and the three-dimensional points in this depth image are the three-dimensional points of the object model.

[0042] Exemplarily, determining the target void of the target container based on the pose of the object can include, but is not limited to: fitting the central pose of the object area in the container based on the pose of the object; determining the central pose of the container of the target container; determining the target void based on the relative positions of the central pose of the object area and the central pose of the container.

[0043] Exemplarily, the central pose of the object area includes a first X-direction coordinate value and a first Y-direction coordinate value, and the central pose of the container includes a second X-direction coordinate value and a second Y-direction coordinate value; determining the target gap based on the relative positions of the central pose of the object area and the central pose of the container may include, but is not limited to: if the difference between the first X-direction coordinate value and the second X-direction coordinate value is greater than the difference between the first Y-direction coordinate value and the second Y-direction coordinate value, then determining the target gap as the left direction or the right direction based on the first X-direction coordinate value and the second X-direction coordinate value; if the difference between the first X-direction coordinate value and the second X-direction coordinate value is less than the difference between the first Y-direction coordinate value and the second Y-direction coordinate value, then determining the target gap as the lower direction or the upper direction based on the first Y-direction coordinate value and the second Y-direction coordinate value.

[0044] For example, when determining the target gap as the left direction or the right direction based on the first X-direction coordinate value and the second X-direction coordinate value, if the first X-direction coordinate value is on the left side of the second X-direction coordinate value (for example, when the positive direction of the X-axis is horizontally to the right, the first X-direction coordinate value is less than the second X-direction coordinate value), then the target gap is determined as the right direction. If the first X-direction coordinate value is on the right side of the second X-direction coordinate value (for example, the first X-direction coordinate value is greater than the second X-direction coordinate value), then the target gap is determined as the left direction.

[0045] For example, when determining the target gap as the lower direction or the upper direction based on the first Y-direction coordinate value and the second Y-direction coordinate value, if the first Y-direction coordinate value is on the upper side of the second Y-direction coordinate value (for example, when the positive direction of the Y-axis is horizontally downward, the first Y-direction coordinate value is less than the second Y-direction coordinate value), then the target gap is determined as the lower direction. If the first Y-direction coordinate value is on the lower side of the second Y-direction coordinate value (for example, the first Y-direction coordinate value is greater than the second Y-direction coordinate value), then the target gap is determined as the upper direction.

[0046] Exemplarily, determining the object located at the edge of the target gap as the object to be grasped may include, but is not limited to: if there are multiple objects located at the edge of the target gap, then determining the grasping order of the multiple objects, and determining the object to be grasped from the multiple objects based on the grasping order; wherein, there is at least one intermediate object and two edge objects among the multiple objects, the edge object is the object close to the target container, and the intermediate object is the remaining object except the edge object; in the grasping order, the intermediate object is in front of the edge object.

[0047] Exemplarily, obtaining the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose of the object to be grasped may include, but is not limited to: determining the first post-movement pose of the object to be grasped based on the pose of the object to be grasped (such as the pose of the object to be grasped in the container coordinate system), the first movement direction, and the configured movement distance; wherein, the first movement direction is a lateral movement direction or an oblique movement direction towards the target gap; determining the second post-movement pose of the object to be grasped based on the first post-movement pose, the configured height movement distance value, and the configured reference offset position; wherein, the height movement distance value is greater than or equal to the height of a single object, and the reference offset position includes the lateral offset distance and the longitudinal offset distance between the second post-movement pose and the origin position; converting the pose of the object to be grasped into the grasping pose, converting the first post-movement pose into the first obstacle avoidance pose, and converting the second post-movement pose into the second obstacle avoidance pose.

[0048] For example, the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose may be poses in the robot coordinate system of the robotic arm, or may be poses in other coordinate systems, such as poses in the world coordinate system, poses in the container coordinate system, etc., and there is no limitation on this. Taking the pose in the robot coordinate system as an example.

[0049] For example, the pose of the object to be grasped, the first post-movement pose, and the second post-movement pose may be poses in the container coordinate system of the target container, or may be poses in other coordinate systems, such as poses in the world coordinate system, poses in the robot coordinate system, etc. Taking the pose in the container coordinate system as an example.

[0050] In summary, the first post-movement pose of the object to be grasped in the container coordinate system can be determined based on the pose of the object to be grasped in the container coordinate system, the first movement direction, and the configured movement distance. The second post-movement pose of the object to be grasped in the container coordinate system is determined based on the first post-movement pose, the configured height movement distance value, and the configured reference offset position. Then, the pose of the object to be grasped in the container coordinate system can be converted into the grasping pose in the robot coordinate system, the first post-movement pose in the container coordinate system can be converted into the first obstacle avoidance pose in the robot coordinate system, and the second post-movement pose in the container coordinate system can be converted into the second obstacle avoidance pose in the robot coordinate system.

[0051] Exemplarily, after determining the object located at the edge of the target gap as the object to be grasped, the distance between the object to be grasped and the edge of the target container can also be determined; if the distance is not greater than the configured blank distance threshold, an alarm message is output; and / or, when determining that the object to be grasped moves to the first obstacle avoidance pose, whether the object to be grasped collides with the edge of the target container is determined. If so, an alarm message is output; wherein, the alarm message can indicate that the object to be grasped cannot be grasped.

[0052] For example, it is also possible to determine the distance between the object to be grasped and the edge of the target container based on the pose of the object to be grasped in the container coordinate system, the size of the target container, and the size of the object to be grasped. If the distance is not greater than the configured blank distance threshold, an alarm message can be output; if the distance is greater than the blank distance threshold, an operation to determine the first post-movement pose is performed, and based on the first post-movement pose and the size of the object to be grasped, it is determined whether the object to be grasped collides with the edge of the target container. If so, an alarm message can be output; if not, an operation to determine the second post-movement pose can be performed. For example, the alarm message can indicate that the object to be grasped cannot be grasped.

[0053] As can be seen from the above technical solutions, in the embodiments of the present application, based on the pose of the object, the target gap of the target container is determined, the object located at the edge of the target gap is determined as the object to be grasped, and the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose of the object to be grasped are obtained. Since the first obstacle avoidance pose is the pose located in the target gap and the second obstacle avoidance pose is the pose located in the upper area of the object to be grasped in the target container, therefore, by controlling the robotic arm to move from the grasping pose to the first obstacle avoidance pose and controlling the robotic arm to move from the first obstacle avoidance pose to the second obstacle avoidance pose, obstacle avoidance during object grasping can be achieved, and obstacle avoidance in all directions of the object is considered. Even if the gap between the objects in the container is small, when using a robotic arm to grasp the objects in the container, the robotic arm will not collide with other objects and will not cause damage to other objects. When using a robotic arm to grasp the objects in the container, the robotic arm will not collide with the container and will not cause damage to the container.

[0054] The above technical solutions of the embodiments of the present application are described below in combination with specific application scenarios.

[0055] In the embodiments of the present application, an object grasping method is proposed. This method can be applied to an object grasping system, and the object grasping system can include a depth camera, a robotic arm, and a control device of the robotic arm.

[0056] A depth camera is a camera capable of collecting depth images. The depth camera emits signals through infrared or laser and receives the reflection of the signals to determine the distance and depth of objects. The depth camera can obtain very accurate depth information, and the working principle of the depth camera is not limited here. Depth Images, also known as distance images, are images in which the distance (depth) values of each point in the scene are used as pixel values. Depth images directly reflect the geometric shape of the visible surface of the scene. Based on this, a depth image includes multiple three-dimensional points, that is, a depth image includes a three-dimensional point cloud, which can reflect the distance between each pixel point and the depth camera.

[0057] A robotic arm is a type of robot, an automated device with multiple joints that can simulate the movements of a human arm and perform various operations such as handling, assembly, and machining. Robotic arms can include three-axis robotic arms and six-axis robotic arms, etc. Taking the six-axis robotic arm as an example, a six-axis robotic arm refers to a robotic arm with six degrees of freedom, driven by six joints. The six joints can respectively achieve control of angles, positions, and speeds, and can move freely in three spatial directions such as X, Y, and Z, and can complete complex three-dimensional movements. For example, a six-axis robotic arm can be composed of joints, motors, sensors, etc. The motors are used to provide driving force for the six-axis robotic arm, and the sensors are used to provide feedback signals to achieve control and adjustment of the movement of the six-axis robotic arm.

[0058] The control device of the robotic arm is a device used to control the robotic arm, which can be a PC (Personal Computer), laptop, server, smart terminal, etc. The type of this control device is not limited in this embodiment. In this embodiment, the control device is used to control the robotic arm to grasp an object in a target container. For example, there are multiple objects placed in the target container. The target container is a container used to load various objects, and the target container can also be called a material box. The target container can be a rectangular or non-rectangular container, and the four sides of the target container are enclosed, and the objects cannot exceed the container range inside the target container. The object can also be called a workpiece or a material, and usually multiple layers of objects are loaded using the target container. On this basis, the control device can control the robotic arm to grasp the object in the target container, and the grasping obstacle avoidance is realized during the object grasping process.

[0059] In an application scenario where a robotic arm is required to automatically grasp an object, an object grasping method is proposed in an embodiment of the present application. Refer to Figure 2 As shown, it is a flowchart of the object grasping method, and the method includes:

[0060] Step 201, the control device acquires a depth image through a depth camera.

[0061] For example, when the target container moves to a specified position, the control device can send an image acquisition instruction to the depth camera. After receiving the image acquisition instruction, the depth camera can acquire a depth image and send the depth image to the control device, and the control device obtains the depth image.

[0062] For example, the specified position is a pre-configured position, and the specified position is within the field of view of the depth camera (such as the specified position is the central area of the field of view), and the target container needs to be moved to the specified position. On this basis, when the depth camera acquires a depth image, since the target container is within the field of view of the depth camera, the depth image includes the target container and multiple objects inside the target container.

[0063] For example, the depth image includes a plurality of three-dimensional points, that is, the depth image includes a three-dimensional point cloud. Since the depth image includes the target container and a plurality of objects, among the plurality of three-dimensional points, there are three-dimensional points of the target container (subsequently denoted as the first type of three-dimensional points) and three-dimensional points of each object (subsequently denoted as the second type of three-dimensional points).

[0064] Step 202: The control device obtains the pose of the target container in the camera coordinate system of the depth camera based on the depth image, and obtains the pose of the object (such as a plurality of objects) in the camera coordinate system based on the depth image.

[0065] Exemplarily, based on the depth image, the control device can adopt the following steps to obtain the pose of the target container in the camera coordinate system and obtain the poses of the plurality of objects in the camera coordinate system:

[0066] Step S11: If the depth image includes a plurality of three-dimensional points, then obtain the first type of three-dimensional points of the target container and the second type of three-dimensional points of the plurality of objects from the plurality of three-dimensional points of the depth image.

[0067] In a possible implementation manner, based on the complete region image of the depth image, a plurality of three-dimensional points of the complete region image can be obtained, and the first type of three-dimensional points of the target container and the second type of three-dimensional points of the plurality of objects can be obtained from the plurality of three-dimensional points. Alternatively, based on the ROI (Region Of Interest) sub-image of the depth image, a plurality of three-dimensional points of the ROI sub-image can be obtained, and the first type of three-dimensional points of the target container and the second type of three-dimensional points of the plurality of objects can be obtained from the plurality of three-dimensional points.

[0068] For example, for the ROI sub-image of the depth image, a detection ROI can be pre-drawn. The detection ROI is used to specify the recognition range of the depth image. The width of the detection ROI is greater than or equal to the width of the target container, and the height of the detection ROI is greater than or equal to the height of the target container. In order to ensure that the ROI sub-image can completely include the target container when the target container has a position offset, the width of the detection ROI is greater than (such as slightly greater than) the width of the target container, and the height of the detection ROI (such as slightly greater than) is greater than the height of the target container.

[0069] On this basis, after obtaining the depth image, the center position point of the target container in the depth image can be determined, the center position point of the detection ROI is aligned with the center position point of the target container, and the area where the detection ROI is located is intercepted from the depth image, and the area where the detection ROI is located is the ROI sub-image of the depth image. See Figure 3AAs shown, it is a schematic diagram of intercepting the area where the detection ROI is located from the depth image. In this way, the ROI sub-image can be intercepted from the depth image, and then multiple three-dimensional points within the ROI sub-image can be obtained. The first type of three-dimensional points of the target container and the second type of three-dimensional points of multiple objects can be obtained from the multiple three-dimensional points.

[0070] In a possible implementation manner, the three-dimensional points of the container model and the three-dimensional points of the object model can be obtained in advance. The three-dimensional points of the container model are the three-dimensional point model for the sample container. This three-dimensional point model includes the three-dimensional points of the sample container, and the sample container and the target container are the same container and have the same characteristics. The three-dimensional points of the object model are the three-dimensional point model for the sample object. This three-dimensional point model includes the three-dimensional points of the sample object, and the sample object and the object within the target container are the same object and have the same characteristics.

[0071] Regarding the acquisition process of the three-dimensional points of the container model, the sample container can be moved to a specified position. The depth camera captures the depth image corresponding to the sample container and sends the depth image corresponding to the sample container to the control device. The control device obtains the depth image corresponding to the sample container, and this depth image can only include the sample container. In this way, the three-dimensional points of the container model can be determined based on the depth image corresponding to the sample container. For example, the three-dimensional points within this depth image are used as the three-dimensional points of the container model. See Figure 3B As shown, it is a schematic diagram of the three-dimensional points of the container model. The three-dimensional points of the container model can also be referred to as the container model or the bin model.

[0072] Regarding the acquisition process of the three-dimensional points of the object model, the sample object can be moved to a specified position. The depth camera captures the depth image corresponding to the sample object and sends the depth image corresponding to the sample object to the control device. The control device obtains the depth image corresponding to the sample object, and this depth image can only include the sample object. In this way, the three-dimensional points of the object model can be determined based on the depth image corresponding to the sample object. For example, the three-dimensional points within this depth image are used as the three-dimensional points of the object model. See Figure 3C As shown, it is a schematic diagram of the three-dimensional points of the object model. The three-dimensional points of the object model can also be referred to as the object model or the workpiece model.

[0073] In a possible implementation manner, given the three-dimensional points of the container model, after obtaining the multiple three-dimensional points of the complete region image or the ROI sub-image of the depth image, the first type of three-dimensional points of the target container can be obtained from the multiple three-dimensional points based on the three-dimensional points of the container model, and the similarity between the first type of three-dimensional points and the three-dimensional points of the container model is greater than the first threshold, that is, the similarity between the features of the first type of three-dimensional points and the features of the three-dimensional points of the container model is greater than the first threshold. That is to say, it is necessary to screen out a part of the three-dimensional points that meet the similarity greater than the first threshold from the multiple three-dimensional points, and use this part of the three-dimensional points as the first type of three-dimensional points of the target container.

[0074] For example, the first type of 3D points is a point cloud composed of a large number of 3D points, and the 3D points of the container model are also a point cloud composed of a large number of 3D points. Here, the features can be point cloud arrangement features, point cloud position features, and relative position features of the 3D points within the point cloud. There is no restriction on the type of these features. Based on this, if the similarity between the features of some of the 3D points among multiple 3D points and the features of the 3D points of the container model is greater than a first threshold (which can be configured according to experience, such as 0.9, 0.95, etc.), then these 3D points are regarded as the first type of 3D points.

[0075] Given the 3D points of the object model, after obtaining the 3D points of the complete region image or the ROI sub-image of the depth image, the second type of 3D points of multiple objects can be obtained from the multiple 3D points based on the 3D points of the object model. For example, for each object among multiple objects, the similarity between the second type of 3D points of this object and the 3D points of the object model is greater than a second threshold, that is, the similarity between the features of the second type of 3D points and the features of the 3D points of the object model is greater than the second threshold. That is to say, it is necessary to screen out some 3D points from the multiple 3D points that meet the condition that the similarity is greater than the second threshold, and regard these 3D points as the second type of 3D points of an object. In addition, since there are multiple objects in the target container, the second type of 3D points of multiple objects can be obtained.

[0076] In summary, the first type of 3D points of the target container and the second type of 3D points of multiple objects can be obtained from the 3D points of the complete region image or the ROI sub-image of the depth image.

[0077] Step S12: Determine the pose of the target container in the camera coordinate system based on the first type of 3D points.

[0078] Exemplarily, the first type of 3D points of the target container is the 3D points of the target container in the camera coordinate system, that is, the first type of 3D points represents the 3D position in the camera coordinate system. Therefore, the position of the target container in the camera coordinate system can be determined based on the first type of 3D points. For example, the average value of all the first type of 3D points (such as the average value of x, the average value of y, and the average value of z) is used as the position of the target container in the camera coordinate system, or any one of all the first type of 3D points is used as the position of the target container in the camera coordinate system, or the central 3D point among all the first type of 3D points is used as the position of the target container in the camera coordinate system.

[0079] Based on the position of the target container in the camera coordinate system, the attitude of the target container in the camera coordinate system can be determined, that is, the attitude of the target container relative to the depth camera. For example, there is a functional relationship (or transformation relationship) between the attitude in the camera coordinate system and the position in the camera coordinate system, and this functional relationship can be pre-calibrated, and there is no restriction on this calibration process. The input of this functional relationship is the position in the camera coordinate system, and the output of this functional relationship is the attitude in the camera coordinate system. Based on this, the position of the target container in the camera coordinate system can be substituted into this functional relationship to obtain the attitude of the target container in the camera coordinate system.

[0080] In summary, the position and attitude of the target container in the camera coordinate system can be obtained, so as to obtain the pose of the target container in the camera coordinate system, that is, the pose includes the position and the attitude. For example, the position is the positions of three axes (such as the position of the x-axis, the position of the y-axis, and the position of the z-axis), and the attitude is the attitudes of three directions (such as the rotation angle around the x-axis, the rotation angle around the y-axis, and the rotation angle around the z-axis).

[0081] Step S13: For each object among multiple objects, based on the second type of three-dimensional points of the object, determine the pose of the object in the camera coordinate system, so as to obtain the pose of each object in the camera coordinate system.

[0082] Exemplarily, the second type of three-dimensional points of the object are the three-dimensional points of the object in the camera coordinate system, that is, the second type of three-dimensional points represent the three-dimensional position in the camera coordinate system. Therefore, the position of the object in the camera coordinate system can be determined based on the second type of three-dimensional points. Based on the position of the object in the camera coordinate system, the attitude of the object in the camera coordinate system can be determined, that is, the attitude of the object relative to the depth camera. For example, based on the functional relationship between the attitude in the camera coordinate system and the position in the camera coordinate system, the position of the object in the camera coordinate system can be substituted into this functional relationship to obtain the attitude of the object in the camera coordinate system.

[0083] In summary, the position and attitude of the object in the camera coordinate system can be obtained, so as to obtain the pose of the object in the camera coordinate system, that is, the pose can include the position and the attitude.

[0084] So far, step 202 is completed, and the pose of the target container in the camera coordinate system (such as the positioning pose of the target container) is obtained, and the poses of multiple objects in the camera coordinate system (such as the positioning poses of the objects) are obtained.

[0085] Step 203: Based on the conversion relationship between the camera coordinate system and the robot coordinate system, the control device converts the pose of the target container in the camera coordinate system into the pose of the target container in the robot coordinate system, and converts the poses of multiple objects in the camera coordinate system into the poses of each object in the robot coordinate system.

[0086] Exemplarily, the camera coordinate system is a coordinate system with a certain position (such as the central position) of the depth camera as the origin. When the depth camera captures a depth image, the three-dimensional points in the depth image are three-dimensional points in the camera coordinate system. In step 202, the pose of the target container in the camera coordinate system and the poses of multiple objects in the camera coordinate system have been obtained. The robot coordinate system is a coordinate system with a certain position (such as the center position of the base of the robotic arm) of the robotic arm as the origin. When the robotic arm grasps an object, it grasps the object based on the pose in the robot coordinate system, that is, the robotic arm senses the pose in the robot coordinate system and grasps the object in the robot coordinate system.

[0087] Before executing the object grasping process, the conversion relationship (which can also be called the transformation matrix) between the camera coordinate system and the robot coordinate system can be pre-calibrated. For example, Y = AX, where A represents the transformation matrix between the camera coordinate system and the robot coordinate system, X represents the pose in the camera coordinate system, and Y represents the pose in the robot coordinate system. For multiple calibration points in the actual physical space, the pose of the calibration point in the camera coordinate system and the pose of the calibration point in the robot coordinate system can be determined, that is, multiple pose pairs corresponding to the multiple calibration points are obtained, and each pose pair includes the pose in the camera coordinate system and the pose in the robot coordinate system.

[0088] Then, substitute the multiple pose pairs into the above formula, that is, substitute the pose in the camera coordinate system into X in the above formula, and substitute the pose in the robot coordinate system into Y in the above formula, and the transformation matrix A can be obtained. By pre-calibrating the transformation matrix A, in this way, the conversion relationship between the camera coordinate system and the robot coordinate system can be pre-stored.

[0089] Exemplarily, based on the conversion relationship between the camera coordinate system and the robot coordinate system, such as the transformation matrix A, after obtaining the pose of the target container in the camera coordinate system, the pose of the target container in the camera coordinate system can be substituted into X in the above formula, so as to obtain the pose of the target container in the robot coordinate system. In this way, the container positioning pose in the camera coordinate system can be converted to the robot coordinate system.

[0090] Similarly, after obtaining the pose of the object in the camera coordinate system, the pose of the object in the camera coordinate system can be substituted into X in the above formula, so as to obtain the pose of the object in the robot coordinate system. In this way, the object positioning pose in the camera coordinate system can be converted to the robot coordinate system.

[0091] In step 204, for the convenience of distinction, the pose of the target container in the robot coordinate system is called the reference pose, and the control device establishes a container coordinate system with this reference pose as the origin. For each object among the multiple objects, the pose of the object in the robot coordinate system is converted into the pose of the object in the container coordinate system.

[0092] Exemplarily, after obtaining the reference pose of the target container in the robot coordinate system, this reference pose (also known as the bin pose) can be stored. On this basis, a container coordinate system (also known as the bin coordinate system) is established with this reference pose as the origin, and the container coordinate system can serve as the coordinate system of the target container.

[0093] On this basis, the pose of the object in the robot coordinate system can be converted into the pose of the object in the container coordinate system. For example, since the origin of the container coordinate system is the reference pose in the robot coordinate system, therefore, after knowing the pose of the object in the robot coordinate system and the reference pose, the pose relationship between the pose of the object in the robot coordinate system and the reference pose in the robot coordinate system (i.e., the relationship between two poses in the robot coordinate system) can be obtained, and then based on this pose relationship, the pose of the object in the robot coordinate system is converted into the pose of the object in the container coordinate system.

[0094] For example, assume that the reference pose in the robot coordinate system is (x1, y1, z1, Rx1, Ry1, Rz1), where (x1, y1, z1) represents the position in the robot coordinate system, and (Rx1, Ry1, Rz1) represents the orientation in the robot coordinate system. When establishing the container coordinate system with this reference pose as the origin, the reference pose (x1, y1, z1, Rx1, Ry1, Rz1) in the robot coordinate system is equivalent to the origin (0, 0, 0, 0, 0, 0) in the container coordinate system. Assume that the pose of the object in the robot coordinate system is (x2, y2, z2, Rx2, Ry2, Rz2), then the pose relationship between the pose of the object in the robot coordinate system and the reference pose in the robot coordinate system is (x2 - x1, y2 - y1, z2 - z1, Rx2 - Rx1, Ry2 - Ry1, Rz2 - Rz1). Obviously, the pose relationship (x2 - x1, y2 - y1, z2 - z1, Rx2 - Rx1, Ry2 - Ry1, Rz2 - Rz1) can be used as the pose of the object in the container coordinate system, that is, converting the pose in the robot coordinate system into the pose in the container coordinate system.

[0095] In summary, for each object, the pose of the object in the robot coordinate system can be converted into the pose of the object in the container coordinate system, so as to obtain the pose of each object in the container coordinate system.

[0096] Step 205: The control device fits the central pose of the object area in the container based on the poses of multiple objects in the container coordinate system, and determines the container center pose of the target container. For example, the central pose of the object area includes a first X-direction coordinate value and a first Y-direction coordinate value. The container center pose includes a second X-direction coordinate value and a second Y-direction coordinate value. Among them, the first X-direction coordinate value can be determined based on the average value of the X-direction coordinate values in the poses of multiple objects in the container coordinate system, and the first Y-direction coordinate value can be determined based on the average value of the Y-direction coordinate values in the poses of multiple objects in the container coordinate system.

[0097] For example, the pose in the container coordinate system can include position and orientation, and the position includes an X-direction coordinate value, a Y-direction coordinate value, and a Z-direction coordinate value. Based on this, the average value of the X-direction coordinate values in the poses of all objects in the container coordinate system is determined, and the first X-direction coordinate value is determined based on this average value, such as taking this average value as the first X-direction coordinate value. The average value of the Y-direction coordinate values in the poses of all objects in the container coordinate system is determined, and the first Y-direction coordinate value is determined based on this average value, such as taking this average value as the first Y-direction coordinate value. In this way, the first X-direction coordinate value and the first Y-direction coordinate value can be used as the central pose of the object area in the container, which can be understood as the central pose of all objects.

[0098] For example, the central pose of the target container can be used as the container center pose, that is, the second X-direction coordinate value is the horizontal central pose of the target container, and the second Y-direction coordinate value is the vertical central pose of the target container.

[0099] Step 206: The control device determines the target gap based on the relative position between the central pose of the object area and the container center pose. The target gap represents the opposite direction in which the object is biased in the target container, and this target gap can also be referred to as the target gap direction. For example, if the central pose of the object area is on the left side of the container center pose, the target gap can be determined as the right side direction, or, if the central pose of the object area is on the right side of the container center pose, the target gap can be determined as the left side direction, or, if the central pose of the object area is on the upper side of the container center pose, the target gap can be determined as the lower side direction, or, if the central pose of the object area is on the lower side of the container center pose, the target gap can be determined as the upper side direction.

[0100] Exemplarily, since the central pose of the object region includes a first X-direction coordinate value and a first Y-direction coordinate value, and the central pose of the container includes a second X-direction coordinate value and a second Y-direction coordinate value, therefore, the difference between the first X-direction coordinate value and the second X-direction coordinate value can reflect the deviation between the overall object and the central position of the target container (i.e., the bias of all objects in the target container). For example, when the first X-direction coordinate value is on the right side of the second X-direction coordinate value, it means that the overall object is biased to the right compared to the central position of the target container; when the first X-direction coordinate value is on the left side of the second X-direction coordinate value, it means that the overall object is biased to the left compared to the central position of the target container; when the first Y-direction coordinate value is above the second Y-direction coordinate value, it means that the overall object is biased upward compared to the central position of the target container; when the first Y-direction coordinate value is below the second Y-direction coordinate value, it means that the overall object is biased downward compared to the central position of the target container.

[0101] Based on the above principle, if the difference between the first X-direction coordinate value and the second X-direction coordinate value is greater than the difference between the first Y-direction coordinate value and the second Y-direction coordinate value (i.e., the overall deviation in the X direction is greater), and the first X-direction coordinate value is on the right side of the second X-direction coordinate value, then the overall object is biased to the right, so the target gap is determined to be the left side direction of the target container. Or, if the difference between the first X-direction coordinate value and the second X-direction coordinate value is greater than the difference between the first Y-direction coordinate value and the second Y-direction coordinate value, and the first X-direction coordinate value is on the left side of the second X-direction coordinate value, then the overall object is biased to the left, so the target gap is determined to be the right side direction of the target container. In addition, if the difference between the first X-direction coordinate value and the second X-direction coordinate value is less than the difference between the first Y-direction coordinate value and the second Y-direction coordinate value (i.e., the overall deviation in the Y direction is greater), and the first Y-direction coordinate value is above the second Y-direction coordinate value, then the overall object is biased upward, so the target gap is determined to be the lower side direction of the target container. Or, if the difference between the first X-direction coordinate value and the second X-direction coordinate value is less than the difference between the first Y-direction coordinate value and the second Y-direction coordinate value, and the first Y-direction coordinate value is below the second Y-direction coordinate value, then the overall object is biased downward, so the target gap is determined to be the upper side direction of the target container.

[0102] For example, referring to Figure 4A as shown, it is a schematic diagram of multiple objects in the target container, that is, a schematic diagram of multiple objects in the target container. The central pose of the object region can be determined based on the pose of each object in the container coordinate system. Referring to Figure 4B as shown, it is a schematic diagram of the central pose of the object region, and the red dot represents the central pose of the object region. Referring to Figure 4CAs shown, it is a comparison schematic diagram of the central pose of the object area and the central position of the target container (i.e., the central pose of the container), and the green dot represents the central pose of the container.

[0103] Step 207, the control device determines the object located at the edge of the target gap as the object to be grasped.

[0104] Exemplarily, the object located at the edge of the target gap can be determined first. If there is 1 object located at the edge of the target gap, then this object is used as the object to be grasped. If there are multiple objects located at the edge of the target gap, then the grasping order of the multiple objects is determined, and the object to be grasped is determined from the multiple objects based on the grasping order.

[0105] See Figure 4D As shown, it is a schematic diagram of the object located at the edge of the target gap. If the target gap is in the left direction of the target container, then the object at the left edge is determined as the object at the edge of the target gap, that is, all the objects in the first column are used as the objects at the edge of the target gap. If the target gap is in the right direction of the target container, then the object at the right edge is determined as the object at the edge of the target gap, that is, all the objects in the last column are used as the objects at the edge of the target gap. If the target gap is in the upper direction of the target container, then the object at the upper edge is determined as the object at the edge of the target gap, that is, all the objects in the first row are used as the objects at the edge of the target gap. If the target gap is in the lower direction of the target container, then the object at the lower edge is determined as the object at the edge of the target gap, that is, all the objects in the last row are used as the objects at the edge of the target gap.

[0106] After determining multiple objects at the edge of the target gap, the grasping order of these objects can be determined. To determine the grasping order, all the objects can be classified into intermediate objects and edge-attached objects. The number of edge-attached objects is 2, and the edge-attached objects are the objects close to the target container. For example, for all the objects in the first column or the last column, then one object in the first row and one object in the last row are used as edge-attached objects. Or, for all the objects in the first row or the last row, then one object in the first column and one object in the last column are used as edge-attached objects.

[0107] The number of intermediate objects can be at least 1, or can be 0. The intermediate objects are the remaining objects other than the edge-attached objects. When determining the grasping order of all objects, the intermediate objects can be located in front of the edge-attached objects. When there are multiple intermediate objects, the grasping order of the multiple intermediate objects can be configured arbitrarily. For example, the multiple intermediate objects can be in the grasping order from bottom to top (from right to left), or the multiple intermediate objects can be in the grasping order from top to bottom (from left to right). In addition, for two edge-attached objects, the grasping order of the two edge-attached objects can be configured arbitrarily. For example, the two edge-attached objects can be in the grasping order from bottom to top (from right to left), or in the grasping order from top to bottom (from left to right).

[0108] See Figure 4E As shown, it is a schematic diagram of the grasping order. The objects in the first row and the last row can be used as edge-attached objects, and the remaining two objects can be used as intermediate objects. The two intermediate objects need to be located in front of the two edge-attached objects. For the two intermediate objects, the grasping order can be from bottom to top or from top to bottom. In Figure 4E taking the grasping order from bottom to top as an example. For the two edge-attached objects, the grasping order can be from bottom to top or from top to bottom. In Figure 4E taking the grasping order from bottom to top as an example. To sum up, the grasping order of all objects can be obtained, that is, in the grasping order of 1, 2, 3, 4.

[0109] After obtaining the grasping order of all objects, each object is traversed in turn based on this grasping order as the object to be grasped. For example, first traverse the object with the grasping order of 1 as the object to be grasped, then traverse the object with the grasping order of 2 as the object to be grasped, and so on.

[0110] Step 208: The control device acquires the first post-movement pose of the object to be grasped in the container coordinate system and the second post-movement pose of the object to be grasped in the container coordinate system.

[0111] For example, the following steps can be used to acquire the first post-movement pose and the second post-movement pose.

[0112] Step S21: Based on the pose of the object to be grasped in the container coordinate system, the size of the target container, and the size of the object to be grasped, determine the distance between the object to be grasped and the edge of the target container.

[0113] For example, taking the case where the object to be grasped is located in the first column (or the last column), that is, the object to be grasped needs to move towards the left edge of the target container. Then, the pose of the object to be grasped in the container coordinate system (such as the coordinate value in the X direction) can reflect the distance D1 between the center position of the object to be grasped and the center position of the target container. The size of the target container (such as half of the width) can reflect the distance D2 between the center position of the target container and the left edge of the target container. In this way, the difference between the distance D2 and the distance D1 represents the distance D3 between the center position of the object to be grasped and the left edge of the target container.

[0114] The size of the object to be grasped can reflect the distance D4 between the center position of the object to be grasped and the edge position of the object to be grasped. For example, when the object to be grasped is rectangular, half of the width represents the distance D4 between the center position and the edge position. When the object to be grasped is circular, the radius represents the distance D4 between the center position and the edge position. There is no limitation on this, as long as the distance D4 can be obtained.

[0115] Obviously, the difference between the distance D3 and the distance D4 represents the distance D5 between the edge position of the object to be grasped and the left edge of the target container, that is, the distance D5 between the object to be grasped and the edge of the target container.

[0116] Step S22: Determine whether this distance is not greater than the configured blank distance threshold.

[0117] Exemplarily, a blank distance threshold can be pre-configured. This blank distance threshold indicates how large the distance between the object to be grasped and the edge of the target container is when an offset in the blank direction can be performed. There is no limitation on this blank distance threshold. Based on this, it can be determined whether the distance D5 is not greater than this blank distance threshold.

[0118] If so, step S23 can be executed. If not, step S24 can be executed.

[0119] Step S23: Output an alarm message, which indicates that the object to be grasped cannot be grasped.

[0120] For example, as shown in Figure 5A the figure is a schematic diagram of the object to be grasped against the edge. When traversing the object to be grasped in the third row, the distance between the object to be grasped and the edge of the target container is not greater than the blank distance threshold. For example, due to the jitter during transportation causing the object to shift, the distance becomes not greater than the blank distance threshold, that is, the object to be grasped is sticking to the blank direction, resulting in an inability to move horizontally and an abnormal situation where the object cannot be grasped. Based on this, an alarm message can be output, so that manual intervention is required for the grasping operation of the object to be grasped.

[0121] In a possible implementation manner, in step 205, after obtaining the fitting coordinate values in the X direction and the fitting coordinate values in the Y direction, the fitting coordinate values in the X direction and the fitting coordinate values in the Y direction reflect the deviation between the overall object and the central position of the target container. If the fitting coordinate value in the X direction is less than the threshold value and the fitting coordinate value in the Y direction is less than the threshold value, it means that the difference between the fitting pose and the central position of the target container is too small, and there is no blank direction in any of the four directions of the target container. Therefore, the target gap of the target container cannot be determined. Based on this, an alarm message can be directly output, so that manual intervention is required for the grasping operation of the object to be grasped. Refer to Figure 5B As shown, it is a schematic diagram of the situation where there is no blank direction in any of the four directions of the target container.

[0122] Step S24: Based on the pose of the object to be grasped in the container coordinate system, the first moving direction, and the configured moving distance, determine the first post-movement pose of the object to be grasped in the container coordinate system.

[0123] Exemplarily, the first moving direction can be a lateral moving direction or an oblique moving direction towards the target gap. For example, refer to Figure 5C As shown, it is a schematic diagram of the object moving direction. When the object to be grasped is the middle object to be grasped, the first moving direction is the lateral moving direction towards the target gap. By laterally moving the middle object to be grasped, collisions with surrounding objects can be avoided. When the object to be grasped is the edge object to be grasped, the first moving direction is the oblique moving direction towards the target gap. By obliquely moving the edge object to be grasped, collisions with the edge of the target container can be avoided. The first moving direction can also include the height direction, and the distance value in the height direction can be configured arbitrarily. By moving the object to be grasped in the height direction, the object to be grasped can be prevented from colliding with the objects on the lower layer.

[0124] Exemplarily, the moving distance can be configured according to experience, such as 5 millimeters, 6 millimeters, etc., and there is no limit to this moving distance. On this basis, record the pose of the object to be grasped in the container coordinate system as pose A, and record the first post-movement pose of the object to be grasped in the container coordinate system as pose B. Starting from pose A, move the object to be grasped. The object moving direction is the first moving direction, and the object moving distance is the configured moving distance. In this way, take the post-movement pose of the object to be grasped as pose B. In summary, the first post-movement pose can be determined based on pose A, the first moving direction, and the configured moving distance.

[0125] Step S25: Based on the first post-movement pose, the size of the target container, and the size of the object to be grasped, determine whether a collision occurs between the object to be grasped and the edge of the target container.

[0126] If so, step S26 can be executed; if not, step S27 can be executed.

[0127] For example, the first post-movement pose (such as the X-direction coordinate value) can reflect the distance W1 between the center position after movement and the center position of the target container. The size of the target container (such as half of the width) can reflect the distance W2 between the center position of the target container and the left edge of the target container. The difference between the distance W2 and the distance W1 represents the distance W3 between the center position after movement and the left edge of the target container. The size of the object to be grasped can reflect the distance W4 between the center position of the object to be grasped and the edge position of the object to be grasped. Obviously, the difference between the distance W3 and the distance W4 represents the distance W5 between the edge position of the object to be grasped and the left edge of the target container. If the distance W5 is 0 or negative, it means that the object to be grasped collides with the edge of the target container (such as the left edge). If the distance W5 is greater than 0, it means that the object to be grasped does not collide with the edge of the target container (such as the left edge).

[0128] Step S26: Output an alarm message, which indicates that the object to be grasped cannot be grasped.

[0129] Step S27: Based on the first post-movement pose, the configured height movement distance value, and the configured reference offset position, determine the second post-movement pose of the object to be grasped in the container coordinate system.

[0130] Exemplarily, the height movement distance value can be configured according to experience. The height movement distance value can be greater than or equal to the height of a single object. By configuring the height movement distance value, the object to be grasped can move upward, and the upward movement distance is the height movement distance value, that is, the upward movement distance is greater than or equal to the height of a single object, so as to avoid colliding with the objects on the next layer.

[0131] Exemplarily, the reference offset position can be configured according to experience. The reference offset position includes the lateral offset distance and the longitudinal offset distance between the second post-movement pose and the origin of the container coordinate system (i.e., the center position of the target container). For example, if both the lateral offset distance and the longitudinal offset distance are 0, it means that the second post-movement pose is the center position of the target container. On this basis, record the first post-movement pose as pose B, and record the second post-movement pose of the object to be grasped in the container coordinate system as pose C. Starting from pose B, move the object to be grasped upward by the height movement distance value, and then move to the reference offset position (such as the center position of the target container), and use the post-movement pose of the object to be grasped as pose C. In summary, the second post-movement pose is determined based on pose B, the height movement distance value, and the reference offset position.

[0132] At this point, step 208 is completed, and the position and posture of the object to be grasped in the container coordinate system, the position and posture of the object to be grasped after the first movement in the container coordinate system, and the position and posture of the object to be grasped after the second movement in the container coordinate system are obtained.

[0133] Step 209: The control device converts the posture of the object to be grasped in the container coordinate system into the grasping posture of the object to be grasped in the robot coordinate system, converts the first posture after movement into the first obstacle avoidance posture of the object to be grasped in the robot coordinate system, and converts the second posture after movement into the second obstacle avoidance posture of the object to be grasped in the robot coordinate system. For example, the grasping posture may be the actual posture of the object to be grasped, the first obstacle avoidance posture may be the posture of the blank area of the target container, and the second obstacle avoidance posture may be the posture of the upper area of the object to be grasped in the target container.

[0134] Exemplarily, the posture of the object to be grasped in the container coordinate system, the posture after the first movement, and the posture after the second movement are all postures in the container coordinate system. The posture in the container coordinate system needs to be converted into the posture in the robot coordinate system so as to control the robot to grasp the object to be grasped in the robot coordinate system.

[0135] Referring to step 204, the container coordinate system takes the reference posture as its origin, and the reference posture is the posture of the target container in the robot coordinate system. Therefore, the origin of the container coordinate system is the reference posture in the robot coordinate system. In this way, the conversion relationship between the robot coordinate system and the container coordinate system can be obtained based on the reference posture.

[0136] On this basis, based on the conversion relationship between the robot coordinate system and the container coordinate system, the position of the object to be grasped in the container coordinate system can be converted into the grasping position of the object to be grasped in the robot coordinate system. The grasping position can be the actual position of the object to be grasped, that is, the current position. For example, the grasping position is the point (coordinate) that the robot arm can grasp under visual positioning.

[0137] Based on the conversion relationship between the robot coordinate system and the container coordinate system, the first post-movement posture can be converted into the first obstacle avoidance posture. Obviously, when the first post-movement posture is determined based on the first moving direction and the moving distance, the first post-movement posture is in the blank area of the target container and can avoid colliding with the edge of the target container and other objects. Therefore, the first obstacle avoidance posture in the robot coordinate system is also located in the blank area of the target container and can avoid colliding with the edge of the target container and other objects.

[0138] Based on the conversion relationship between the robot coordinate system and the container coordinate system, the second post-movement pose can be converted into the second obstacle-avoidance pose. Obviously, when determining the second post-movement pose based on the height movement distance value and the reference offset position, the second post-movement pose is located in the upper area of the object to be grasped and can avoid colliding with other objects. Therefore, the second obstacle-avoidance pose in the robot coordinate system is also located in the upper area of the object to be grasped in the target container and can avoid colliding with other objects.

[0139] Step 210: The control device controls the robotic arm to move to the grasping pose and grasp the object to be grasped based on the grasping pose, the first obstacle-avoidance pose, and the second obstacle-avoidance pose, controls the robotic arm to move from the grasping pose to the first obstacle-avoidance pose, and controls the robotic arm to move from the first obstacle-avoidance pose to the second obstacle-avoidance pose.

[0140] For example, the control device can send a control instruction to the robotic arm, and the control instruction includes the grasping pose, the first obstacle-avoidance pose, and the second obstacle-avoidance pose. After receiving the control instruction, the robotic arm moves to the grasping pose and grasps the object to be grasped, moves from the grasping pose to the first obstacle-avoidance pose, and moves from the first obstacle-avoidance pose to the second obstacle-avoidance pose based on the grasping pose, the first obstacle-avoidance pose, and the second obstacle-avoidance pose. After moving to the second obstacle-avoidance pose, the object to be grasped can be placed at a specified position, and the present embodiment does not limit the processing procedure after the second obstacle-avoidance pose.

[0141] In a possible implementation manner, multiple objects can be used as the objects to be grasped. In this way, the robotic arm can be controlled to grasp each object to be grasped in sequence. After completing the grasping operations of all the objects to be grasped, return to step 205 and repeat the above steps until the grasping operations of all the objects in the topmost layer are completed. Then, for all the objects in the next layer (which is the topmost layer at this time), return to step 205 and repeat the above steps, and so on, to complete the grasping operations of all the objects in each layer in sequence.

[0142] As can be seen from the above technical solutions, in the embodiments of the present application, by controlling the robotic arm to move from the grasping pose to the first obstacle-avoidance pose and controlling the robotic arm to move from the first obstacle-avoidance pose to the second obstacle-avoidance pose, obstacle avoidance during object grasping can be achieved, and obstacle avoidance in all directions of the object is considered. Even if the intervals between the objects in the container are small, when using the robotic arm to grasp the objects in the container, the robotic arm will not collide with other objects and will not cause damage to other objects. When using the robotic arm to grasp the objects in the container, the robotic arm will not collide with the container and will not cause damage to the container. The above method can be adapted to any container and any object, is applicable to the application scenario of grasping objects in a deep box, and considers obstacle avoidance in all directions of the object.

[0143] Based on the same application concept as the above method, this embodiment proposes an object grasping device. A plurality of objects are placed in the target container. Refer to Figure 6 As shown in the figure, which is a schematic structural diagram of the device. The device includes:

[0144] A determination module 61, configured to determine a target gap of the target container based on the pose of the object, and determine an object located at the edge of the target gap as the object to be grasped; an acquisition module 62, configured to acquire the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose of the object to be grasped; wherein, the grasping pose is the actual pose of the object to be grasped, the first obstacle avoidance pose is the pose located in the target gap, and the second obstacle avoidance pose is the pose located in the upper area of the object to be grasped in the target container; a control module 63, configured to control the robotic arm to move to the grasping pose and grasp the object to be grasped based on the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose, control the robotic arm to move from the grasping pose to the first obstacle avoidance pose, and control the robotic arm to move from the first obstacle avoidance pose to the second obstacle avoidance pose.

[0145] Exemplarily, when the determination module 61 determines the target gap of the target container based on the pose of the object, it is specifically configured to: fit the central pose of the object area in the container based on the pose of the object; determine the container central pose of the target container; determine the target gap based on the relative position between the central pose of the object area and the container central pose.

[0146] Exemplarily, the central pose of the object area includes a first X-direction coordinate value and a first Y-direction coordinate value, and the container central pose includes a second X-direction coordinate value and a second Y-direction coordinate value; when the determination module 61 determines the target gap based on the relative position between the central pose of the object area and the container central pose, it is specifically configured to: if the difference between the first X-direction coordinate value and the second X-direction coordinate value is greater than the difference between the first Y-direction coordinate value and the second Y-direction coordinate value, then determine the target gap as the left direction or the right direction based on the first X-direction coordinate value and the second X-direction coordinate value; if the difference between the first X-direction coordinate value and the second X-direction coordinate value is less than the difference between the first Y-direction coordinate value and the second Y-direction coordinate value, then determine the target gap as the lower direction or the upper direction based on the first Y-direction coordinate value and the second Y-direction coordinate value.

[0147] Exemplarily, when the determining module 61 determines the object located at the edge of the target gap as the object to be grasped, it specifically is used for: if there are multiple objects located at the edge of the target gap, determining the grasping order of the multiple objects, and determining the object to be grasped from the multiple objects based on the grasping order; wherein, there is at least one intermediate object and two edge objects among the multiple objects, the edge object is the object close to the target container, and the intermediate object is the remaining object except the edge object; in the grasping order, the intermediate object is in front of the edge object.

[0148] Exemplarily, when the obtaining module 62 obtains the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose of the object to be grasped, it specifically is used for: based on the pose of the object to be grasped, the first moving direction, and the configured moving distance, determining the first post-movement pose of the object to be grasped; wherein, the first moving direction is the lateral moving direction or the oblique moving direction towards the target gap; based on the first post-movement pose, the configured height moving distance value, and the configured reference offset position, determining the second post-movement pose of the object to be grasped; wherein, the height moving distance value is greater than or equal to the height of a single object, and the reference offset position includes the lateral offset distance and the longitudinal offset distance between the second post-movement pose and the origin position; converting the pose of the object to be grasped into the grasping pose, converting the first post-movement pose into the first obstacle avoidance pose, and converting the second post-movement pose into the second obstacle avoidance pose.

[0149] Exemplarily, the obtaining module 62 is further used for, after determining the object located at the edge of the target gap as the object to be grasped, determining the distance between the object to be grasped and the edge of the target container; if the distance is not greater than the configured blank distance threshold, outputting an alarm message; and / or, determining whether a collision occurs between the object to be grasped and the edge of the target container when the object to be grasped moves to the first obstacle avoidance pose, and if so, outputting an alarm message; the alarm message indicates that the object to be grasped cannot be grasped.

[0150] Exemplarily, the pose of the object is the pose of the object in the container coordinate system of the target container. The device further includes: a processing module, configured to collect a depth image through a depth camera, and obtain the pose of the target container in the camera coordinate system of the depth camera and the pose of the object in the camera coordinate system based on the depth image; based on the conversion relationship between the camera coordinate system and the robot coordinate system, convert the pose of the target container in the camera coordinate system into a reference pose of the target container in the robot coordinate system, and convert the pose of the object in the camera coordinate system into the pose of the object in the robot coordinate system; establish the container coordinate system with the reference pose as the origin, and convert the pose of the object in the robot coordinate system into the pose of the object in the container coordinate system.

[0151] Exemplarily, if the depth image includes a plurality of three-dimensional points, when the processing module obtains the pose of the target container in the camera coordinate system of the depth camera and the pose of the object in the camera coordinate system based on the depth image, it is specifically configured to: obtain a first type of three-dimensional points of the target container and a second type of three-dimensional points of the object from the plurality of three-dimensional points; determine the pose of the target container in the camera coordinate system based on the first type of three-dimensional points, and determine the pose of the object in the camera coordinate system based on the second type of three-dimensional points of the object; wherein, the similarity between the first type of three-dimensional points and the three-dimensional points of the container model is greater than a first threshold, and the similarity between the second type of three-dimensional points of the object and the three-dimensional points of the object model is greater than a second threshold; wherein, the three-dimensional points of the container model are determined based on the depth image corresponding to the sample container, the depth image only includes the sample container, and the three-dimensional points in the depth image are the three-dimensional points of the container model; wherein, the three-dimensional points of the object model are determined based on the depth image corresponding to the sample object, the depth image only includes the sample object, and the three-dimensional points in the depth image are the three-dimensional points of the object model.

[0152] Based on the same application concept as the above method, this embodiment proposes an object grasping system. A plurality of objects are placed in a target container. The object grasping system includes a depth camera, a robotic arm, and a control device for the robotic arm, where: The depth camera is configured to collect a depth image and send the depth image to the control device; The control device is configured to obtain the pose of the object based on the depth image, determine the target gap of the target container based on the pose of the object, and determine the object located at the edge of the target gap as the object to be grasped; obtain the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose of the object to be grasped; where the grasping pose is the actual pose of the object to be grasped, the first obstacle avoidance pose is the pose located in the target gap, and the second obstacle avoidance pose is the pose located in the upper area of the object to be grasped in the target container; and send a control instruction to the robotic arm, where the control instruction includes the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose; The robotic arm is configured to, after receiving the control instruction, control the robotic arm to move to the grasping pose and grasp the object to be grasped based on the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose, control the robotic arm to move from the grasping pose to the first obstacle avoidance pose, and control the robotic arm to move from the first obstacle avoidance pose to the second obstacle avoidance pose.

[0153] Based on the same application concept as the above method, an electronic device (such as the control device of a robotic arm) is proposed in an embodiment of the present application. Refer to Figure 7 As shown, it includes: a processor 71 and a machine-readable storage medium 72. The machine-readable storage medium 72 stores machine-executable instructions that can be executed by the processor 71; the processor 71 is configured to execute the machine-executable instructions to implement the object grasping method disclosed in the above example of the present application.

[0154] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium. A number of computer instructions are stored on the machine-readable storage medium. When the computer instructions are executed by a processor, the object grasping method disclosed in the above example of the present application can be implemented.

[0155] Wherein, the above machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium can be: RAM (Radom Access Memory, random access memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or a combination thereof.

[0156] Based on the same application concept as the above method, an embodiment of the present application further provides a computer program product, and the computer program product may include a computer program. Wherein, when the computer program is executed by a processor, it can implement the object grasping method disclosed in the above examples of the present application.

[0157] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0158] The above are only the embodiments of the present application and are not used to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A method for grasping an object, characterized in that, A plurality of objects are placed in the target container, including: Determine the target void of the target container based on the poses of the objects, and determine the objects located at the edge of the target void as the objects to be grasped; Obtain the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose of the object to be grasped; wherein, the grasping pose is the actual pose of the object to be grasped, the first obstacle avoidance pose is the pose located in the target void, and the second obstacle avoidance pose is the pose located in the upper area of the object to be grasped in the target container; Based on the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose, control the robotic arm to move to the grasping pose and grasp the object to be grasped, control the robotic arm to move from the grasping pose to the first obstacle avoidance pose, and control the robotic arm to move from the first obstacle avoidance pose to the second obstacle avoidance pose.

2. The method according to claim 1, wherein The determining the target void of the target container based on the poses of the objects includes: Fit the central pose of the object area in the container based on the poses of the objects; Determine the central pose of the container of the target container; Determine the target void based on the relative positions of the central pose of the object area and the central pose of the container.

3. The method according to claim 2, wherein The central pose of the object area includes a first X-direction coordinate value and a first Y-direction coordinate value, and the central pose of the container includes a second X-direction coordinate value and a second Y-direction coordinate value; the determining the target void based on the relative positions of the central pose of the object area and the central pose of the container includes: If the difference between the first X-direction coordinate value and the second X-direction coordinate value is greater than the difference between the first Y-direction coordinate value and the second Y-direction coordinate value, determine the target void as the left direction or the right direction based on the first X-direction coordinate value and the second X-direction coordinate value; If the difference between the first X-direction coordinate value and the second X-direction coordinate value is less than the difference between the first Y-direction coordinate value and the second Y-direction coordinate value, determine the target void as the lower direction or the upper direction based on the first Y-direction coordinate value and the second Y-direction coordinate value.

4. The method according to claim 1, wherein The determining the objects located at the edge of the target void as the objects to be grasped includes: If there are multiple objects located at the edge of the target void, determine the grasping order of the multiple objects, and determine the object to be grasped from the multiple objects based on the grasping order; Wherein, there is at least one intermediate object and two edge objects among the multiple objects, the edge objects are the objects close to the target container, and the intermediate object is the remaining object except the edge objects; in the grasping order, the intermediate object is in front of the edge objects.

5. The method according to claim 1, characterized in that The obtaining the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose of the object to be grasped includes: Determine the first post-movement pose of the object to be grasped based on the pose of the object to be grasped, the first movement direction, and the configured movement distance; wherein, the first movement direction is the lateral movement direction or the oblique movement direction towards the target void; Determine the second post-movement pose of the object to be grasped based on the first post-movement pose, the configured height movement distance value, and the configured reference offset position; wherein, the height movement distance value is greater than or equal to the height of a single object, and the reference offset position includes the lateral offset distance and the longitudinal offset distance between the second post-movement pose and the origin position; Convert the pose of the object to be grasped into the grasping pose, convert the first post-movement pose into the first obstacle avoidance pose, and convert the second post-movement pose into the second obstacle avoidance pose.

6. The method according to claim 1, wherein After determining the object located at the edge of the target gap as the object to be grasped, the method further includes: Determine the distance between the object to be grasped and the edge of the target container; if the distance is not greater than the configured blank distance threshold, output an alarm message; and / or, Determine whether there is a collision between the object to be grasped and the edge of the target container when the object to be grasped moves to the first obstacle avoidance pose, and if so, output an alarm message; Wherein, the alarm message indicates that the object to be grasped cannot be grasped.

7. The method according to claim 1, wherein, The pose of the object is the pose of the object in the container coordinate system of the target container. Before determining the target gap of the target container based on the pose of the object, the method further includes: Collect a depth image through a depth camera, and obtain the pose of the target container in the camera coordinate system of the depth camera and the pose of the object in the camera coordinate system based on the depth image; Based on the conversion relationship between the camera coordinate system and the robot coordinate system, convert the pose of the target container in the camera coordinate system into the reference pose of the target container in the robot coordinate system, and convert the pose of the object in the camera coordinate system into the pose of the object in the robot coordinate system; Establish the container coordinate system with the reference pose as the origin, and convert the pose of the object in the robot coordinate system into the pose of the object in the container coordinate system.

8. The method according to claim 7, characterized in that, If the depth image includes a plurality of three-dimensional points, the obtaining the pose of the target container in the camera coordinate system of the depth camera and the pose of the object in the camera coordinate system based on the depth image includes: Obtain the first type of three-dimensional points of the target container and the second type of three-dimensional points of the object from the plurality of three-dimensional points; determine the pose of the target container in the camera coordinate system based on the first type of three-dimensional points, and determine the pose of the object in the camera coordinate system based on the second type of three-dimensional points of the object; Wherein, the similarity between the first type of three-dimensional points and the three-dimensional points of the container model is greater than the first threshold, and the similarity between the second type of three-dimensional points of the object and the three-dimensional points of the object model is greater than the second threshold; Wherein, the three-dimensional points of the container model are determined based on the depth image corresponding to the sample container, the depth image only includes the sample container, and the three-dimensional points in the depth image are the three-dimensional points of the container model; Wherein, the three-dimensional points of the object model are determined based on the depth image corresponding to the sample object, the depth image only includes the sample object, and the three-dimensional points in the depth image are the three-dimensional points of the object model.

9. An object grasping device, characterized in that, A plurality of objects are placed in the target container, including: A determination module, configured to determine a target void of the target container based on the poses of the objects, and determine the objects located at the edge of the target void as objects to be grasped; An acquisition module, configured to acquire a grasping pose, a first obstacle avoidance pose, and a second obstacle avoidance pose of the object to be grasped; wherein, the grasping pose is the actual pose of the object to be grasped, the first obstacle avoidance pose is the pose located in the target void, and the second obstacle avoidance pose is the pose located in the upper region of the object to be grasped in the target container; A control module, configured to control the robotic arm to move to the grasping pose and grasp the object to be grasped based on the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose, control the robotic arm to move from the grasping pose to the first obstacle avoidance pose, and control the robotic arm to move from the first obstacle avoidance pose to the second obstacle avoidance pose.

10. The device according to claim 9, characterized in that When the determination module determines the target void of the target container based on the poses of the objects, it is specifically configured to: fit the central pose of the object region in the container based on the poses of the objects; determine the central pose of the container of the target container; determine the target void based on the relative positions of the central pose of the object region and the central pose of the container; Alternatively, the central pose of the object region includes a first X-direction coordinate value and a first Y-direction coordinate value, and the central pose of the container includes a second X-direction coordinate value and a second Y-direction coordinate value; when the determination module determines the target void based on the relative positions of the central pose of the object region and the central pose of the container, it is specifically configured to: if the difference between the first X-direction coordinate value and the second X-direction coordinate value is greater than the difference between the first Y-direction coordinate value and the second Y-direction coordinate value, determine the target void as the left or right direction based on the first X-direction coordinate value and the second X-direction coordinate value; if the difference between the first X-direction coordinate value and the second X-direction coordinate value is less than the difference between the first Y-direction coordinate value and the second Y-direction coordinate value, determine the target void as the lower or upper direction based on the first Y-direction coordinate value and the second Y-direction coordinate value; Alternatively, when the determination module determines the objects located at the edge of the target void as objects to be grasped, it is specifically configured to: if there are multiple objects located at the edge of the target void, determine the grasping order of the multiple objects, and determine the object to be grasped from the multiple objects based on the grasping order; wherein, there is at least one intermediate object and two edge objects among the multiple objects, the edge objects are the objects close to the target container, and the intermediate object is the remaining object except the edge objects; in the grasping order, the intermediate object is in front of the edge objects; Alternatively, when obtaining the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose of the object to be grasped, the obtaining module is specifically configured to: determine the first post-movement pose of the object to be grasped based on the pose of the object to be grasped, the first movement direction, and the configured movement distance; wherein the first movement direction is a lateral movement direction or an oblique movement direction towards the target gap; determine the second post-movement pose of the object to be grasped based on the first post-movement pose, the configured height movement distance value, and the configured reference offset position; wherein the height movement distance value is greater than or equal to the height of a single object, and the reference offset position includes the lateral offset distance and the longitudinal offset distance between the second post-movement pose and the origin position; convert the pose of the object to be grasped into the grasping pose, convert the first post-movement pose into the first obstacle avoidance pose, and convert the second post-movement pose into the second obstacle avoidance pose; Alternatively, after determining the object located at the edge of the target gap as the object to be grasped, the obtaining module is further configured to determine the distance between the object to be grasped and the edge of the target container; if the distance is not greater than the configured blank distance threshold, output an alarm message; and / or determine whether a collision occurs between the object to be grasped and the edge of the target container when the object to be grasped moves to the first obstacle avoidance pose, and if so, output an alarm message; the alarm message indicates that the object to be grasped cannot be grasped; Alternatively, the pose of the object is the pose of the object in the container coordinate system of the target container, and the device further includes: a processing module, configured to collect a depth image through a depth camera, and obtain the pose of the target container in the camera coordinate system of the depth camera and the pose of the object in the camera coordinate system based on the depth image; based on the conversion relationship between the camera coordinate system and the robot coordinate system, convert the pose of the target container in the camera coordinate system into the reference pose of the target container in the robot coordinate system, and convert the pose of the object in the camera coordinate system into the pose of the object in the robot coordinate system; establish the container coordinate system with the reference pose as the origin, and convert the pose of the object in the robot coordinate system into the pose of the object in the container coordinate system; Alternatively, if the depth image includes a plurality of three-dimensional points, when the processing module obtains the pose of the target container in the camera coordinate system of the depth camera and the pose of the object in the camera coordinate system based on the depth image, it specifically: obtains a first type of three-dimensional points of the target container and a second type of three-dimensional points of the object from the plurality of three-dimensional points; determines the pose of the target container in the camera coordinate system based on the first type of three-dimensional points, and determines the pose of the object in the camera coordinate system based on the second type of three-dimensional points of the object; wherein, the similarity between the first type of three-dimensional points and the three-dimensional points of the container model is greater than a first threshold, and the similarity between the second type of three-dimensional points of the object and the three-dimensional points of the object model is greater than a second threshold; wherein, the three-dimensional points of the container model are determined based on the depth image corresponding to the sample container, the depth image only includes the sample container, and the three-dimensional points in the depth image are the three-dimensional points of the container model; wherein, the three-dimensional points of the object model are determined based on the depth image corresponding to the sample object, the depth image only includes the sample object, and the three-dimensional points in the depth image are the three-dimensional points of the object model.

11. An object grasping system, characterized in that, A plurality of objects are placed in the target container. The object grasping system includes a depth camera, a robotic arm, and a control device for the robotic arm, where: The depth camera is configured to collect a depth image and send the depth image to the control device; The control device is configured to obtain the pose of the object based on the depth image, determine the target void of the target container based on the pose of the object, and determine the object located at the edge of the target void as the object to be grasped; obtain the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose of the object to be grasped; wherein, the grasping pose is the actual pose of the object to be grasped, the first obstacle avoidance pose is the pose located in the target void, and the second obstacle avoidance pose is the pose located in the upper area of the object to be grasped in the target container; and send a control command to the robotic arm, the control command including the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose; The robotic arm is configured to, after receiving the control command, control the robotic arm to move to the grasping pose and grasp the object to be grasped based on the grasping pose, the first obstacle avoidance pose, and the second obstacle avoidance pose, control the robotic arm to move from the grasping pose to the first obstacle avoidance pose, and control the robotic arm to move from the first obstacle avoidance pose to the second obstacle avoidance pose.

Images