A robot unstacking method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a robot unstacking method and device, electronic equipment and storage medium, and relate to the field of robots. The method comprises: determining the pose of each material to be unstacked in a stack from an environment in which a robot is located based on perception data collected by an environment perception device deployed on the robot; generating stack structure description information of each material to be unstacked according to the pose of each material to be unstacked; determining a target material that satisfies an unstacking condition from each material to be unstacked based on the stack structure description information, wherein the unstacking condition comprises not providing support for other materials to be unstacked; and controlling the robot to unstack the target material from the stack. The embodiments of the present application can reduce the probability of material damage during the unstacking process.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a robot destacking method, apparatus, electronic device, and storage medium. Background Technology

[0002] In warehousing, logistics, and industrial manufacturing, various materials are typically stacked into pallets, which are then used for transportation to improve efficiency. After the pallets are transported to their destination, they need to be unstacked, meaning the materials are unloaded from the pallets so they can be further transferred to production lines, warehouses, and other areas.

[0003] Currently, robots can be used for automated depalletizing to reduce the intensity of manual labor.

[0004] In related technologies, when robots destacking, they typically select and unload the stacked materials one by one according to a set order. However, depending on the actual scenario, the stacking method of the materials often varies. When the robot destacking in a fixed order, it may damage the stability of the stack, causing it to deform or even collapse, resulting in damage to the materials. Summary of the Invention

[0005] The purpose of this invention is to provide a robotic depalletizing method, apparatus, electronic device, and storage medium to reduce the probability of material damage during depalletizing. The specific technical solution is as follows:

[0006] In a first aspect, embodiments of the present invention provide a robot depalletizing method, the method comprising:

[0007] Based on the sensing data collected by the environmental sensing devices deployed on the robot, the position and orientation of each material to be disassembled in the stack are determined from the environment in which the robot is located.

[0008] Based on the pose of each material to be disassembled, generate stacking structure description information of each material to be disassembled;

[0009] Based on the stacking structure description information, a target material that meets the disassembly conditions is determined from the materials to be disassembled, wherein the disassembly conditions include: not providing support for other materials to be disassembled;

[0010] Control the robot to unload the target material from the stack.

[0011] Secondly, embodiments of the present invention provide a robotic depalletizing device, the device comprising:

[0012] The pose determination module is used to determine the pose of each material to be disassembled in the stack based on the perception data collected by the environmental perception device deployed on the robot.

[0013] The information generation module is used to generate stacking structure description information of each material to be disassembled based on the pose of each material to be disassembled.

[0014] The material determination module is used to determine the target material that meets the disassembly conditions from the materials to be disassembled based on the stacking structure description information, wherein the disassembly conditions include: not providing support for other materials to be disassembled;

[0015] The destacking module is used to control the robot to unload the target material from the stack.

[0016] Thirdly, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0017] Memory, used to store computer programs;

[0018] When a processor executes a program stored in memory, it implements the steps of the method described in the first aspect.

[0019] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0020] Fifthly, embodiments of the present invention also provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps of the method described in the first aspect.

[0021] As can be seen from the above, when controlling the robot to automatically depalletize using the solution provided in this embodiment of the invention, the robot uses environmental sensing devices to collect data and detect the material poses within the stack in the environment. Based on these material poses, the robot analyzes the stack structure in real time. Then, based on the analyzed stack structure, a target material that meets the dismantling conditions is selected. This selected target material does not provide support for other materials in the stack. Finally, the robot is controlled to dismantle the target material from the stack. It is evident that by analyzing the stack structure in real time, a suitable target material for the robot's dismantling is selected. In this way, after the robot dismantles the target material from the stack, it is less likely to disrupt the support relationships between materials in the stack, thus reducing the probability of stack deformation and collapse, and consequently reducing the probability of material damage during the dismantling process. Furthermore, it ensures that the robot can continuously and stably perform depalletizing operations in various diverse and complex stacking scenarios.

[0022] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0024] Figure 1 A flowchart illustrating the first robot depalletizing method provided in an embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram of a material pose determination process provided in an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of a material stacking scenario provided in an embodiment of the present invention;

[0027] Figure 4 A flowchart illustrating the second robotic depalletizing method provided in an embodiment of the present invention;

[0028] Figure 5 This is a schematic diagram of a target material determination process provided in an embodiment of the present invention;

[0029] Figure 6 This is a schematic diagram of a target material disassembly process provided in an embodiment of the present invention;

[0030] Figure 7 This is a schematic diagram of the structure of a robot depalletizing device provided in an embodiment of the present invention;

[0031] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on the present invention are within the scope of protection of the present invention.

[0033] First, the application scenarios of the solutions provided in the embodiments of the present invention will be introduced.

[0034] The application scenario of the solution provided in this embodiment of the invention is: various robots automatically unpacking materials from a stack. The aforementioned materials can be various bins for storing specific items. Of course, if the aforementioned items themselves have a regular shape that can be stacked, the aforementioned materials can also be the items themselves.

[0035] The robot depalletizing scheme provided in the embodiments of the present invention will be described in detail below with reference to the flowchart.

[0036] See Figure 1 The above is a flowchart illustrating the first robot depalletizing method provided in an embodiment of the present invention. The method includes the following steps S101 to S104.

[0037] Step S101: Based on the sensing data collected by the environmental sensing device deployed on the robot, determine the pose of each material to be disassembled in the stack from the environment in which the robot is located.

[0038] The aforementioned environmental sensing devices may include image acquisition devices or point cloud acquisition devices. Specifically, the image acquisition devices may be cameras used to acquire various types of visible light images, or RGB-D cameras used to simultaneously acquire visible light and depth images; the point cloud acquisition devices may be lidar, structured light sensors, or binocular stereo cameras used to acquire environmental point clouds, etc.

[0039] Correspondingly, the sensing data collected by environmental sensing devices can include visible light images, depth images, and point clouds.

[0040] The embodiments of the present invention do not limit the method of determining the pose of the material to be disassembled in the stack based on the sensing data. Specifically, one or more of the data mentioned above, such as visible light images, depth images and point clouds, can be used to determine the pose of the material to be disassembled. Examples are given below.

[0041] In one possible implementation, the advantages of visible light images and point clouds can be combined to determine the pose in the following way:

[0042] First, a target detection algorithm is used to detect the materials to be disassembled from the environmental image. Then, based on the position of the materials to be disassembled in the environmental image, the material point cloud corresponding to each material to be disassembled is extracted from the environmental point cloud. Finally, for each material to be disassembled, the material point cloud of the material to be disassembled is registered with the point cloud template of the material to be disassembled to obtain the pose of the material to be disassembled. The aforementioned point cloud template is a pre-constructed standard point cloud template for the material.

[0043] Understandably, when there are multiple types of materials to be disassembled, a point cloud template corresponding to the detected material type can be selected first, and then the point cloud of the material to be disassembled can be registered with the selected point cloud template. This allows the pose of different types of materials to be determined, making the solution provided in this embodiment of the invention adaptable to scenarios where there are multiple specifications of materials in the stack.

[0044] For details, see Figure 2The following steps, P1 to P7, can be used to determine the position of the material to be disassembled.

[0045] Step P1: Detect and segment the materials in the stack to obtain a pixel-level mask image.

[0046] After detecting the material to be disassembled from the environmental image, for each material to be disassembled, the image can be segmented into pixels, and a segmentation mask, i.e., a pixel-level mask image, can be generated.

[0047] Step P2: Project the mask image onto the environmental point cloud, and extract the material point cloud based on the projection result.

[0048] Based on the pre-defined transformation relationship between the image coordinate system and the point cloud coordinate system, the segmentation mask is mapped to the point cloud coordinate system to obtain a local point cloud aligned with the environmental point cloud. Then, points located in the area of ​​the local point cloud are extracted from the environmental point cloud to obtain the material point cloud of the material to be disassembled.

[0049] Step P3: Point cloud normal vector estimation.

[0050] In this step, normal vector estimation needs to be performed on the points in the material point cloud and the point cloud template, respectively.

[0051] Specifically, for each point in the material point cloud or point cloud template, the plane in which the point is located can be fitted using the least squares method, and the normal vector of the plane can be used as the normal vector of the point; or, the normal vector of the point can be determined using algorithms such as covariance matrix decomposition. This embodiment of the invention does not limit this.

[0052] Step P4: Calculate the structural features of the point cloud and use structural feature matching to determine the initial matching point pairs.

[0053] Specifically, the normal vector relationship between each point and its neighboring points can be calculated. Based on the above normal vector relationship, structural features such as Fast Point Feature Histogram (FPFH) can be calculated for each point in the two types of point clouds. Then, by calculating the similarity of FPFH features between point pairs in the two types of point clouds, the initial matching point pairs are determined in the two types of point clouds.

[0054] Step P5: Perform coarse registration of the point cloud using the initial matched point pairs as input.

[0055] Specifically, using the initial matching point pairs as input, algorithms such as Random Sample Consensus (RANSAC) are employed to solve the pose of the material point cloud relative to the point cloud template, which serves as the coarse registration pose of the material to be disassembled.

[0056] Step P6: Using the coarse registration result as the initial value, perform fine registration of the point cloud.

[0057] Using the obtained coarse registration pose as the initial value, the two types of point clouds are finely registered using algorithms such as Iterative Closest Point (ICP).

[0058] Step P7: Obtain the 6D pose of the material through iterative optimization.

[0059] The initial values ​​are updated based on the fine registration results, and the fine registration of the point cloud is repeated multiple times to obtain a high-precision six-dimensional (6D) pose of the material to be disassembled. The 6D pose includes three-dimensional translation parameters and three-dimensional rotation parameters. The translation parameters can be three-dimensional coordinate values, and the rotation parameters can include roll angle parameters, pitch angle parameters, and yaw angle parameters.

[0060] In this way, when extracting the point cloud of the material to be disassembled, not only point cloud data is utilized, but also visual information provided by visible light images is combined. Through multimodal data fusion perception, the adverse effects of local occlusion and noise points on point cloud extraction are reduced, the interference of background and nearby objects when extracting material point cloud is reduced, and the accuracy of the determined material pose is improved.

[0061] In one possible implementation, in addition to visible light images and point clouds, depth images can be combined to determine the pose of the material to be disassembled. This allows for a more accurate conversion of pixel coordinates in the image into 3D point cloud coordinates, guided by depth information. The difference between this implementation and the previously described one lies in the fact that, when determining the local point cloud, the pixel coordinates corresponding to the segmentation mask are first mapped to the depth image based on camera intrinsic parameters to obtain the depth information of each pixel of the material to be disassembled. Then, the pixel coordinates and depth information of the material to be disassembled are converted into 3D point cloud coordinates using the camera imaging model, resulting in the local point cloud of the material to be disassembled. The remaining steps are similar to the previous implementation and will not be detailed here.

[0062] It should be noted that in this step, based on the sensing data, not only can the pose of each material to be disassembled be detected, but also the size information of each material to be disassembled can be detected at the same time. The specific method will not be described in detail.

[0063] Step S102: Generate stacking structure description information for each material to be disassembled based on the pose of each material to be disassembled.

[0064] The above-mentioned stacking structure description information can be any information used to describe the stacking method and relative positional relationship of materials in the stack, and the embodiments of the present invention do not limit it.

[0065] Preferably, the stacking structure description information may include, but is not limited to: the stacking level to which the material to be disassembled belongs, the local position information of each material to be disassembled in its stacking level, and the projection information of each material to be disassembled on the horizontal plane.

[0066] The stacking level of the materials to be disassembled can be determined by analyzing their height within the stack. Specifically, the previous step has already obtained the position of each material to be disassembled, which can be the three-dimensional coordinates of the center point of the material. In this case, the target coordinate values ​​(usually Z-axis coordinates) corresponding to the height of each material can be extracted first. Considering differences in actual scenarios and data acquisition errors, materials whose target coordinate values ​​differ by less than a threshold can be identified as belonging to the same stacking level. Then, based on the relationship between the target coordinate values ​​of materials in different stacking levels, the order of each stacking level can be determined, ultimately identifying the specific stacking level to which each material to be disassembled belongs.

[0067] The local location information of the material to be dismantled within its stacking hierarchy can include local two-dimensional coordinates of the material within the area covered by the stacking hierarchy, or it can include the local region category of the material within its stacking hierarchy, which can include a central region, an edge region, etc. Based on the known poses of each material to be dismantled, the local location information can be specifically obtained by analyzing the horizontal coordinates of the material to be dismantled.

[0068] The projection information of each material to be disassembled on the horizontal plane is used to describe the projection area of ​​each material to be disassembled on the horizontal plane, which can be obtained based on the pose and size of each material to be disassembled.

[0069] Step S103: Based on the stacking structure description information, determine the target material that meets the disassembly conditions from each material to be disassembled.

[0070] The aforementioned dismantling conditions include: not providing support for other materials to be dismantled. Target materials that meet these dismantling conditions are those within the stack that do not provide support for other materials to be dismantled. Dismantling the target material from the stack will not damage the stacking structure and will therefore not affect the stability of the stack.

[0071] Similarly, the embodiments of the present invention do not limit the specific method of determining the material that does not provide support for other materials to be disassembled from the materials to be disassembled. The following is an example.

[0072] For materials to be dismantled at the top of the stack, there are no other materials vertically above them, meaning they are not covered by any other materials. Therefore, materials to be dismantled at the top of the stack will not provide support for other materials. In this regard, one possible implementation is to directly determine the material at the top of the stack from among the materials to be dismantled, based on the stacking level of each material included in the stacking structure description information, as the target material that will not provide support for other materials to be dismantled. This improves the efficiency of determining the target material and ensures that material dismantling does not compromise the stability of the stack.

[0073] In some cases, materials to be dismantled that are not located on the top layer of the stack may not be covered by other materials, and in such cases, these materials will not provide support for other materials. See also Figure 3 The diagram shows a side view of a stack, where rectangles represent materials. It can be seen that not only do the two materials to be dismantled at the top of the stack (represented by shaded rectangles) not provide support for other materials, but also one material to be dismantled on the third layer from the top (also represented by a shaded rectangle) does not provide support for other materials. Therefore, in one possible implementation, based on the stacking level of each material to be dismantled and the projection information of each material on the horizontal plane included in the stacking structure description information, materials from among the materials to be dismantled that have no other materials above them can be identified as target materials that do not provide support for other materials to be dismantled.

[0074] Specifically, for a material to be disassembled in a stacking layer other than the top layer of the stack, it can be determined whether the projected area corresponding to the material to be disassembled overlaps with the projected areas corresponding to each material to be disassembled in the previous stacking layer. If there is no overlap, it means that there are no other materials covering the material to be disassembled, thus it can be determined that there are no other materials to be disassembled above the material to be disassembled; otherwise, it means that there are other materials covering the material to be disassembled, thus it can be determined that there are other materials to be disassembled above the material to be disassembled.

[0075] In this way, based on the projection information of each material to be disassembled on the horizontal plane, materials that do not have other materials to be disassembled above them can be accurately identified, so that the disassembly of materials will not damage the stability of the stack.

[0076] Step S104: Control the robot to unload the target material from the stack.

[0077] Once a suitable target material is selected, the robot's gripping posture can be planned based on the target material's position, and the robot can be controlled to unload the target material from the stack.

[0078] In one possible implementation, in order to improve the robot's grasping efficiency and grasping success rate, a grasping preparation posture that allows the robot to move toward the grasping favorable surface of the target material can be determined based on the first posture of the target material and the second posture of the robot; then, the robot is controlled to reach the grasping preparation posture and unload the target material from the stack.

[0079] The surface to be gripped can be a surface that is set by the operator to facilitate the gripping and handling of materials by the end effector. The angle between the gripping favorable surface and the target material's set gripping surface is within the set angle range.

[0080] The aforementioned angle range can be set by the operator based on the specific shape of the material to be disassembled in the actual scenario. This embodiment of the invention does not limit this, only requiring that the included angle between the advantageous gripping surface and the surface to be gripped is close to 90°. In this way, when the robot faces the advantageous gripping surface, its end effector is parallel or nearly parallel to the surface to be gripped, facilitating direct contact with the surface to be gripped for gripping.

[0081] Considering that the materials to be disassembled are mostly regular in shape and that the surfaces of the materials are perpendicular to each other, preferably, the advantageous gripping surface can be directly determined as: the surface that is perpendicular to the gripping surface and perpendicular to the horizontal plane.

[0082] For example, assuming the material is a rectangular box, the surfaces to be gripped are the width and height of the box. The advantageous gripping surfaces can be the length and height of the box. In this way, when the robot faces the advantageous gripping surfaces, its end effector is parallel to the surface to be gripped, which is beneficial for the end effector to grip the box.

[0083] The embodiments of the present invention do not limit the specific manner in which the robot is determined to be in a preparatory gripping posture that enables it to grip the target material toward a favorable gripping surface.

[0084] In one possible implementation, if the material to be disassembled is stacked on a pallet, a perpendicular line can be drawn from the centerline of the projection area of ​​the target material on the horizontal plane to the target edge of the pallet, such that the perpendicular line passes through the target edge by a set distance. The endpoint of the perpendicular line is taken as the position in the grasping pre-pose, and the direction from the endpoint of the perpendicular line to the starting point is taken as the robot's orientation (posture) in the grasping pre-pose. Here, the target edge is the edge of the pallet with the smallest angle between it and the grasping surface of the target material.

[0085] It should be noted that the robot can repeatedly execute this scheme during the entire automated depalletizing process, thereby determining the target material that is suitable to be grasped and dismantling each time, until all the materials to be dismantled have been dismantled.

[0086] As can be seen from the above, when controlling the robot to automatically depalletize using the solution provided in this embodiment of the invention, the robot uses environmental sensing devices to collect data and detect the material poses within the stack in the environment. Based on these material poses, the robot analyzes the stack structure in real time. Then, based on the analyzed stack structure, a target material that meets the dismantling conditions is selected. This selected target material does not provide support for other materials in the stack. Finally, the robot is controlled to dismantle the target material from the stack. It is evident that by analyzing the stack structure in real time, a suitable target material for the robot's dismantling is selected. In this way, after the robot dismantles the target material from the stack, it is less likely to disrupt the support relationships between materials in the stack, thus reducing the probability of stack deformation and collapse, and consequently reducing the probability of material damage during the dismantling process. Furthermore, it ensures that the robot can continuously and stably perform depalletizing operations in various diverse and complex stacking scenarios.

[0087] exist Figure 1 Based on the illustrated embodiment, considering that materials may be tightly stacked in some cases, the robot's end effector has limited operating space and is difficult to grasp when gripping tightly stacked materials. To screen materials with sufficient operating space and thus improve the success rate of the robot's material gripping, the dismantling condition may further include the presence of at least one grippable surface. In view of the above, this embodiment of the invention provides a second robot destacking method.

[0088] See Figure 4 The above is a flowchart illustrating the second robot depalletizing method provided in the embodiment of the present invention. The method includes the following steps S401 to S407.

[0089] Step S401: Based on the sensing data collected by the environmental sensing device deployed on the robot, determine the pose of each material to be disassembled in the stack from the environment in which the robot is located.

[0090] Step S402: Generate stacking structure description information for each material to be disassembled based on the pose of each material to be disassembled.

[0091] Step S403: Based on the stack structure description information, determine the first material from the materials to be disassembled that does not provide support for other materials to be disassembled.

[0092] The method for determining the first material can be found in [reference needed]. Figure 1 The method for determining the target material described in the illustrated embodiment will not be repeated here.

[0093] Step S404: Based on the pose and size of the first material, determine the distance between the gripping surface of each first material and other materials to be disassembled.

[0094] Given the orientation and dimensions of the materials, the distance between the gripping surface of the first material and its adjacent materials can be calculated based on the orientation and dimensions of the first material and its adjacent materials.

[0095] Step S405: Determine at least one second material from the first material whose distance to the surface to be grasped is greater than a set distance.

[0096] In this way, not only is there no other material covering the second material, but at least one of the surfaces of the second material to be grasped also has a large gap with other materials, which makes it easy for the robot's end effector to grasp it.

[0097] Step S406: Determine the target material from the second material.

[0098] Specifically, the material with the largest distance between the surface to be grasped and its adjacent materials can be selected from the second material as the target material, which further facilitates the robot's end effector to grasp it; alternatively, the material closest to the robot can be selected from the second material to improve the robot's grasping efficiency.

[0099] To further reduce the adverse effects of dismantled materials on the stack structure and lower the probability of compromising stack stability, one possible implementation is to select target materials from the second set of materials, preferably those located at the edge of the stack. Specifically, based on the local location information of each material to be dismantled within its stacking level, as included in the stack structure description information, a third material located outside the target stacking level can be determined from the second set of materials.

[0100] The target stacking level, i.e. the stacking level to which the second material belongs, can be determined based on the stacking level of each material to be disassembled, as included in the stacking structure description information.

[0101] Step S407: Control the robot to unload the target material from the stack.

[0102] For instructions on controlling the robot to unload the target material from the stack, please refer to the aforementioned text. Figure 1 The description of step S104 in the illustrated embodiment will not be repeated here.

[0103] Considering that the target material determined in the above manner may have one grippable surface or two grippable surfaces, in order for the robot to adaptively determine the gripping mode based on the number of grippable surfaces of the target material, so as to successfully unload the target material in complex scenarios, in one possible implementation, different methods for disassembling the target material can be determined for the robot according to the different number of grippable surfaces.

[0104] Specifically, when the number of grippable surfaces of the target material is 1, the target gripping mode is determined to be a single-hand gripping mode; when the number of grippable surfaces of the target material is 2, the target gripping mode is determined to be a two-hand gripping mode. Then, according to the target gripping mode, the robot's end effector is controlled to grip the gripping area on the grippable surface of the target material so that the robot can unload the target material from the stack.

[0105] The gripping area on the gripping surface can be a designated recessed area, an anti-slip area, or the area where gripping components such as pull rings and handles are located on the gripping surface.

[0106] The following is an example of how to control the end effector to grasp the above-mentioned grasping area according to the target grasping mode.

[0107] For example, when the target grasping mode includes a two-hand grasping mode, the two end effectors of the robot can be directly controlled to grasp the gripping areas on the two grippable surfaces of the target material.

[0108] When the target grasping mode includes a single-handed grasping mode, the first end effector is determined from the two end effectors based on the position of the graspable surface in the target material; the first end effector is controlled to grasp the gripping area on the graspable surface and move along the target direction until a new graspable surface is detected on the target material, the target direction being: away from other materials to be disassembled located in the same stack level as the target material; then, the robot's second end effector is controlled to grasp the gripping area on the new graspable surface.

[0109] The first end effector can be the end effector closest to the graspable surface, and the second end effector is the end effector other than the first end effector.

[0110] In other words, if the target material has only one grippable surface, one end effector of the robot can be controlled to grasp the gripping area of ​​the grippable surface and drag it until a new grippable surface appears; then the other end effector of the robot can be controlled to grasp the gripping area of ​​the new grippable surface. In this way, the robot can successfully transport the object.

[0111] In this way, even if the target material has only one graspable surface, the robot can successfully grasp the target material by first grasping, then dragging, and then grasping again, which improves the robot's adaptability to scenarios with densely stacked materials.

[0112] based on Figure 4 The following is a description of a specific process for a robot to select suitable target materials for grasping, based on the example shown in the embodiment.

[0113] See Figure 5 This illustrates a target material selection process, including the following steps M1 to M9.

[0114] Step M1: Generate stacking structure description information based on the material pose.

[0115] Step M2: Select the top layer of the stack as the candidate material set.

[0116] Step M3: Determine whether there are other materials within the set range of the left and right gripping surfaces of each material in the set. If yes, proceed to step M4; otherwise, proceed to step M5.

[0117] In this step, it is determined whether there are other materials on both the left and right gripping surfaces of the material in the set, that is, whether the material has a gripping surface.

[0118] Step M4: Remove the material from the collection.

[0119] That is, if there are other materials within the set range of the left and right gripping surfaces of the material, it means that there is no gap between the left and right gripping surfaces of the material and other materials to meet the gripping requirements. In other words, the material has no gripping surface and cannot be gripped, so it is removed from the set.

[0120] Step M5: Retain the material in the collection.

[0121] That is, if there are no other materials within the set range of at least one gripping surface of the material, it means that there is a gap between at least one gripping surface of the material and other materials that meets the gripping requirements. In other words, the material has a gripping surface that can be gripped by the robot and retained.

[0122] Step M6: Determine if there are materials located at the outer edge of the stack in the set. If yes, proceed to step M7; otherwise, proceed to step M8.

[0123] Step M7: Retain the material located at the outer edge of the stack in the collection.

[0124] Step M8: Keep the set unchanged.

[0125] Step M9: Select the material that is closest to the robot from the set as the target material.

[0126] Following the above process, a target material located at the top layer of the stack that does not provide support for other materials can be selected; and the gripping surface of the target material is at a certain distance from other materials, which facilitates the robot's end effector to perform gripping operations; in addition, the target material is located at the outer edge of the stack, which further reduces the probability of affecting the stability of the stack after destabilization.

[0127] To improve the gripping stability of the robot's end effector and achieve continuous closed-loop depalletizing operations, one possible implementation is to introduce technologies such as end-effector posture compensation and multi-segment gripping trajectory during material dismantling. The following example, using the end effector as a gripper, illustrates a specific material dismantling process provided by an embodiment of the present invention.

[0128] See Figure 6 This illustrates a material disassembly process, including the following steps N1 to N9.

[0129] Step N1: Determine that the robot has reached the grasping preparatory pose.

[0130] Step N2: Determine the grasping mode based on visual and material characteristics.

[0131] As mentioned earlier, the grasping modes can include single-hand grasping mode and two-hand grasping mode.

[0132] It should be noted that if the robot detects that the target material is not present or the position does not meet the disassembly conditions after reaching the pre-grabbing posture, the process of determining the target material can be re-executed.

[0133] Step N3: Determine the end attitude compensation parameters.

[0134] End-effector attitude compensation parameters may include, but are not limited to, tool center point (TCP) offset compensation parameters, gripping angle compensation parameters, and anti-collision compensation parameters.

[0135] Step N4: Generate a multi-segment grasping trajectory based on the end-effector attitude compensation parameters and the grasping mode.

[0136] In this step, by combining the aforementioned end-effector posture compensation parameters and grasping mode, segmented robot grasping trajectories are generated while avoiding grasping offset and environmental collisions.

[0137] Step N5: Execute the grasping trajectory through synchronous interpolation of both arms.

[0138] Dual-arm synchronous interpolation refers to the robot automatically completing the transition paths of each action stage during the grasping process, while simultaneously controlling the movement rhythm, timing, and relative pose of the two arms to ensure that they complete the actions synchronously, achieving smooth cooperative grasping and handling. The aforementioned action stages may include, but are not limited to, approaching, pressing down and fitting, gripping, lifting, and adjusting the handling pose.

[0139] Step N6: Determine the gripper's grasping status in real time. If the grasping is successful, proceed to step N7; if the grasping fails, proceed to step N9.

[0140] Step N7: Use both arms to move the object to the placement location.

[0141] Step N8: Coordinated handling with both arms.

[0142] In this step, dual-arm coordinated handling and posture adjustment are performed to keep the center of gravity of the material at the center line of the two arms, and visual correction is combined when necessary.

[0143] Step N9: Gripper motion control, return to step N6.

[0144] If the grasping fails, the robotic arm can be adjusted based on the visual perception results to retry the grasping.

[0145] This enables the robot's grasping actions to be adjusted in a closed loop based on the grasping results, thereby ensuring the stable completion of continuous automatic depalletizing operations and improving operational efficiency and reliability.

[0146] Corresponding to the above-described robot depalletizing method, this embodiment of the invention also provides a robot depalletizing device.

[0147] See Figure 7 The above is a schematic diagram of a robot depalletizing device provided in an embodiment of the present invention. The device includes the following modules:

[0148] The pose determination module 701 is used to determine the pose of each material to be disassembled in the stack based on the perception data collected by the environmental perception device deployed on the robot.

[0149] The information generation module 702 is used to generate stacking structure description information of each material to be disassembled based on the pose of each material to be disassembled.

[0150] The material determination module 703 is used to determine the target material that meets the disassembly conditions from the materials to be disassembled based on the stacking structure description information, wherein the disassembly conditions include: not providing support for other materials to be disassembled;

[0151] The destacking module 704 is used to control the robot to unload the target material from the stack.

[0152] As can be seen from the above, when controlling the robot to automatically depalletize using the solution provided in this embodiment of the invention, the robot uses environmental sensing devices to collect data and detect the material poses within the stack in the environment. Based on these material poses, the robot analyzes the stack structure in real time. Then, based on the analyzed stack structure, a target material that meets the dismantling conditions is selected. This selected target material does not provide support for other materials in the stack. Finally, the robot is controlled to dismantle the target material from the stack. It is evident that by analyzing the stack structure in real time, a suitable target material for the robot's dismantling is selected. In this way, after the robot dismantles the target material from the stack, it is less likely to disrupt the support relationships between materials in the stack, thus reducing the probability of stack deformation and collapse, and consequently reducing the probability of material damage during the dismantling process. Furthermore, it ensures that the robot can continuously and stably perform depalletizing operations in various diverse and complex stacking scenarios.

[0153] In one possible implementation, the disassembly condition further includes: the existence of at least one graspable surface, which belongs to a predetermined graspable surface and is at a distance greater than a predetermined distance from other materials to be disassembled; the material determination module includes:

[0154] The first material determination submodule is used to determine, based on the stacking structure description information, a first material that does not provide support for other materials to be disassembled from the materials to be disassembled.

[0155] The distance determination submodule is used to determine the distance between the gripping surface of each of the first materials and other materials to be disassembled, based on the pose and size of the first materials.

[0156] The second material determination submodule is used to determine from the first material at least one second material whose distance to the surface to be grasped is greater than the set distance;

[0157] The target material determination submodule is used to determine the target material from the second material.

[0158] This allows for the selection of materials with sufficient operating space, thereby increasing the success rate of the robot in grasping materials.

[0159] In one possible implementation, the first material determination submodule is specifically used to determine, based on the stacking level of each material to be disassembled included in the stacking structure description information, the material located at the top layer of the stack as the first material; or, based on the stacking level of each material to be disassembled included in the stacking structure description information and the projection information of each material to be disassembled on the horizontal plane, determine, based on the material to be disassembled, the material above which there are no other materials to be disassembled as the first material.

[0160] This improves the efficiency of identifying target materials and ensures that material removal does not compromise the stability of the stack. Furthermore, based on the projection information of each material to be removed onto the horizontal plane, materials that are not above other materials to be removed can be accurately identified.

[0161] In one possible implementation, the target material determination submodule is specifically used to determine a third material located outside the target stacking level from the second material based on the local position information of each material to be disassembled in its stacking level, as included in the stacking structure description information, wherein the target stacking level is: the stacking level to which the second material belongs, determined based on the stacking levels of each material to be disassembled included in the stacking structure description information; and to determine the target material from the third material.

[0162] This can further reduce the adverse effects of dismantling materials on the stack structure and lower the probability of damaging the stack's stability.

[0163] In one possible implementation, the destacking module includes:

[0164] The first mode determination submodule is used to determine the target gripping mode as a single-hand gripping mode when the number of grippable surfaces of the target material is 1.

[0165] The second mode determination submodule is used to determine the target gripping mode as a two-hand gripping mode when the number of grippable surfaces of the target material is 2.

[0166] The destacking module, according to the target grasping mode, controls the robot's end effector to grasp the grasping area on the graspable surface of the target material, so that the robot unloads the target material from the stack.

[0167] This allows the robot to adaptively determine the gripping mode based on the number of grippable surfaces of the target material, thus enabling it to successfully unload the target material in complex scenarios.

[0168] In one possible implementation, the destacking module is specifically configured to perform at least one of the following: when the target gripping mode includes a two-hand gripping mode, controlling the two end effectors of the robot to grip the gripping areas on two grippable surfaces of the target material respectively; when the target gripping mode includes a one-hand gripping mode, determining a first end effector from the two end effectors based on the position of the grippable surface in the target material; controlling the first end effector to grip the gripping area on the grippable surface and move along a target direction until a new grippable surface is detected on the target material, the target direction being: away from other materials to be destacking located in the same stack level as the target material; controlling the second end effector of the robot to grip the gripping area on the new grippable surface, wherein the second end effector is: the end effector other than the first end effector among the two end effectors.

[0169] In this way, even if the target material has only one graspable surface, the robot can successfully grasp the target material by first grasping, then dragging, and then grasping again, which improves the robot's adaptability to scenarios with densely stacked materials.

[0170] In one possible implementation, the destacking module is specifically used to determine a pre-grabbing posture for the robot to face the advantageous gripping surface of the target material based on the first posture of the target material and the second posture of the robot, wherein the advantageous gripping surface is a surface perpendicular to a pre-defined gripping surface of the target material; and to control the robot to unload the target material from the stack after reaching the pre-grabbing posture.

[0171] This can improve the robot's grasping efficiency and grasping success rate.

[0172] In one possible implementation, the perceived data includes environmental images and environmental point clouds.

[0173] The pose determination module is specifically used to detect the material to be disassembled from the environmental image; extract the material point cloud corresponding to each material to be disassembled from the environmental point cloud according to the position of the material to be disassembled in the environmental image; and register the material point cloud of the material to be disassembled with the point cloud template of the material to be disassembled for each material to be disassembled to obtain the pose of the material to be disassembled.

[0174] In this way, when extracting the point cloud of the material to be disassembled, not only point cloud data is utilized, but also visual information provided by visible light images is combined. Through multimodal data fusion perception, the adverse effects of local occlusion and noise points on point cloud extraction are reduced, the interference of background and nearby objects when extracting material point cloud is reduced, and the accuracy of the determined material pose is improved.

[0175] Corresponding to the above-described robot depalletizing method, embodiments of the present invention also provide an electronic device, a storage medium, and a program product.

[0176] This invention also provides an electronic device, such as... Figure 8 As shown, it includes a processor 801, a communication interface 802, a memory 803, and a communication bus 804, wherein the processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804.

[0177] Memory 803 is used to store computer programs;

[0178] The processor 801 is used to execute the program stored in the memory 803 to implement the aforementioned robot destacking method.

[0179] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0180] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0181] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0182] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0183] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described robot depalletizing methods.

[0184] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the robot depalletizing methods described above.

[0185] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0186] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0187] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, and storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0188] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A robot depalletizing method, characterized in that, The method includes: Based on the sensing data collected by the environmental sensing devices deployed on the robot, the position and orientation of each material to be disassembled in the stack are determined from the environment in which the robot is located. Based on the pose of each material to be disassembled, a stacking structure description information of each material to be disassembled is generated, the stacking structure description information including the stacking level to which the material to be disassembled belongs; Based on the stacking structure description information, a target material that meets the disassembly conditions is determined from the materials to be disassembled, wherein the disassembly conditions include: not providing support for other materials to be disassembled; Control the robot to unload the target material from the stack; The disassembly conditions further include: the existence of at least one grippable surface, which is a designated grippable surface and is at a distance greater than a designated distance from other materials to be disassembled; and determining the target material that meets the disassembly conditions from among the materials to be disassembled based on the stacking structure description information includes: Based on the stacking level of each material to be disassembled and the projection information of each material to be disassembled on the horizontal plane included in the stacking structure description information, a material with no other material to be disassembled above it is identified as the first material; based on the pose and size of the first material, the distance between the grasping surface of each first material and other materials to be disassembled is determined; a second material with at least one grasping surface corresponding to a distance greater than the set distance is identified from the first material; and a target material is identified from the second material.

2. The method according to claim 1, characterized in that, The step of determining the target material from the second material includes: Based on the local position information of each material to be disassembled in its stacking level, which is included in the stacking structure description information, a third material located outside the target stacking level is determined from the second material. The target stacking level is the stacking level to which the second material belongs, which is determined based on the stacking level of each material to be disassembled, which is included in the stacking structure description information. The target material is determined from the third material.

3. The method according to claim 1, characterized in that, The control of the robot to unload the target material from the stack includes: When the number of grippable surfaces of the target material is 1, the target gripping mode is determined to be a single-hand gripping mode; When the number of grippable surfaces of the target material is 2, the target gripping mode is determined to be a two-hand gripping mode; According to the target grasping mode, the robot's end effector is controlled to grasp the grasping area on the graspable surface of the target material, so that the robot unloads the target material from the stack.

4. The method according to claim 3, characterized in that, The step of controlling the robot's end effector to grasp the gripping area on the grippable surface of the target material according to the target grasping mode includes at least one of the following: When the target grasping mode includes a two-hand grasping mode, the two end effectors of the robot are controlled to grasp the gripping areas on the two grippable surfaces of the target material respectively; When the target grasping mode includes a single-handed grasping mode, a first end effector is determined from the two end effectors based on the position of the graspable surface in the target material; the first end effector is controlled to grasp the gripping area on the graspable surface and move along a target direction until a new graspable surface is detected on the target material, the target direction being: away from other materials to be disassembled located in the same stack level as the target material; the second end effector of the robot is controlled to grasp the gripping area on the new graspable surface, wherein the second end effector is: the end effector other than the first end effector among the two end effectors.

5. The method according to any one of claims 1 to 4, characterized in that, The control of the robot to unload the target material from the stack includes: Based on the first pose of the target material and the second pose of the robot, a pre-grabbing pose is determined so that the robot faces the advantageous gripping surface of the target material. The advantageous gripping surface is a surface whose angle with the set gripping surface of the target material is within a set angle range. After the robot reaches the pre-grabbing pose, it unloads the target material from the stack. And / or, The sensing data includes environmental images and environmental point clouds. The determination of the pose of each material to be disassembled in the stack based on the sensing data collected by the environmental sensing device deployed on the robot, from the environment in which the robot is located, includes: Detect the material to be disassembled from the environmental image; extract the material point cloud corresponding to each material to be disassembled from the environmental point cloud according to the position of the material to be disassembled in the environmental image; for each material to be disassembled, register the material point cloud of the material to be disassembled with the point cloud template of the material to be disassembled to obtain the pose of the material to be disassembled.

6. A robotic depalletizing device, characterized in that, The device includes: The pose determination module is used to determine the pose of each material to be disassembled in the stack based on the perception data collected by the environmental perception device deployed on the robot. The information generation module is used to generate stacking structure description information of each material to be disassembled based on the pose of each material to be disassembled. The stacking structure description information includes the stacking level to which the material to be disassembled belongs. The material determination module is used to determine the target material that meets the disassembly conditions from the materials to be disassembled based on the stacking structure description information, wherein the disassembly conditions include: not providing support for other materials to be disassembled; A destacking module is used to control the robot to unload the target material from the stack. The disassembly conditions further include: the existence of at least one graspable surface, which is a designated graspable surface and is at a distance greater than a designated distance from other materials to be disassembled; the material determination module includes: The first material determination submodule is used to determine, based on the stacking level of each material to be disassembled and the projection information of each material to be disassembled on the horizontal plane included in the stacking structure description information, a material from the materials to be disassembled that has no other materials to be disassembled above it, as the first material; the distance determination submodule is used to determine, based on the pose and size of the first material, the distance between the grasping surface of each first material and other materials to be disassembled; the second material determination submodule is used to determine, from the first material, at least one second material whose grasping surface corresponds to a distance greater than the set distance; the target material determination submodule is used to determine, from the second material, a target material.

7. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1 to 5.

Images