A robot control method, device, control apparatus, and storage medium

Abstract

This invention discloses a robot control method, apparatus, control device, and storage medium. The method includes: acquiring 3D point cloud data of each material in the highest material layer of a material stack; determining, based on the 3D point cloud data, at least two first materials to be simultaneously grasped and a second material to be grasped individually from all materials in the highest material layer; determining the first center point position information of the at least two first materials as a whole and the second center point position information of the second material; and controlling the robot to simultaneously grasp at least two first materials based on the first center point position information, and controlling the robot to individually grasp the second material based on the second center point position information, according to a preset grasping order. The technical solution of this invention uses a 3D camera to acquire a large amount of material point cloud data and determines the materials that can be grasped simultaneously based on the point cloud data, enabling the simultaneous grasping of multiple materials and improving equipment production efficiency.

Description

Technical Field

[0001] This invention relates to the field of robot control, and more particularly to a robot control method, apparatus, control device, and storage medium. Background Technology

[0002] Robotic material handling systems are widely used in automated production lines, logistics warehousing, and other industrial fields, efficiently completing material transportation and delivery tasks. However, traditional 3D vision-guided robot material handling technology uses a single gripper / nozzle, which can only grasp one material at a time, resulting in slow material handling speed and low equipment production efficiency. Summary of the Invention

[0003] This invention provides a robot control method, device, control equipment, and storage medium. It uses a 3D camera to acquire a large amount of material point cloud data and determines the materials that can be grasped simultaneously based on the point cloud data, thereby enabling the simultaneous grasping of multiple materials and improving equipment production efficiency.

[0004] According to one aspect of the present invention, a robot control method is provided, the method comprising:

[0005] Obtain 3D point cloud data of each material in the highest material layer of the material stack;

[0006] Based on the 3D material point cloud data, determine at least two first materials to be grasped simultaneously and a second material to be grasped separately from all materials in the highest material layer;

[0007] Determine the first center point location information of the at least two first materials as a whole and the second center point location information of the second material;

[0008] According to the preset grasping order, the robot is controlled to simultaneously grasp at least two first materials based on the first center point position information, and the robot is controlled to grasp the second material individually based on the second center point position information.

[0009] Based on the 3D material point cloud data, at least two first materials to be simultaneously grasped and a second material to be grasped separately are determined from all materials in the highest material layer, including:

[0010] The relative position information between each material is determined based on the 3D material point cloud data; wherein, the relative position information includes contour parallelism, left and right gap of contour, front and back tolerance of contour, and center point height difference;

[0011] Based on the relative position information, determine at least two first materials to be grabbed simultaneously and a second material to be grabbed separately from all materials in the highest material layer.

[0012] According to another aspect of the present invention, a robot control device is provided, the device comprising:

[0013] The point cloud data acquisition module is used to acquire the 3D point cloud data of each material in the highest material layer of the material stack.

[0014] The material determination module is used to determine, based on the 3D material point cloud data, at least two first materials to be grasped simultaneously and a second material to be grasped separately from all materials in the highest material layer.

[0015] The location information determination module is used to determine the location information of the first center point of the at least two first materials as a whole and the location information of the second center point of the second material.

[0016] The material grasping module is used to control the robot to simultaneously grasp at least two first materials based on the first center point position information according to a preset grasping order, and to control the robot to grasp the second material individually based on the second center point position information.

[0017] The material determination module includes:

[0018] The relative position information determination unit is used to determine the relative position information between each material based on the 3D material point cloud data; wherein, the relative position information includes contour parallelism, left and right gap of contour, front and back tolerance of contour, and center point height difference;

[0019] The material determination unit is used to determine, based on the relative position information, at least two first materials to be grasped simultaneously and a second material to be grasped separately from all materials in the highest material layer.

[0020] According to another aspect of the present invention, a control device is provided, the control device comprising:

[0021] At least one processor; and

[0022] A memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the robot control method according to any embodiment of the present invention.

[0024] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the robot control method according to any embodiment of the present invention.

[0025] The technical solution of this invention involves acquiring 3D point cloud data of each material in the highest material layer of a material stack; determining, based on the 3D point cloud data, at least two first materials to be simultaneously grasped and a second material to be grasped separately from all materials in the highest material layer; determining the first center point position information of the at least two first materials as a whole and the second center point position information of the second material; and controlling the robot to simultaneously grasp at least two first materials based on the first center point position information, and controlling the robot to grasp the second material separately based on the second center point position information, according to a preset grasping order. This technical solution utilizes a 3D camera to acquire a large amount of material point cloud data and determines the materials that can be grasped simultaneously based on the point cloud data, enabling the simultaneous grasping of multiple materials and improving equipment production efficiency.

[0026] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of a robot control method provided according to Embodiment 1 of the present invention;

[0029] Figure 2a This is a schematic diagram of the contour parallelism between materials according to Embodiment 1 of the present invention;

[0030] Figure 2b This is a schematic diagram of the left and right gaps between materials according to Embodiment 1 of the present invention;

[0031] Figure 2c This is a schematic diagram of the front and back profile tolerances between materials according to Embodiment 1 of the present invention;

[0032] Figure 2d This is a schematic diagram of the height difference between the center points of materials according to Embodiment 1 of the present invention;

[0033] Figure 3 This is a flowchart of classifying all materials in the highest material layer according to Embodiment 1 of the present invention;

[0034] Figure 4This is a schematic diagram showing the center point positions of the first material and the second material according to Embodiment 1 of the present invention;

[0035] Figure 5 This is a flowchart of a robot control method provided according to Embodiment 2 of the present invention;

[0036] Figure 6 This is a flowchart of a first material grasping process provided according to Embodiment 2 of the present invention;

[0037] Figure 7 This is a flowchart of a second material grabbing process provided according to Embodiment 2 of the present invention;

[0038] Figure 8 This is a schematic diagram of the structure of a robot control device according to Embodiment 3 of the present invention;

[0039] Figure 9 This is a schematic diagram of the structure of the control device for implementing the robot control method of this invention. Detailed Implementation

[0040] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0041] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0042] Example 1

[0043] Figure 1The flowchart illustrates a robot control method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations involving robot control. The method can be executed by a robot control device, which can be implemented in hardware and / or software and can be configured within a control device. Figure 1 As shown, the method includes:

[0044] S110. Obtain the 3D point cloud data of each material in the highest material layer of the material stack.

[0045] Material pallets, also known as cargo pallets, cargo trays, or shelf panels, are platform structures used for transporting, storing, and stacking goods. They are made of sturdy materials such as wood, plastic, or metal, and typically have a square or rectangular shape.

[0046] Point cloud data is a data format representing a discrete set of points in three-dimensional space. It consists of a large number of 3D points and is used to describe the geometry and appearance of objects or scenes. The characteristics of point cloud data include: 1. Three-dimensional coordinates: Each point has position coordinates in three-dimensional space, usually represented using a Cartesian coordinate system (X, Y, Z); 2. Attribute information: In addition to position information, each point may contain other attribute information, such as color, normal vector, reflectivity, etc. These attributes can provide more descriptions of the scene or object; 3. Sparsity: Point cloud data is typically sparse, meaning that not every point is measured or captured. Therefore, when processing point cloud data, it is necessary to consider how to fill or interpolate to obtain a complete representation. Point cloud data is usually acquired through methods such as 3D scanning, LiDAR, or depth cameras.

[0047] In this embodiment of the invention, 3D point cloud data of each material in the highest material layer of a material stack can be acquired using a 3D camera.

[0048] S120. Based on the 3D material point cloud data, determine at least two first materials to be grabbed simultaneously and a second material to be grabbed separately from all materials in the highest material layer.

[0049] It is understandable that due to factors such as unstable stacking of materials and vibrations and impacts during transportation, the positions of materials in a material layer may be scattered and disordered. Compared to existing technologies where the robot has a single gripper / nozzle and only needs to calculate the independent pose of each material, the robot in this embodiment of the invention has multiple grippers / nozzles and needs to simultaneously grasp at least two first materials. Therefore, it is necessary to determine, based on 3D material point cloud data, at least two first materials that can be grasped simultaneously and a second material that can be grasped individually from all materials in the highest material layer.

[0050] Specifically, based on 3D material point cloud data, at least two first materials to be simultaneously grasped and a second material to be grasped separately are determined from all materials in the highest material layer. This includes: determining the relative position information between each material based on the 3D material point cloud data; wherein the relative position information includes contour parallelism, left-right gap of the contour, front-back tolerance of the contour, and center point height difference; and determining at least two first materials to be simultaneously grasped and a second material to be grasped separately from all materials in the highest material layer based on the relative position information.

[0051] In this embodiment of the invention, when at least two first materials are grasped simultaneously, the relative positional relationship between the at least two grippers / nozzles is fixed. Therefore, to ensure that at least two first materials can be grasped smoothly and simultaneously, it is necessary to first determine the relative positional information between each material, and then, based on the relative positional information between each material, determine the at least two first materials that can be grasped simultaneously and the second material that can only be grasped individually. The relative positional information includes contour parallelism, left-right gap of the contour, front-back tolerance of the contour, and center point height difference.

[0052] Specifically, based on relative position information, from all materials in the highest material layer, at least two first materials to be grasped simultaneously and a second material to be grasped separately are determined, including: selecting at least two materials in the highest material layer whose relative position information meets preset relative position conditions as the first materials to be grasped simultaneously, and selecting materials whose relative position information does not meet the preset relative position conditions as the second materials to be grasped separately; wherein, the preset relative position conditions are to simultaneously meet the following four conditions: first, the parallelism of the contour is less than a preset parallelism threshold; second, the left-right gap of the contour is less than a preset gap threshold; third, the front-back tolerance of the contour is less than a preset tolerance threshold; fourth, the height difference of the center point is less than a preset height difference threshold.

[0053] Here, contour parallelism refers to the angle between the inner contours of two materials, and the preset parallelism threshold is the critical value of the contour angle that satisfies the condition of simultaneous grasping. For example, Figure 2a A schematic diagram showing the parallelism of the material profiles is shown, such as... Figure 2a As shown, the angle between the inner contours of material 1 and material 2 is a, and the angle between the inner contours of material 3 and material 4 is b. The preset parallelism threshold can be set to c, where a < c < b. Then, material 1 and 2 can meet the condition of simultaneous grasping, while material 3 and 4 cannot meet the condition of simultaneous grasping.

[0054] The left and right clearance of the outline refers to the distance between two materials, and the preset clearance threshold is the critical value of the clearance distance that satisfies the condition of simultaneous gripping. For example, Figure 2b A schematic diagram showing the left and right gaps between the materials is shown, such as... Figure 2bAs shown, the gap distance between material 5 and material 6 is d, and the gap distance between material 7 and material 8 is e. The preset gap threshold can be set to f, where d < f < e. Then, material 5 and 6 can meet the condition of simultaneous grasping, while material 7 and 8 cannot meet the condition of simultaneous grasping.

[0055] Contour tolerance refers to the front-to-back tolerance between two materials, and the preset tolerance threshold is the critical value of the front-to-back tolerance that satisfies the simultaneous grasping conditions. For example, Figure 2c A schematic diagram showing the front and back tolerances of the material profiles is provided, such as... Figure 2c As shown, the tolerance between material 9 and material 10 is g, and the tolerance between material 11 and material 12 is h. The preset tolerance threshold can be set to i, where g < i < h. Then, material 9 and 10 can meet the condition of simultaneous grasping, while material 11 and 12 cannot meet the condition of simultaneous grasping.

[0056] The center point height difference refers to the height difference between two materials, and the preset height difference threshold is the critical value of the height difference that satisfies the condition of simultaneous grasping. For example, Figure 2d A schematic diagram showing the height difference between the center points of the materials is shown, such as... Figure 2d As shown, the height difference between material 13 and material 14 is j, and the height difference between material 15 and material 16 is k. The preset height difference threshold can be set to l, where j < l < k. Then, material 13 and 14 can meet the condition of simultaneous grasping, while material 15 and 16 cannot meet the condition of simultaneous grasping.

[0057] The preset parallelism threshold, preset gap threshold, preset tolerance threshold, and preset height difference threshold can be set according to actual conditions. In this embodiment of the invention, the relative position information between at least two materials is considered to satisfy the preset relative position conditions only when the relative position information between at least two materials simultaneously meets the above four conditions; if any one of the above four conditions is not met, the relative position information between at least two materials is considered not to satisfy the preset relative position conditions. This ensures a high success rate when simultaneously grasping materials.

[0058] It is understandable that the first material can be two or more materials, while the second material is a single material. If all the materials in the entire layer can be divided into first materials that can be grasped simultaneously, then there is no second material that can be grasped separately. If not all the materials in the entire layer can be divided into first materials that can be grasped simultaneously, then there is a second material that can be grasped separately. Optionally, when classifying materials, the relative position information between all adjacent materials of the selected material and the selected material can be determined. Materials whose relative position information meets the preset relative position conditions are designated as first materials, and materials that do not meet the preset relative position conditions are designated as second materials. Alternatively, the relative position information between one of the adjacent materials of the selected material can be determined. If the relative position information between two materials meets the preset relative position conditions, the two materials are treated as a whole, and then the relative position information between them and a third adjacent material is determined. This process continues until the first material that meets the preset relative position conditions and the second material that does not meet the preset relative position conditions are identified.

[0059] For example, Figure 3 A flowchart illustrating the classification of all materials in the highest material layer is shown, such as... Figure 3 As shown, after obtaining the 3D material point cloud data of each material in the highest material layer of the material stack, the parallelism of the contour, the left and right gap of the contour, the front and back tolerance of the contour, and the height difference of the center point between two adjacent materials are judged to ensure that they can be grasped simultaneously. If all of them are satisfied, they are determined to be the first material to be grasped simultaneously. If one of them is not satisfied, it is determined to be the second material to be grasped separately. This process continues until all materials in the highest material layer are classified.

[0060] S130. Determine the first center point location information of at least two first materials as a whole and the second center point location information of the second material.

[0061] In this embodiment of the invention, at least two first materials are grasped simultaneously. Therefore, the first center point position information can be determined by treating the at least two first materials as a whole. However, when grasping a second material individually, the second center point position information needs to be determined for each individual material. The first center point information can be the coordinates of the common center point of the at least two first materials, and the second center point position information can be the coordinates of the center point of a single second material. For example, the coordinates of the common center point of the at least two first materials can be determined based on the 3D point cloud data of each of the at least two first materials. The center point coordinates of the second material can be determined based on the 3D point cloud data of the second material.

[0062] For example, Figure 4 A schematic diagram showing the center point positions of the first and second materials is provided, as follows: Figure 4As shown, materials 9 and 10 are the first materials, and materials 11 and 12 are the second materials. When determining the center point information, materials 9 and 10 are treated as a whole, and the coordinates of the common center point of the two materials are calculated, while the coordinates of the center point of materials 11 and 12 are calculated separately.

[0063] S140. According to the preset grasping sequence, control the robot to grasp at least two first materials simultaneously based on the first center point position information, and control the robot to grasp the second material individually based on the second center point position information.

[0064] In this embodiment of the invention, after determining at least two first materials to be grasped simultaneously and a second material to be grasped separately, as well as the first center point position information of at least two first materials and the second center point position information of the second material, the robot can be controlled to grasp at least two first materials simultaneously based on the first center point position information and to grasp the second material separately based on the second center point position information, according to a preset grasping order.

[0065] Specifically, according to a preset sequence, the robot is controlled to simultaneously grasp at least two first materials based on the first center point position information, and the robot is controlled to individually grasp a second material based on the second center point position information. This includes: converting the first center point position information into first pose information of the robot's various first grasping devices as the center point of the whole, and converting the second center point position information into second pose information of the robot's second grasping device; wherein, the first grasping device is a grasping device in the robot that simultaneously grasps at least two first materials, and the first grasping device corresponds one-to-one with the first material; the second grasping device is any grasping device of the robot; according to a preset sequence, the robot is controlled to simultaneously grasp at least two first materials based on the first pose information through the various first grasping devices, and the robot is controlled to individually grasp a second material based on the second pose information through the second grasping device.

[0066] Understandably, the gripping device in the robot that simultaneously grasps at least two first materials is designated as the first gripping device, where each first gripping device corresponds one-to-one with a first material; the gripping device in the robot used to grasp a second material is designated as the second gripping device, where the second gripping device is any one of all gripping devices in the robot. Optionally, the gripping device includes a gripper or a suction nozzle. Since both the first and second center point position information are positional information of the material relative to the 3D material point cloud data acquisition device (e.g., a 3D camera), the first and second center point position information are both positional information in the camera coordinate system. Therefore, when controlling the robot to grasp materials, it is necessary to convert the first center point position information into the first pose information of the robot's various first gripping devices as the overall center point, and the second gripping device, which is any one of the robot's gripping devices, into the second pose information of the robot's second gripping device. It is understandable that the first pose information is the pose information of each first gripping device as a whole relative to the first material, and the second pose information is the pose information of the second gripping device relative to the second material.

[0067] After determining the first and second pose information of the first and second gripping devices, the robot is controlled to simultaneously grasp at least two first materials using each of the first gripping devices according to a preset sequence, based on the first pose information, and the robot is also controlled to individually grasp the second material using the second gripping device based on the second pose information.

[0068] This invention involves acquiring 3D point cloud data of all materials in the highest material layer of a material stack; determining, based on the 3D point cloud data, at least two first materials to be simultaneously grasped and a second material to be grasped individually from all materials in the highest material layer; determining the first center point position information of the at least two first materials as a whole and the second center point position information of the second material; and controlling the robot to simultaneously grasp at least two first materials based on the first center point position information, and controlling the robot to individually grasp the second material based on the second center point position information, according to a preset grasping order. This technical solution utilizes a 3D camera to acquire a large amount of material point cloud data and determines materials that can be grasped simultaneously based on the point cloud data, enabling the simultaneous grasping of multiple materials and improving equipment production efficiency.

[0069] Example 2

[0070] Figure 5 This is a flowchart of a robot control method provided in Embodiment 2 of the present invention. The embodiments of the present invention are optimized based on the above embodiments. Solutions not described in detail in the embodiments of the present invention can be found in the above embodiments. Figure 5 As shown, the method includes:

[0071] S210. Obtain the 3D point cloud data of each material in the highest material layer of the material stack.

[0072] S220. Based on the 3D material point cloud data, determine at least two first materials to be grabbed simultaneously and a second material to be grabbed separately from all materials in the highest material layer.

[0073] S230. Determine the first center point location information of at least two first materials as a whole and the second center point location information of the second material.

[0074] S240. The first center point position information is converted into the first pose information of the robot's various first gripping devices as the center point of the whole, and the second center point position information is converted into the second pose information of the robot's second gripping device.

[0075] S250. Based on the first pose information, determine the first motion trajectory of each first gripping device as a whole. When it is determined that each first gripping device meets the motion feasibility conditions according to the first motion trajectory, control the robot to simultaneously grip at least two first materials through each first gripping device based on the first pose information.

[0076] In practical applications, uncertainties can interfere with the grasping process. Therefore, after determining the first motion trajectory of each grasping device as a whole based on the first pose information, it is necessary to determine whether each grasping device meets the motion feasibility conditions based on the first motion trajectory. Only when each grasping device meets the motion feasibility conditions can the robot be controlled to simultaneously grasp at least two first materials based on the first pose information, ensuring the safety of equipment production. If any grasping device does not meet the motion feasibility conditions, an alarm is triggered.

[0077] Specifically, the motion feasibility conditions for each first gripping device are determined based on the first motion trajectory, including: each first gripping device is determined to meet the motion feasibility conditions when the following three conditions are met simultaneously: First, it is determined based on the first motion trajectory that each first gripping device will not collide with the preset restriction area; Second, it is determined based on the first motion trajectory that each first gripping device can reach the location area of ​​the corresponding first material; Third, there are no singularities in the first motion trajectory, wherein a singularity is a trajectory point that prevents any first gripping device from moving.

[0078] The preset restricted area can be defined according to the fixed range of each incoming material pallet to prevent the robot's gripping device from colliding with other equipment.

[0079] In this embodiment of the invention, a first gripping device is deemed to meet the motion feasibility condition only when three conditions are simultaneously met: the first gripping device is determined not to collide with a preset restricted area based on the first motion trajectory; the first gripping device is determined to be able to reach the location area of ​​the corresponding first material based on the first motion trajectory; and there are no singularities in the first motion trajectory. Otherwise, an alarm is triggered. This improves the safety of equipment production. Furthermore, compared to traditional technologies where the robot needs to be unable to reach the target material after movement to trigger an insufficient reach alarm, the solution provided by this embodiment reduces the time cost of equipment alarm downtime maintenance.

[0080] For example, Figure 6 A flowchart illustrating the grabbing of a first material is shown, such as... Figure 6 As shown, after determining the first motion trajectory of each first gripping device as a whole based on the first pose information, the system sequentially performs the following checks: determining that each first gripping device will not collide with the preset restricted area based on the first motion trajectory; determining that each first gripping device can reach the location area of ​​the corresponding first material based on the first motion trajectory; and determining that there are no singular points in the first motion trajectory. Only when all three conditions are met will the robot be controlled to simultaneously grip at least two first materials based on the first pose information and using each first gripping device. If any condition is not met, an alarm will be triggered.

[0081] S260. Determine the second motion trajectory of the second gripping device based on the second pose information. When it is determined from the second motion trajectory that the second gripping device meets the motion feasibility conditions, control the robot to grip the second material separately using the second gripping device based on the second pose information.

[0082] Similarly, after determining the second motion trajectory of the second gripping device based on the second pose information, it is also necessary to determine whether the second gripping device meets the motion feasibility conditions based on the second motion trajectory. Only when the second gripping device meets the motion feasibility conditions can the robot be controlled to grasp the second material independently based on the second pose information. It is understandable that the motion feasibility conditions for the second gripping device are the same as those for the first gripping device. However, when the second gripping device does not meet the motion feasibility conditions, the handling method differs from that for the first gripping device.

[0083] Specifically, when it is determined that the second gripping device does not meet the motion feasibility conditions based on the second motion trajectory, the robot polls each of the third gripping devices, converts the second center point position information into the third pose information of the current third gripping device, and determines the third motion trajectory of the current third gripping device based on the third pose information. Based on the third motion trajectory, it judges whether the current third gripping device meets the motion feasibility conditions, until a fourth gripping device that meets the motion feasibility conditions is determined from each of the third gripping devices. The robot is then controlled to grip the second material individually using the fourth gripping device based on the third pose information.

[0084] It is understood that the second gripping device for individually grasping the second material can be any gripping device of the robot. Therefore, when it is determined from the second motion trajectory that the second gripping device does not meet the motion feasibility conditions, the robot polls all other gripping devices until a gripping device that meets the motion feasibility conditions, i.e., the fourth gripping device, is identified. Optionally, if none of the robot's other gripping devices meet the motion feasibility conditions, an alarm is triggered, and the material on the current layer where the problem occurs is marked.

[0085] For example, Figure 7 A flowchart illustrating the grasping process for a second material is shown, such as... Figure 7 As shown, after determining the second motion trajectory of the second gripping device based on the second pose information, the feasibility of the second gripping device's motion is assessed. If the second gripping device meets the motion feasibility requirements, the robot controls the second gripping device to individually grasp the second material based on the second pose information. Otherwise, other gripping devices are polled, their motion trajectories are determined based on the second pose information, and their motion feasibility is reassessed to identify the gripping device that meets the motion feasibility conditions. The robot then uses this gripping device to individually grasp the second material. If none of the robot's other gripping devices meet the motion feasibility conditions, an alarm is triggered.

[0086] It should be noted that, in this embodiment of the invention, in order to improve the efficiency of robot loading, before controlling the robot to simultaneously or individually grasp materials through various gripping devices based on pose information, the classification of all materials in the highest material layer and the feasibility judgment of each gripping device should be completed. Then, according to a preset order, the robot is controlled to simultaneously or individually grasp materials through various gripping devices based on pose information.

[0087] In this embodiment of the invention, 3D material point cloud data of each material in the highest material layer of a material stack is obtained; based on the 3D material point cloud data, at least two first materials to be grasped simultaneously and a second material to be grasped individually are determined from all materials in the highest material layer; the first center point position information of the at least two first materials as a whole and the second center point position information of the second material are determined; the first center point position information is converted into the first pose information of the robot's first grasping devices as the center point of the whole, and the second center point position information is converted into the second pose information of the robot's second grasping device; based on the first pose information, the first motion trajectory of each first grasping device as a whole is determined; when the first motion trajectory determines that each first grasping device meets the motion feasibility conditions, the robot is controlled to grasp at least two first materials simultaneously using the first pose information; based on the second pose information, the second motion trajectory of the second grasping device is determined; when the second motion trajectory determines that the second grasping device meets the motion feasibility conditions, the robot is controlled to grasp the second material individually using the second grasping device based on the second pose information. The technical solution of this invention uses a 3D camera to acquire a large amount of material point cloud data, and determines the materials that can be grasped simultaneously based on the point cloud data. This enables the simultaneous grasping of multiple materials, improving equipment production efficiency. Simultaneously, it enhances equipment production safety and reduces the time and cost of equipment alarm downtime maintenance.

[0088] Example 3

[0089] Figure 8 This is a schematic diagram of a robot control device provided in Embodiment 3 of the present invention. Figure 8 As shown, the device includes:

[0090] The point cloud data acquisition module 310 is used to acquire the 3D point cloud data of each material in the highest material layer of the material stack.

[0091] The material determination module 320 is used to determine, based on the 3D material point cloud data, at least two first materials to be grasped simultaneously and a second material to be grasped separately from all materials in the highest material layer.

[0092] The location information determination module 330 is used to determine the location information of the first center point of the at least two first materials as a whole and the location information of the second center point of the second material.

[0093] The material gripping module 340 is used to control the robot to simultaneously grip at least two first materials based on the first center point position information according to a preset gripping order, and to control the robot to grip the second material individually based on the second center point position information.

[0094] Optionally, the material determination module 320 includes:

[0095] The relative position information determination unit is used to determine the relative position information between each material based on the 3D material point cloud data; wherein, the relative position information includes contour parallelism, left and right gap of contour, front and back tolerance of contour, and center point height difference;

[0096] The material determination unit is used to determine, based on the relative position information, at least two first materials to be grasped simultaneously and a second material to be grasped separately from all materials in the highest material layer.

[0097] Optionally, the material determination unit is specifically used for:

[0098] Among all the materials in the highest material layer, at least two materials whose relative position information satisfies the preset relative position condition are taken as the first materials to be grasped simultaneously, and materials whose relative position information does not satisfy the preset relative position condition are taken as the second materials to be grasped separately.

[0099] The preset relative position condition is to simultaneously satisfy the following four conditions:

[0100] First, the parallelism of the contour is less than the preset parallelism threshold;

[0101] Second, the left and right gaps of the contour are less than the preset gap threshold;

[0102] Third, the front and rear tolerances of the profile are less than the preset tolerance threshold;

[0103] Fourth, the height difference between the center points is less than the preset height difference threshold.

[0104] Optionally, the material gripping module 340 includes:

[0105] The pose information determination unit is used to convert the first center point position information into first pose information of each of the robot's first gripping devices as the center point of the whole, and to convert the second center point position information into second pose information of the robot's second gripping device; wherein, the first gripping device is a gripping device in the robot that simultaneously grips at least two first materials, and the first gripping device corresponds one-to-one with the first materials; the second gripping device is any gripping device of the robot.

[0106] The material gripping unit is used to control the robot to simultaneously grip at least two first materials based on the first pose information using the first gripping devices, and to control the robot to grip the second material individually using the second gripping device based on the second pose information.

[0107] Optionally, the device further includes:

[0108] The first motion trajectory determination module is used to determine the first motion trajectory of each first grasping device as a whole based on the first pose information.

[0109] The first motion feasibility determination module is used to determine whether each of the first grasping devices meets the motion feasibility conditions based on the first motion trajectory.

[0110] The material gripping unit includes:

[0111] The first material grasping subunit is used to control the robot to simultaneously grasp at least two first materials based on the first pose information when it is determined from the first motion trajectory that each of the first grasping devices meets the motion feasibility conditions.

[0112] Optionally, the motion feasibility determination module is specifically used for:

[0113] The first grasping device is deemed to meet the motion feasibility condition when all three of the following conditions are met simultaneously:

[0114] First, when it is determined from the first motion trajectory that each of the first grasping devices will not collide with the preset restriction area;

[0115] Second, when it is determined from the first motion trajectory that each of the first gripping devices can reach the corresponding location area of ​​the first material;

[0116] Third, there are no singular points in the first motion trajectory, wherein the singular point is a trajectory point that prevents any of the first grasping devices from moving.

[0117] Optionally, the device further includes:

[0118] The second motion trajectory determination module is used to determine the second motion trajectory of the second grasping device based on the second pose information.

[0119] The second motion feasibility determination module is used to determine whether the second grasping device meets the motion feasibility conditions based on the second motion trajectory.

[0120] The grasping unit includes:

[0121] The second material grasping subunit is used to control the robot to grasp the second material individually using the second grasping device based on the second pose information when it is determined from the second motion trajectory that the second grasping device meets the motion feasibility conditions.

[0122] The grasping unit further includes:

[0123] The fourth grasping device determination subunit is used to poll each of the robot's third grasping devices when it is determined from the second motion trajectory that the second grasping device does not meet the motion feasibility conditions, convert the second center point position information into the third pose information of the current third grasping device, determine the third motion trajectory of the current third grasping device based on the third pose information, and determine whether the current third grasping device meets the motion feasibility conditions based on the third motion trajectory, until a fourth grasping device that meets the motion feasibility conditions is determined from each of the third grasping devices.

[0124] The fourth material grasping subunit is used to control the robot to grasp the second material individually through the fourth grasping device based on the third pose information.

[0125] Optionally, the device further includes:

[0126] An alarm module is used to trigger an alarm when the aforementioned conditions for motion feasibility are not met.

[0127] The robot control device provided in the embodiments of the present invention can execute the robot control method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0128] Example 4

[0129] Figure 9 A schematic diagram of a control device 10, which can be used to implement embodiments of the present invention, is shown. The control device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The control device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0130] like Figure 9As shown, the control device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer programs stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the control device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0131] Multiple components in the control device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, optical disk, etc.; and a communication unit 19, such as a network card, modem, wireless transceiver, etc. The communication unit 19 allows the control device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0132] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as robot control methods.

[0133] In some embodiments, the robot control method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the control device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the robot control method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the robot control method by any other suitable means (e.g., by means of firmware).

[0134] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0136] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0137] To provide interaction with the user, the systems and techniques described herein can be implemented on a control device having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the control device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0138] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0139] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0140] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0141] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A robot control method characterized by, include: Obtain 3D point cloud data of each material in the highest material layer of the material stack; Based on the 3D material point cloud data, determine at least two first materials to be grasped simultaneously and a second material to be grasped separately from all materials in the highest material layer; Determine the first center point location information of the at least two first materials as a whole and the second center point location information of the second material; According to the preset grasping order, the robot is controlled to simultaneously grasp at least two first materials based on the first center point position information, and the robot is controlled to grasp the second material individually based on the second center point position information. Based on the 3D material point cloud data, at least two first materials to be simultaneously grasped and a second material to be grasped separately are determined from all materials in the highest material layer, including: The relative position information between each material is determined based on the 3D material point cloud data; wherein, the relative position information includes contour parallelism, left and right gap of contour, front and back tolerance of contour, and center point height difference; Based on the relative position information, determine at least two first materials to be grabbed simultaneously and a second material to be grabbed separately from all materials in the highest material layer; Following a preset sequence, the robot is controlled to simultaneously grasp at least two first materials based on the first center point position information, and the robot is also controlled to individually grasp the second material based on the second center point position information, including: The first center point position information is converted into the first pose information of the robot's various first gripping devices as the overall center point, and the second center point position information is converted into the second pose information of the robot's second gripping device; wherein, the first gripping device is a gripping device in the robot that simultaneously grips at least two first materials, and the first gripping device corresponds one-to-one with the first materials; the second gripping device is any gripping device of the robot. According to a preset sequence, the robot is controlled to simultaneously grasp at least two first materials using the first grasping devices based on the first pose information, and the robot is also controlled to individually grasp the second material using the second grasping device based on the second pose information. Before controlling the robot to individually grasp the second material using the second gripping device based on the second pose information, the method further includes: The second motion trajectory of the second grasping device is determined based on the second pose information; Determine whether the second grasping device meets the motion feasibility conditions based on the second motion trajectory; Controlling the robot to individually grasp the second material using the second grasping device based on the second pose information includes: When it is determined from the second motion trajectory that the second gripping device meets the motion feasibility conditions, the robot is controlled to grip the second material individually using the second gripping device based on the second pose information. When it is determined that the second grasping device does not meet the motion feasibility conditions based on the second motion trajectory, the robot polls each of the third grasping devices, converts the second center point position information into the third pose information of the current third grasping device, and determines the third motion trajectory of the current third grasping device based on the third pose information. Based on the third motion trajectory, it is determined whether the current third grasping device meets the motion feasibility conditions, until a fourth grasping device that meets the motion feasibility conditions is determined from each of the third grasping devices. The robot is controlled to individually grasp the second material using the fourth grasping device based on the third pose information.

2. The method of claim 1, wherein, Based on the relative position information, at least two first materials to be simultaneously grasped and a second material to be grasped separately are determined from all materials in the highest material layer, including: Among all the materials in the highest material layer, at least two materials whose relative position information satisfies the preset relative position condition are taken as the first materials to be grasped simultaneously, and materials whose relative position information does not satisfy the preset relative position condition are taken as the second materials to be grasped separately. The preset relative position condition is to simultaneously satisfy the following four conditions: First, the parallelism of the contour is less than the preset parallelism threshold; Second, the left and right gaps of the contour are less than the preset gap threshold; Third, the front and rear tolerances of the profile are less than the preset tolerance threshold; Fourth, the height difference between the center points is less than the preset height difference threshold.

3. The method of claim 1, wherein, Before controlling the robot to simultaneously grasp the at least two first materials using the respective first grasping devices based on the first pose information, the method further includes: Based on the first pose information, the first motion trajectory of each first grasping device as a whole is determined. Based on the first motion trajectory, determine whether each of the first grasping devices meets the motion feasibility conditions; Controlling the robot to simultaneously grasp the at least two first materials based on the first pose information using the first grasping device includes: When it is determined from the first motion trajectory that each of the first grasping devices meets the motion feasibility conditions, the robot is controlled to grasp the at least two first materials simultaneously through the first grasping devices based on the first pose information.

4. The method according to claim 3, characterized in that, Determining that each of the first grasping devices meets the motion feasibility conditions based on the first motion trajectory includes: The first grasping device is deemed to meet the motion feasibility condition when all three of the following conditions are met simultaneously: When it is determined from the first motion trajectory that each of the first grasping devices will not collide with the preset restriction area; When it is determined from the first motion trajectory that each of the first gripping devices can reach the corresponding location area of ​​the first material; There are no singular points in the first motion trajectory, wherein the singular point is a trajectory point that prevents any of the first grasping devices from moving.

5. The method according to any one of claims 3-4, characterized in that, Also includes: An alarm will be triggered if the aforementioned conditions for motion feasibility are not met.

6. A robot control device, characterized in that, include: The point cloud data acquisition module is used to acquire the 3D point cloud data of each material in the highest material layer of the material stack. The material determination module is used to determine, based on the 3D material point cloud data, at least two first materials to be grasped simultaneously and a second material to be grasped separately from all materials in the highest material layer. The location information determination module is used to determine the location information of the first center point of the at least two first materials as a whole and the location information of the second center point of the second material. The material grasping module is used to control the robot to simultaneously grasp at least two first materials based on the first center point position information according to a preset grasping order, and to control the robot to grasp the second material individually based on the second center point position information. The material determination module includes: The relative position information determination unit is used to determine the relative position information between each material based on the 3D material point cloud data; wherein, the relative position information includes contour parallelism, left and right gap of contour, front and back tolerance of contour, and center point height difference; The material determination unit is used to determine, based on the relative position information, at least two first materials to be grasped simultaneously and a second material to be grasped separately from all materials in the highest material layer; The material grasping module includes: The pose information determination unit is used to convert the first center point position information into first pose information of each of the robot's first gripping devices as the center point of the whole, and to convert the second center point position information into second pose information of the robot's second gripping device; wherein, the first gripping device is a gripping device in the robot that simultaneously grips at least two first materials, and the first gripping device corresponds one-to-one with the first materials; the second gripping device is any gripping device of the robot. The material gripping unit is used to control the robot to simultaneously grip at least two first materials based on the first pose information using the first gripping devices according to a preset order, and to control the robot to grip the second material individually using the second gripping device based on the second pose information. The second motion trajectory determination module is used to determine the second motion trajectory of the second grasping device based on the second pose information. The second motion feasibility determination module is used to determine whether the second grasping device meets the motion feasibility conditions based on the second motion trajectory. The material gripping unit includes: The second material grasping subunit is used to control the robot to grasp the second material individually through the second grasping device based on the second pose information when it is determined from the second motion trajectory that the second grasping device meets the motion feasibility conditions. The fourth grasping device determination subunit is used to poll each of the robot's third grasping devices when it is determined from the second motion trajectory that the second grasping device does not meet the motion feasibility conditions, convert the second center point position information into the third pose information of the current third grasping device, determine the third motion trajectory of the current third grasping device based on the third pose information, and determine whether the current third grasping device meets the motion feasibility conditions based on the third motion trajectory, until a fourth grasping device that meets the motion feasibility conditions is determined from each of the third grasping devices. The fourth material grasping subunit is used to control the robot to grasp the second material individually through the fourth grasping device based on the third pose information.

7. A control device, characterized in that, The control device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the robot control method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the robot control method according to any one of claims 1-5.

Images