A method and apparatus for determining a grasping pose, an electronic device, and a storage medium

Abstract

Embodiments of the present application provide a kind of to grab pose determination method, device, electronic equipment and storage medium, involve robot technical field;Among them, a kind of to grab pose determination method includes: obtaining the target point cloud image obtained by acquisition equipment acquisition, according to predetermined image analysis mode, target point cloud image is analyzed, obtains the multiple corner point coordinates of each target object and corresponding order number, obtains the multiple corner point coordinates of each calibration object and the order number corresponding to each calibration object, identify whether there is object that does not exist in each target object, if there is, then output alarm information, and in response to meet the trigger condition that the placement position has been adjusted, the step of obtaining the target point cloud image obtained by acquisition equipment acquisition is executed, otherwise, determine the pose information of mechanical arm for single time simultaneously grabbing each target object is obtained.It can be seen that the present scheme can guarantee effective grabbing for object.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a method, apparatus, electronic device, and storage medium for determining grasping pose. Background Technology

[0002] As a type of multi-joint collaborative robot, robotic arms are increasingly widely used in manufacturing. They play a vital role in automated production processes in fields such as automobile and auto parts manufacturing, machining, electronics and electrical production, rubber and plastics manufacturing, and food processing.

[0003] In scenarios where a robotic arm uses a single gripper to simultaneously grasp multiple objects, the robotic arm's pose is typically taught beforehand, using the multiple objects as a whole for grasping. During operation, based on the teaching results, the required robotic arm pose for simultaneously grasping multiple objects is determined, and the grasping is performed according to the determined pose. However, during operation, some of the objects may be misplaced, which undoubtedly affects the success rate of the robotic arm grasping multiple objects.

[0004] Currently, there is an urgent need for a method to determine the grasping pose, thereby improving the success rate of simultaneously grasping multiple objects. Summary of the Invention

[0005] The purpose of this application is to provide a method, apparatus, electronic device, and storage medium for determining the grasping pose, so as to ensure effective grasping of multiple objects. The specific technical solution is as follows:

[0006] In a first aspect, embodiments of this application provide a method for determining a grasping pose, the method comprising:

[0007] Acquire a target point cloud image containing point cloud data of each target object, obtained by a data acquisition device; wherein, each target object is an object to be simultaneously grasped by a robotic arm in a single operation.

[0008] According to a predetermined image analysis method, the target point cloud image is analyzed to obtain the coordinates of multiple corner points of each target object and the corresponding sorting number of each target object.

[0009] Based on the coordinates of multiple corner points and their corresponding sorting numbers for each target object, as well as the coordinates of multiple corner points and their corresponding sorting numbers for each calibration object, the system identifies whether any objects in each target object are improperly placed. Each calibration object is a group of objects whose relative positions allow the robotic arm to grasp them simultaneously in a single operation. The corner point coordinates and corresponding sorting numbers of each calibration object are obtained by analyzing the point cloud image containing each calibration object acquired by the acquisition device using a predetermined image analysis method.

[0010] In response to the existence of at least one non-compliant target object placement, an alarm message is output to characterize the non-compliant object placement. In response to the trigger condition that the placement position has been adjusted, a new target point cloud image containing point cloud data of each target object is acquired by the acquisition device, and the step of analyzing the target point cloud image according to a predetermined image analysis method is returned.

[0011] In response to the absence of any non-compliant placement of target objects, the pose information of the robotic arm for simultaneously grasping each target object in a single operation is determined based on the coordinates of multiple corner points of each target object.

[0012] Secondly, embodiments of this application provide a grasping pose determination device, the device comprising:

[0013] The acquisition module is used to acquire a target point cloud image containing point cloud data of each target object, which is obtained by the acquisition device; wherein, each target object is an object that the robotic arm is to grasp simultaneously in a single operation.

[0014] The analysis module is used to analyze the target point cloud image according to a predetermined image analysis method to obtain the coordinates of multiple corner points of each target object and the sorting number corresponding to each target object.

[0015] The identification module is used to identify whether there are any improperly placed objects among the target objects based on the coordinates of multiple corner points and their corresponding sorting numbers of each target object, as well as the coordinates of multiple corner points and their corresponding sorting numbers of each calibration object. Each calibration object is a group of objects whose relative positions allow the robotic arm to grasp them simultaneously in a single operation. The corner point coordinates and corresponding sorting numbers of each calibration object are obtained by analyzing the point cloud image containing each calibration object acquired by the acquisition device based on the predetermined image analysis method.

[0016] The output module is used to output alarm information to characterize the non-compliant placement of objects in response to the existence of at least one non-compliant target object, and to acquire a new target point cloud image containing point cloud data of each target object collected by the acquisition device in response to the trigger condition that the placement position has been adjusted, and to return the step of analyzing the target point cloud image according to the predetermined image analysis method.

[0017] The first determining module is used to determine the pose information of the robotic arm for simultaneously grasping each target object in a single operation, based on the coordinates of multiple corner points of each target object, in response to the absence of any non-compliant placement of any target object.

[0018] Thirdly, embodiments of this application provide an electronic device, including:

[0019] Memory, used to store computer programs;

[0020] The processor, when executing a program stored in memory, implements any of the above-mentioned grasping pose determination methods.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described grasping pose determination methods.

[0022] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements any of the above-described grasping pose determination methods.

[0023] Beneficial effects of the embodiments in this application:

[0024] The grasping pose determination method provided in this application embodiment can first acquire a target point cloud image containing point cloud data of each target object obtained by a data acquisition device, and then analyze the target point cloud image according to a predetermined image analysis method to obtain multiple corner coordinates of each target object and the corresponding sorting number of each target object. Based on the acquired multiple corner coordinates and corresponding sorting number of each target object, as well as multiple corner coordinates and corresponding sorting number of each calibration object, it can identify whether there are any improperly placed objects among the target objects. If at least one target object is improperly placed, an alarm message indicating improper object placement is output. When the placement position is detected to have been adjusted, a new target point cloud image containing point cloud data of each target object is acquired again by the data acquisition device, and the step of analyzing the target point cloud image according to the predetermined image analysis method is returned. If no target object is improperly placed, the pose information of the robotic arm for simultaneously grasping each target object in a single operation is determined based on the multiple corner coordinates of each target object.

[0025] As can be seen, the embodiments of this application can analyze each target object based on a target point cloud image containing point cloud data of each target object, thereby obtaining the coordinates of multiple corner points of each target object and the corresponding sorting number of each target object, instead of analyzing the point cloud data of each target object as a whole. In this way, when identifying whether there are any improperly placed objects among the target objects, identification can be performed on a per-target-object basis. If no object is improperly placed, it indicates that the placement of each target object is compliant, so the accurate pose to be grasped can be directly determined, ensuring effective grasping of multiple objects. Furthermore, if at least one object is improperly placed, an alarm message can be output. After the improperly placed position is detected to have been adjusted, the target point cloud image containing point cloud data of each target object, acquired by the acquisition device, is re-acquired until it can be identified that no object is improperly placed, thereby ensuring effective grasping of multiple objects and improving the success rate of simultaneous grasping of multiple objects.

[0026] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0028] Figure 1 A flowchart illustrating a grasping pose determination method provided in an embodiment of this application;

[0029] Figure 2 A top view of a point cloud region provided in an embodiment of this application;

[0030] Figure 3(a) is a schematic diagram of a three-dimensional modeling of an object provided in an embodiment of this application;

[0031] Figure 3(b) is a schematic diagram of corner points for three-dimensional modeling of an object provided in an embodiment of this application;

[0032] Figure 4 This is a schematic diagram illustrating a process for identifying whether an object is placed in accordance with regulations, as provided in an embodiment of this application.

[0033] Figure 5 This is a schematic diagram of the structure of a grasping pose determination device provided in an embodiment of this application;

[0034] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0036] To better understand this solution, a brief introduction to the relevant technologies is provided below:

[0037] In scenarios where a robotic arm uses a gripper to simultaneously grasp multiple objects, the robotic arm's pose is typically taught beforehand, using the multiple objects to be grasped as the overall grasping unit. Specifically, multiple objects are modeled simultaneously to obtain a unified 3D model. Based on this unified 3D model and the robotic arm's taught pose, the teaching result is determined, i.e., the conversion relationship between the pose information of the unified 3D model and the robotic arm's taught pose. Thus, during the operation, based on the teaching result, the required robotic arm pose for simultaneously grasping multiple objects is determined, and the grasping is performed according to the determined pose.

[0038] During the operation, when dealing with multiple objects to be grasped, some objects may be improperly positioned, i.e., not in a standard manner. For example, among five horizontally arranged batteries, the second battery might be misplaced, such as tilted. Current technologies that analyze multiple objects as a whole cannot identify improperly positioned objects, and the determined robotic arm pose may not be able to successfully grasp individual objects. Ultimately, this undoubtedly affects the success rate of the robotic arm in grasping multiple objects.

[0039] Based on the problems existing in related technologies, this application provides a grasping pose determination method to ensure effective grasping of multiple objects.

[0040] The grasping pose determination method provided in the embodiments of this application will be introduced first below.

[0041] The grasping pose determination method provided in this application embodiment can be applied to electronic devices. In specific applications, the electronic device can be a robotic arm or a server for determining the grasping pose. The robotic arm can directly determine the grasping pose, or the server in the background can determine the grasping pose. This application embodiment does not limit the specific form of the electronic device.

[0042] Specifically, the executing entity of this grasping pose determination method can be a grasping pose determination device. For example, when the grasping pose determination method is applied to a terminal device, the grasping pose determination device can be a client running on the terminal device for grasping pose determination. For example, when the grasping pose determination method is applied to a server, the grasping pose determination device can be a computer program running on the server, which can be used to determine the grasping pose.

[0043] It is understood that the application scenarios of this application embodiment can be diverse, such as automated production and processing scenarios in automobile and auto parts manufacturing, machining, electronic and electrical production, rubber and plastic manufacturing, and food processing; and this application can also be applied to production lines in various scenarios. For example, in the battery processing scenario, a robotic arm needs to simultaneously grasp multiple batteries. In this case, the grasping pose determination method can be used to enable the robotic arm to effectively grasp multiple batteries. It is understood that the application scenarios of this application embodiment are not specifically limited.

[0044] In addition, the robotic arm in this application is a multi-joint collaborative robot, and any type of robotic arm on the market can be used as the robotic arm in this application.

[0045] One of the methods for determining the grasping pose includes:

[0046] Acquire a target point cloud image containing point cloud data of each target object, obtained by a data acquisition device; wherein, each target object is an object to be simultaneously grasped by a robotic arm in a single operation.

[0047] According to a predetermined image analysis method, the target point cloud image is analyzed to obtain the coordinates of multiple corner points of each target object and the corresponding sorting number of each target object.

[0048] Based on the coordinates of multiple corner points and their corresponding sorting numbers for each target object, as well as the coordinates of multiple corner points and their corresponding sorting numbers for each calibration object, the system identifies whether any objects in each target object are improperly placed. Each calibration object is a group of objects whose relative positions allow the robotic arm to grasp them simultaneously in a single operation. The corner point coordinates and corresponding sorting numbers of each calibration object are obtained by analyzing the point cloud image containing each calibration object acquired by the acquisition device using a predetermined image analysis method.

[0049] In response to the existence of at least one non-compliant target object placement, an alarm message is output to characterize the non-compliant object placement. In response to the trigger condition that the placement position has been adjusted, a new target point cloud image containing point cloud data of each target object is acquired by the acquisition device, and the step of analyzing the target point cloud image according to a predetermined image analysis method is returned.

[0050] In response to the absence of any non-compliant placement of target objects, the pose information of the robotic arm for simultaneously grasping each target object in a single operation is determined based on the coordinates of multiple corner points of each target object.

[0051] As can be seen, the embodiments of this application can analyze each target object based on a target point cloud image containing point cloud data of each target object, thereby obtaining the coordinates of multiple corner points of each target object and the corresponding sorting number of each target object, instead of analyzing the point cloud data of each target object as a whole. In this way, when identifying whether there are any improperly placed objects among the target objects, identification can be performed on a per-target-object basis. If no object is improperly placed, it indicates that the placement of each target object is compliant, so the accurate pose to be grasped can be directly determined, ensuring effective grasping of multiple objects. Furthermore, if at least one object is improperly placed, an alarm message can be output. After the improperly placed position is detected to have been adjusted, the target point cloud image containing point cloud data of each target object, acquired by the acquisition device, is re-acquired until it can be identified that no object is improperly placed, thereby ensuring effective grasping of multiple objects and improving the success rate of simultaneous grasping of multiple objects.

[0052] The following describes a grasping pose determination method provided by an embodiment of this application, with reference to the accompanying drawings.

[0053] like Figure 1 As shown in the embodiment of this application, a grasping pose determination method is provided, which may include the following steps:

[0054] S101, acquire a target point cloud image containing point cloud data of each target object, obtained by the acquisition device;

[0055] Among them, each target object is an object that the robotic arm is to grasp simultaneously in a single operation;

[0056] The acquisition device can be any camera capable of capturing point cloud images. The acquisition device can capture images of various target objects placed on a workbench or carrying area, with the workbench or carrying area corresponding to the application scenario. For example, in a battery processing scenario, the workbench can be a battery processing production line. It is understood that this application embodiment does not specifically limit its scope.

[0057] It is understood that each target object is an object that the robotic arm is to grasp simultaneously in a single operation, which can be considered as an object that the robotic arm can grasp simultaneously in a single operation; and the target point cloud image acquired by the acquisition device may contain only the point cloud data of each target object, that is, the acquisition device only acquires data for the target objects. Of course, the target point cloud image may also contain the point cloud data of each target object and the point cloud data of other objects. This application embodiment does not specifically limit this.

[0058] S102, according to a predetermined image analysis method, the target point cloud image is analyzed to obtain the coordinates of multiple corner points of each target object and the sorting number corresponding to each target object.

[0059] It is understood that, since this application analyzes each target object individually, the point cloud data of each target object contained in the target point cloud image can be analyzed according to a predetermined image analysis method to obtain the coordinates of multiple corner points of each target object and the corresponding sorting number of each target object. The predetermined image analysis method can be considered as an analysis method used to obtain the corner point coordinates and sort the objects. The specific analysis process can be varied. For example, multiple point cloud regions can be located first, then sorted, and finally the corner point information of each object can be calculated. Alternatively, the point cloud regions can be identified, then the corner point information of the object corresponding to each point cloud region can be calculated, and finally the point cloud regions can be sorted. This application does not specifically limit the predetermined image analysis method. Furthermore, in one implementation, a pre-trained network model can be used, and the target point cloud image can be directly input into the network model to obtain the coordinates of multiple corner points of each target object and the corresponding sorting number of each target object.

[0060] In one implementation, the predetermined image analysis method includes steps A1-A4:

[0061] Step A1: Locate the object region in any point cloud image to be analyzed to obtain multiple point cloud regions;

[0062] Understandably, object regions can be located first for any point cloud image to be analyzed, resulting in multiple point cloud regions. Each point cloud region corresponds to an object. Any point cloud region can be considered as the approximate location of the object in the point cloud image, or as the approximate location of the outlines of various objects in the point cloud image. It is important to emphasize that a point cloud region is a three-dimensional region, representing the approximate area of ​​each object in the point cloud image. The point cloud data for the object is located within the point cloud region. In subsequent steps, based on the point cloud region and according to predetermined object size information, a three-dimensional model of the object can be performed within the point cloud region to obtain the object's point cloud model. Understandably, during the process of analyzing the target point cloud image according to the predetermined image analysis method, this target point cloud image is used as the point cloud image to be analyzed.

[0063] Step A2: Sort the obtained point cloud regions according to the predetermined sorting rules to obtain the sorting number corresponding to each point cloud region.

[0064] It is understandable that after obtaining multiple point cloud regions, the obtained point cloud regions can be sorted according to a predetermined sorting rule to obtain a sorting number corresponding to each point cloud region. The sorting rule can specify sequential sorting using numbers or letters, and can also specify the sorting order, such as sorting from left to right or from top to bottom. This embodiment does not specifically limit this. For example, after locating object regions in a point cloud image, 10 point cloud regions can be obtained. According to the predetermined sorting rule, these 10 point cloud regions can be sorted sequentially from left to right using numbers to obtain a sorting number corresponding to each point cloud region, i.e., 1-10.

[0065] Step A3: According to the selection rules for the objects to be grasped that match the sorting rules, based on the sorting number corresponding to each point cloud region, select the point cloud region and the corresponding sorting number of each object in the point cloud image to be analyzed that is grasped simultaneously by the robotic arm in a single operation.

[0066] It is understandable that the selection rules and sorting rules for the objects to be grasped are matched. When selecting the point cloud region of each object simultaneously grasped by the robotic arm in a single operation from the point cloud image to be analyzed, the sorting rules must be considered. The point cloud region of the object to be grasped is selected based on the sorting number corresponding to each point cloud region. For example, after locating the object region in point cloud image A, 10 point cloud regions can be obtained, arranged in a 2×5 pattern, with two rows. Each row has five point cloud regions, and the sorting rule is to sort them from top to bottom and from left to right according to numbers. The sorting numbers of the five point cloud regions in the first row are 1, 3, 5, 7, and 9, and the sorting numbers of the five point cloud regions in the second row are 2, 4, 6, 8, and 10. If the robotic arm can grasp all the objects in a single row, then the point cloud regions with sorting numbers 1, 3, 5, 7, and 9 can be selected. It is understandable that, in the process of analyzing the target point cloud image according to the predetermined image analysis method, the point cloud region of each object that is simultaneously grasped by the robotic arm in the point cloud image to be analyzed and the corresponding sorting number are selected, which are the point cloud region of each target object and the sorting number of each target object. Subsequently, through the processing in step A4, the corner coordinates of multiple corner points of each target object can be obtained.

[0067] Step A4: Based on the point cloud region of each selected object and the predetermined object size information, calculate the corner coordinates of multiple corner points of each object that is simultaneously grasped by the robotic arm in a single operation.

[0068] Understandably, since the point cloud region is a three-dimensional area, after selecting the point cloud region for each object, the corner coordinates of multiple corner points of each object being simultaneously grasped by the robotic arm can be calculated based on the predetermined object size information. This can be achieved by first modeling the point cloud region for the object, and then calculating the corner coordinates of multiple corner points for each object. Furthermore, objects of different shapes have different numbers of corner points; for example, when any object is rectangular, the corner coordinates of its eight corner points can be calculated. For clarity, a detailed explanation of this part will be provided in subsequent embodiments and will not be elaborated upon here.

[0069] In addition, a point cloud coordinate system is set in the point cloud image, and the coordinates of the corner points are determined based on the point cloud coordinate system.

[0070] In one implementation, locating object regions in any point cloud image to be analyzed to obtain multiple point cloud regions may include step A11:

[0071] Step A11: Based on the point cloud contour library, perform contour localization on any point cloud image to be analyzed to obtain the multiple point cloud regions.

[0072] The point cloud contour library contains point cloud contours of various shapes. Based on the shape of the object, a point cloud contour similar to the object's shape can be selected from the library as the target point cloud contour, thereby performing contour localization on the point cloud image. Therefore, the target point cloud contour can be used to perform contour localization on any point cloud image to be analyzed. Alternatively, it can be considered as marking contours with the same shape as the target point cloud contour in the point cloud image, marking multiple point cloud contours, and using them as point cloud regions.

[0073] Of course, there is no single way to locate point cloud regions. Point cloud recognition methods can also be used to mark relatively dense point clouds, thereby roughly locating the outline of the point cloud and obtaining the point cloud region. Any method for locating point cloud regions can be applied to this application, and this application does not specifically limit the specific method for locating point cloud regions.

[0074] To better understand point cloud regions, a brief explanation will be provided below with reference to the accompanying diagram, such as... Figure 2 As shown:

[0075] Figure 2 The top-down view shows the point cloud regions, which are 10 in total. The point cloud regions are rectangular in the top-down view. The 10 point cloud regions are numbered 1-10. The first row of 5 point cloud regions is numbered 1-5, and the second row of 5 point cloud regions is numbered 6-10.

[0076] S103, based on the coordinates of multiple corner points of each target object and the corresponding sorting number, and the coordinates of multiple corner points of each calibration object and the corresponding sorting number, identify whether there are any objects that are not placed in accordance with regulations among the target objects.

[0077] Among them, each calibration object is a multiple object whose relative positional relationship allows the robotic arm to grasp it simultaneously in a single operation; the corner coordinates and corresponding sorting numbers of each calibration object are obtained by analyzing the point cloud image containing each calibration object acquired by the acquisition device based on the predetermined image analysis method.

[0078] Each of the calibration objects is an object set up during the teaching process. The specific teaching process will be described in subsequent embodiments and will not be repeated here.

[0079] It is important to emphasize that the calibration object and the target object are of the same type and have the same size and specifications. For example, if the calibration object is a type A battery, the target object is also a type A battery. Furthermore, the data acquisition device for each calibration object and the data acquisition device for the target object can be the same type of data acquisition device. The relative positions between the data acquisition device and the robotic arm used for teaching are the same as the relative positions between the data acquisition device and the robotic arm used for grasping the target object.

[0080] It is understandable that the process of determining the coordinates of multiple corner points of each calibration object and the corresponding sequence number of each calibration object is similar to the process of analyzing the coordinates of multiple corner points of each target object and the corresponding sequence number of each target object. Both are obtained based on a predetermined image analysis method, and will not be elaborated here.

[0081] It is understandable that identifying whether there are improperly placed objects among the target objects can also be considered as objects that cause the robotic arm to fail to grasp. Therefore, only improperly placed objects that affect the actual grasping of the robotic arm can be considered improperly placed. For example, if the robotic arm is preparing to grasp object a and object b, and the angle of object b is not horizontal, but it does not affect the normal grasping of the robotic arm, then object b cannot be considered improperly placed.

[0082] To ensure clarity of the layout, the specific process of identifying whether there are any improperly placed objects among the target objects will be described in subsequent embodiments, and will not be elaborated on here.

[0083] S104, in response to the existence of at least one non-compliant target object placement, output alarm information to characterize the non-compliant object placement, and in response to the trigger condition that the placement position has been adjusted, acquire a new target point cloud image containing point cloud data of each target object collected by the acquisition device, and return to step S102.

[0084] Understandably, if at least one object is found to be improperly placed, an alarm message can be output. This alarm message indicates that the object is improperly placed. When staff see this alarm message, they can confirm that an object is improperly placed and check and adjust the position of the object. This process can be considered to meet the triggering condition that the placement position has been adjusted.

[0085] Understandably, when the trigger condition that the placement position has been adjusted is met, the acquisition device can be controlled again to acquire new target point cloud images, and the process can return to the step of analyzing the target point cloud images according to the predetermined image analysis method, until it is detected that no target object is placed improperly.

[0086] Of course, the alarm information may also include the sequence number of the target object, so as to facilitate the staff to adjust the target object. In addition, the alarm information may also include other information, which this application does not specifically limit.

[0087] In one implementation, before outputting the alarm information characterizing non-compliant object placement, step B1 is further included:

[0088] Step B1: Determine the sorting number corresponding to the at least one target object to obtain the target sorting number;

[0089] Correspondingly, the output used to characterize alarm information for non-compliant object placement includes:

[0090] Output alarm information indicating that the placement of objects with the target sorting number is non-compliant.

[0091] Understandably, if at least one target object is found to be improperly placed, the sorting number of the target object can be determined first, and an alarm message containing the target sorting number can be output. This alarm message indicates that the target object of the target sorting number is improperly placed. When the staff sees this alarm message, they can adjust the position of the target object of the target sorting number so that the target object of the target sorting number is properly placed. At this time, it can be considered that the trigger condition that the placement position has been adjusted has been met.

[0092] For example, if an object is found to be improperly placed and its corresponding sequence number is determined to be 1, an alarm message containing the number 1 can be output. After viewing the alarm message, the staff can adjust the position of object number 1. After detecting that object number 1 has been adjusted, a new target point cloud image containing point cloud data of each target object is acquired from the acquisition device. The new target point cloud image is analyzed until no object is found to be improperly placed.

[0093] As can be seen, in response to the existence of at least one non-compliant target object placement, an alarm message can be output to characterize the non-compliant placement of the target object with the target sorting number, thereby reflecting the specific abnormal placement of the object and directly reminding manual intervention in the placement of the target object. The embodiments of this application can be applied to multiple scenarios, and are convenient and easy to use.

[0094] S105, in response to the absence of any non-compliant placement of target objects, the pose information of the robotic arm for simultaneously grasping each target object in a single operation is determined based on the coordinates of multiple corner points of each target object.

[0095] Understandably, if it is found that no target object is improperly placed, the coordinates of multiple corner points of each target object can be converted into the pose information of the robotic arm for simultaneously grasping each target object in a single operation, based on the coordinates of multiple corner points of each object. That is, the pose to be grasped for each target object.

[0096] In another implementation, the center coordinates can be calculated based on the coordinates of multiple corner points of each target object to obtain a center point coordinate, which can determine the pose information of the robotic arm for simultaneously grasping each target object in a single operation.

[0097] For example, the target objects are object a, object b, object c, object d and object e. Objects a to e are all rectangular blocks, each with 8 corner points, and the target objects have a total of 40 corner points. If it is found that no target object is improperly placed, the pose information of the robotic arm for simultaneously grasping object a, object b, object c, object d and object e is determined based on the corner coordinates of the 40 corner points.

[0098] Understandably, after determining the pose information of the robotic arm for simultaneously grasping various target objects in a single operation, if the executing entity is the robotic arm, the robotic arm can directly respond to the determined pose information and grasp each target object. Furthermore, if the executing entity is the server, the server can respond to the grasping command and control the robotic arm to grasp each target object. Therefore, this application can grasp various target objects based on the determined pose information, thus offering convenience and ease of use.

[0099] In one implementation, each calibration object is an object set during the teaching process; the teaching process is a process for teaching to obtain the conversion relationship between the coordinates of multiple corner points of each calibration object and the teaching pose of the robotic arm;

[0100] The process of determining the pose information of the robotic arm for simultaneously grasping various target objects in a single operation based on the coordinates of multiple corner points of each target object may include step C1:

[0101] Step C1: Based on the coordinates of multiple corner points of each target object and the transformation relationship, determine the pose information of the robotic arm for simultaneously grasping each target object in a single operation.

[0102] Understandably, if it is found that no target object is improperly placed, the coordinates of multiple corner points of each object can be converted into the pose information of the robotic arm for simultaneously grasping each target object in a single operation, based on the transformation relationship obtained from teaching.

[0103] Understandably, during the teaching process for each calibration object, the robotic arm can first be controlled to move to a position where its relative positional relationship allows it to simultaneously grasp multiple calibration objects in a single operation. The coordinates in the X, Y, and Z directions are recorded, as well as the rotation angles around the X, Y, and Z axes, i.e., (rx, ry, rz). These rotation angles around the three axes can also be called Euler angles. This yields the teaching pose of the robotic arm, and a transformation relationship is established between the teaching pose of the robotic arm and the coordinates of multiple corner points of each calibration object, thus completing the teaching process. This transformation relationship can be in matrix form. It should be emphasized that the teaching pose of the robotic arm described above refers to a six-axis robotic arm. In actual use, the specific form of the teaching pose is related to the type of robotic arm, and this application does not specifically limit the teaching pose.

[0104] For clarity, the teaching process will be described in detail in subsequent embodiments, and will not be elaborated upon here.

[0105] As can be seen, the embodiments of this application can analyze each target object based on a target point cloud image containing point cloud data of each target object, thereby obtaining the coordinates of multiple corner points of each target object and the corresponding sorting number of each target object, instead of analyzing the point cloud data of each target object as a whole. In this way, when identifying whether there are any improperly placed objects among the target objects, identification can be performed on a per-target-object basis. If no object is improperly placed, it indicates that the placement of each target object is compliant, so the accurate pose to be grasped can be directly determined, ensuring effective grasping of multiple objects. Furthermore, if at least one object is improperly placed, an alarm message can be output. After the improperly placed position is detected to have been adjusted, the target point cloud image containing point cloud data of each target object, acquired by the acquisition device, is re-acquired until it can be identified that no object is improperly placed, thereby ensuring effective grasping of multiple objects and improving the success rate of simultaneous grasping of multiple objects.

[0106] Optionally, in one implementation, the processing steps of the teaching process may include steps D1-D4:

[0107] Step D1: Acquire point cloud images containing various calibration objects collected by the acquisition device;

[0108] It is understood that the entity executing the teaching process can be the backend server or the robotic arm itself; this application does not make any specific restrictions on this.

[0109] It is understood that the point cloud image in step D1 is the point cloud image for each calibration object, and the point cloud image in step S101 is the point cloud image for each target object. The specific process is similar to that of step S101 above, and will not be elaborated on here.

[0110] Step D2: Analyze the acquired point cloud image according to the predetermined image analysis method to obtain the coordinates of multiple corner points of each calibration object and the sorting number corresponding to each calibration object, and save the coordinates of multiple corner points of each calibration object and the sorting number corresponding to each calibration object.

[0111] It is understandable that in step D2, the obtained point cloud image is analyzed according to the predetermined image analysis method to obtain the coordinates of multiple corner points of each calibration object and the corresponding sorting number of each calibration object. This is similar to step S102. Step D2 is for the point cloud image of each calibration object, while step S102 is for the point cloud image of each target object. Therefore, it will not be elaborated on here.

[0112] Furthermore, after obtaining the coordinates of multiple corner points of each calibrator and the corresponding sorting number of each calibrator, the coordinates of multiple corner points of each calibrator and the corresponding sorting number of each calibrator can be saved and used in step S104.

[0113] Step D3: In response to the robotic arm moving to a position where it can simultaneously grasp each calibration object, the pose of the robotic arm is obtained as the teaching pose;

[0114] It is understood that the robotic arm can be manually controlled to move to a position where it can simultaneously grasp each of the calibrated objects, or it can be controlled by a backend server to move to a position where it can simultaneously grasp each of the calibrated objects. This application does not make any specific limitations on this.

[0115] Understandably, by moving the robotic arm to a position where it can simultaneously grasp all the calibration objects, and recording the coordinates in the X, Y, and Z directions, as well as the rotation angles around the X, Y, and Z axes, the pose of the robotic arm can be obtained and used as the teaching pose. The teaching pose can be considered the pose in which the robotic arm can effectively grasp all the calibration objects.

[0116] Step D4: Construct the transformation relationship between the coordinates of multiple corner points of each calibration object and the teaching pose to complete the teaching process.

[0117] It is understandable that after constructing the transformation relationship between the coordinates of multiple corner points of each calibration object and the teaching pose, the teaching process can be considered complete. The essence of the teaching process can be considered as the process of obtaining the transformation relationship.

[0118] In one implementation, constructing the transformation relationship between the coordinates of multiple corner points of each calibration object and the teaching pose may include steps D41-D42:

[0119] Step D41: Using a point fitting algorithm, the coordinates of multiple corner points of each calibration object are fitted to obtain the coordinates of the reference point.

[0120] It is understandable that a point fitting algorithm can be used to fit multiple corner points of various calibration objects into a single point, which serves as a reference point, and the coordinates of this reference point can be obtained. Of course, in another implementation, other algorithms can also be used to fit the points, and this application does not specifically limit this approach.

[0121] For example, each calibration object has a total of 40 corner points. Using a point fitting algorithm, the 40 corner points can be fitted into a single reference point, and the coordinates of the reference point can be obtained.

[0122] Step D42: Construct the conversion relationship between the reference point coordinates and the teaching pose to complete the teaching process.

[0123] It is understandable that three-dimensional coordinates can be used to represent teaching poses. Therefore, a transformation relationship from reference point coordinates to teaching poses can be constructed, which is essentially constructing a transformation relationship between two coordinates to complete the teaching process.

[0124] Of course, in another implementation, the conversion relationship between the coordinates of multiple corner points of each calibration object and the teaching pose can be directly constructed to complete the teaching process. This application does not specifically limit this.

[0125] As can be seen, the transformation relationship between the coordinates of multiple corner points of each calibration object and the taught pose can be obtained through the teaching process. Using this transformation relationship, the accurate pose information of the robotic arm for simultaneously grasping each target object can be determined, thereby ensuring effective grasping of multiple objects and improving the success rate of simultaneous grasping of multiple objects.

[0126] Optionally, in one implementation, calculating the corner coordinates of multiple corner points of each object simultaneously grasped by the robotic arm based on the point cloud region of each selected object and predetermined object size information may include steps E1-E2:

[0127] Step E1: For each selected object's point cloud region, perform a 3D model of the object based on the predetermined object size information.

[0128] It is understandable that since any point cloud region can be considered as the approximate location of the target object in the point cloud image, after selecting the point cloud region of each object, a stereo model of the object can be performed on the point cloud region according to the predetermined object size information to obtain the stereo model of the object corresponding to the point cloud region.

[0129] It is understood that any point cloud modeling method can be applied to this application, and this application does not make any specific limitations on it.

[0130] Step E2: After 3D modeling, obtain the corner coordinates of multiple corner points of each object simultaneously grasped by the robotic arm.

[0131] Understandably, after 3D modeling, the coordinates of multiple corner points of the 3D model of each object can be calculated, thereby obtaining the coordinates of multiple corner points of each object simultaneously grasped by the robotic arm. For example, 3D models are now created for objects a, b, c, d, and e. Objects a through e are all rectangular blocks of the same size, and there are a total of 40 corner points. The coordinates of all 40 corner points can be calculated.

[0132] Of course, any method for calculating the corner coordinates of an object is applicable to this application, and this application does not make any specific limitations on it.

[0133] To better understand the concepts of 3D modeling and calculating corner coordinates, the following explanation will be provided with reference to the accompanying figures, as shown in Figures 3(a) and 3(b):

[0134] Figure 3(a) shows the 3D modeling of each object. There are 10 3D models in total. Each 3D model is a rectangular block. The 5 3D models in the first row are numbered 1-5, and the 5 3D models in the second row are numbered 6-10.

[0135] Figure 3(b) shows the corner points of the 3D model with the sorting number 1. The 3D model is a rectangular block with a total of 8 corner points, namely point 1 to point 8. Each corner point has corresponding corner point information.

[0136] As can be seen, the embodiments of this application can perform three-dimensional modeling of objects, thereby calculating the corner coordinates of multiple corner points of each object. In subsequent steps, it can identify whether there are any improperly placed objects among the target objects, thereby ensuring effective grasping of multiple objects and improving the success rate of grasping multiple objects simultaneously.

[0137] Alternatively, in one implementation, such as Figure 4 As shown, the step of identifying whether there are any improperly placed objects among the target objects based on the coordinates of multiple corner points of each target object and its corresponding sorting number, and the coordinates of multiple corner points of each calibrated object and its corresponding sorting number, includes:

[0138] S401, based on the coordinates of multiple corner points of each target object and the coordinates of multiple corner points of each calibration object, detect whether the target objects and calibration objects with the same sorting number meet the position difference condition.

[0139] The position difference condition includes whether the first relative position of the target object with the same sorting number and the second relative position of the marker with the same sorting number exceed a predetermined position difference threshold.

[0140] The first relative position is the target object with the same sorting number, referring to the relative position of each target object; the second relative position is the marker with the same sorting number, referring to the relative position of each marker.

[0141] It is understandable that target objects and markers with the same sequence number can be considered as target objects and markers having the same relative position when grasped by the robotic arm. For example, if the robotic arm can grasp 5 target objects or 5 markers simultaneously, then the target object located in the middle when grasping target objects and the marker located in the middle when grasping markers can be considered as target objects and markers with the same sequence number.

[0142] Understandably, the second relative position corresponding to the calibrator with the same sorting number can be considered as the relative position of the calibrator with respect to all other calibrators. Since the calibrators are correctly positioned, their relative positions with respect to all other calibrators are also correct. In other words, the second relative position corresponding to the calibrator with the same sorting number is an accurate relative position. The first relative position corresponding to the target object with the same sorting number can be considered as the relative position of the target object with respect to all other target objects. If the placement of the target object differs from that of other target objects, the first relative position corresponding to the target object with the same sorting number will show a difference. In this case, it is possible to detect whether the first relative position corresponding to the target object with the same sorting number and the second relative position corresponding to the calibrator with the same sorting number exceed a predetermined position difference threshold.

[0143] For example, for objects a and calibrators 1 with the same sorting number, the relative position of object a with the same sorting number to other objects is the first relative position of object a, and the relative position of calibrator 1 with the same sorting number to other calibrators is the second relative position of calibrator 1. At this time, it is possible to detect whether the first relative position of object a and the second relative position of calibrator 1 exceed a predetermined position difference threshold.

[0144] Furthermore, the first relative position corresponding to the target object with the same sorting number and the second relative position corresponding to the calibrator with the same sorting number can also be calculated using the PNP (perspective-n-point) method. This application does not specifically limit this in the embodiments.

[0145] In one implementation, detecting whether target objects and calibration objects with the same sorting number satisfy the positional difference condition based on multiple corner coordinates of each target object and multiple corner coordinates of each calibration object may include steps F1-F4:

[0146] Step F1: Select the first corner point among the corner points of each target object as the first coordinate origin; wherein, the first corner point is the corner point of the target object with a predetermined sorting number at the target position;

[0147] The predetermined sorting number can be the sorting number of any target object, and the first corner point can be the upper left corner of the target object with the predetermined sorting number, or it can be the lower left corner, etc. This embodiment does not specifically limit the target location. Furthermore, after selecting the first corner point, it can be used as the first coordinate origin. For example, selecting the upper left corner of object a can serve as the first coordinate origin.

[0148] Furthermore, the first coordinate origin is not limited to a corner point, but can also be the center point of any target object. In this application embodiment, the first coordinate origin is not specifically limited.

[0149] Step F2: Select the second corner point among the corner points of each calibrator as the second coordinate origin; wherein, the second corner point is the corner point of the calibrator with a predetermined sorting number at the target position;

[0150] The predetermined sorting number can be the sorting number of any marker, and the second corner point can be the upper left corner of the marker with the predetermined sorting number, or the lower left corner, etc. This embodiment does not specifically limit the target location. Furthermore, after selecting the second corner point, it can be used as the origin of the second coordinate system. For example, selecting the upper left corner of marker 1 can serve as the origin of the second coordinate system.

[0151] It should be emphasized that when selecting the second corner point among the corner points of each calibrator, the predetermined sorting number of the selected calibrator must be the same as the predetermined sorting number in step F1, and the target position of the calibrator with the predetermined sorting number must also be the same as the target position in step F1.

[0152] Furthermore, the second coordinate origin is not limited to a corner point, but can also be the center point of any calibrated object. The embodiments of this application do not specifically limit the second coordinate origin.

[0153] Step F3: Based on the coordinate difference between the first coordinate origin and the second coordinate origin, perform coordinate offset processing on the corner coordinates of each target object to obtain the offset corner coordinates of each target object; wherein, the offset processing is used to make the first coordinate origin coincide with the second coordinate origin.

[0154] It is understandable that after obtaining the first and second coordinate origins, the coordinate difference between the two origins can be calculated. For example, if the coordinates of the first coordinate origin are (10, 10, 10) and the coordinates of the second coordinate origin are (5, 5, 5), then the difference between the two coordinates in the X direction is 5, the difference in the Y direction is 5, and the difference in the Z direction is 5.

[0155] Understandably, by utilizing the coordinate difference between the two origins, the corner coordinates of various target objects can be offset. This can make the first coordinate origin coincide with the second coordinate origin after the coordinate offset, and can also make the corner coordinates of each target object after the coordinate offset close to the corner coordinates of each calibration object, possibly resulting in corner overlap. For example, if the difference between the first and second coordinate origins in the X direction is determined to be 5, the difference in the Y direction is 5, and the difference in the Z direction is 5, the corner coordinates of each target object can be offset. For instance, if the corner coordinates of object a are (12, 12, 12), after the coordinate offset, the offset corner coordinates can be obtained as (7, 7, 7).

[0156] Step F4: For each sorting number, based on the corner coordinates of the target object after offset processing and the corner coordinates of the calibrator at that sorting number, detect whether the positional difference between the target object and the calibrator at that sorting number exceeds a predetermined difference range.

[0157] It is understandable that the corner coordinates of each target object after offset processing are close to the corner coordinates of the calibration object. Of course, there may be errors. In this case, for each sorting number, it can be detected whether the difference between the corner coordinates of the target object and the corner coordinates of the calibration object exceeds a predetermined difference range. The predetermined difference range can be an empirical value or a manually set value. This application embodiment does not specifically limit this.

[0158] For example, the predetermined difference range is that the difference in any direction does not exceed 2. Object a and calibrator a belong to the same sorting sequence. The corner coordinates of point 1 of object a after offset processing are (7, 7, 7), and the corner coordinates of point 1 of calibrator a are (6, 6, 6). At this time, it can be determined that the difference between the two coordinates in the X direction is 1, the difference in the Y direction is 1, and the difference in the Z direction is 1. At this time, it can be considered that point 1 of object a does not exceed the predetermined difference range. Object b and calibrator b belong to the same sorting sequence. The corner coordinates of point 2 of object b after offset processing are (20, 10, 15), and the corner coordinates of point 1 of calibrator a are (20, 10, 13). At this time, it can be determined that there is no difference between the two coordinates in the X direction, no difference in the Y direction, and a difference of 3 in the Z direction. The difference in the Z direction exceeds the predetermined difference range. At this time, it can be considered that point 2 of object b exceeds the predetermined difference range.

[0159] Of course, there is no single method for detecting whether the positional difference between the target object and the calibration object of the sorted sequence number exceeds a predetermined difference range. Alternatively, the corner coordinates of the target object and the calibration object of the sorted sequence number can be input into a trained network model so that the network model outputs a detection result. This detection result indicates whether the positional difference between the target object and the calibration object of the sorted sequence number exceeds a predetermined difference range. This application does not specifically limit this method.

[0160] As can be seen, the embodiments of this application can detect whether the positional difference between the target object and the marker exceeds a predetermined difference range by aligning the first coordinate origin with the second coordinate origin. This can identify whether the placement of each target object is compliant, thereby ensuring effective grasping of multiple objects and improving the success rate of grasping multiple objects simultaneously.

[0161] S402, if there are target objects and markers with the same sorting number that meet the position difference condition, the target object with the same sorting number shall be identified as an object with non-compliant placement.

[0162] It is understandable that the existence of a target object and a marker with the same sorting number that meet the position difference condition can be due to the detection that the first relative position corresponding to the target object with the same sorting number and the second relative position corresponding to the marker with the same sorting number exceed a predetermined position difference threshold, or the detection that the position difference between the target object and the marker with the same sorting number exceeds a predetermined difference range. In this case, the target object at the same sorting number can be identified as an object with non-compliant placement, and thus the above step S105 can be executed.

[0163] As can be seen, the embodiments of this application can detect whether target objects and markers with the same sorting number meet the position difference condition, thereby detecting whether the placement of each target object is compliant. Compared with the prior art, the embodiments of this application can detect each target object individually, rather than detecting all target objects as a whole, thereby improving the detection accuracy of object placement, ensuring effective grasping of multiple objects, and improving the success rate of simultaneous grasping of multiple objects.

[0164] Based on the above method embodiments, such as Figure 5 As shown in the figure, this application provides a grasping pose determination device, the device comprising:

[0165] The acquisition module 510 is used to acquire a target point cloud image containing point cloud data of each target object, which is acquired by the acquisition device; wherein, each target object is an object to be grasped simultaneously by the robotic arm in a single operation.

[0166] The analysis module 520 is used to analyze the target point cloud image according to a predetermined image analysis method to obtain the coordinates of multiple corner points of each target object and the sorting number corresponding to each target object.

[0167] The identification module 530 is used to identify whether there are any improperly placed objects among the target objects based on the coordinates of multiple corner points and their corresponding sorting numbers of each target object, as well as the coordinates of multiple corner points and their corresponding sorting numbers of each calibration object. Each calibration object is a group of objects whose relative positions allow the robotic arm to grasp them simultaneously in a single operation. The corner point coordinates and corresponding sorting numbers of each calibration object are obtained by analyzing the point cloud image containing each calibration object acquired by the acquisition device based on the predetermined image analysis method.

[0168] The output module 540 is used to output alarm information to characterize the non-compliant placement of objects in response to the existence of at least one non-compliant target object, and to acquire a new target point cloud image containing point cloud data of each target object collected by the acquisition device in response to the trigger condition that the placement position has been adjusted, and to return the step of analyzing the target point cloud image according to the predetermined image analysis method.

[0169] The first determining module 550 is used to determine the pose information of the robotic arm for simultaneously grasping each target object in a single operation, based on the coordinates of multiple corner points of each target object, in response to the absence of any non-compliant placement of any target object.

[0170] Optionally, the predetermined image analysis method includes:

[0171] For any point cloud image to be analyzed, object region localization is performed to obtain multiple point cloud regions;

[0172] According to the predetermined sorting rules, the obtained point cloud regions are sorted to obtain the sorting sequence number corresponding to each point cloud region.

[0173] According to the selection rules for the objects to be grasped that match the sorting rules, based on the sorting number corresponding to each point cloud region, the point cloud region and the corresponding sorting number of each object that is grasped simultaneously by the robotic arm in a single operation are selected in the point cloud image to be analyzed.

[0174] Based on the point cloud region of each selected object and the predetermined object size information, the corner coordinates of multiple corner points of each object simultaneously grasped by the robotic arm are calculated.

[0175] Optionally, each calibration object is an object set up during the teaching process; the teaching process is a process for teaching to obtain the conversion relationship between the coordinates of multiple corner points of each calibration object and the teaching pose of the robotic arm;

[0176] The first determining module is specifically used for:

[0177] The process of determining the pose information of the robotic arm for simultaneously grasping multiple target objects in a single operation, based on the coordinates of multiple corner points of each target object, includes:

[0178] Based on the coordinates of multiple corner points of each target object and the transformation relationship, the pose information of the robotic arm for simultaneously grasping each target object in a single operation is determined.

[0179] Optionally, the device further includes:

[0180] The second determining module is used to determine the sorting number corresponding to at least one target object before outputting alarm information that characterizes non-compliant object placement, and obtain the target sorting number.

[0181] The output module is specifically used for:

[0182] Output alarm information indicating that the placement of objects with the target sorting number is non-compliant.

[0183] Optionally, the identification module includes:

[0184] The detection submodule is used to detect whether target objects and calibration objects with the same sorting number meet the position difference condition based on the coordinates of multiple corner points of each target object and the coordinates of multiple corner points of each calibration object.

[0185] The determination submodule is used to determine the object with the same sorting number as an object with non-compliant placement if there are target objects and markers with the same sorting number that meet the position difference condition.

[0186] The position difference condition includes whether the first relative position of the target object with the same sorting number and the second relative position of the marker with the same sorting number exceed a predetermined position difference threshold.

[0187] The first relative position is the target object with the same sorting number, referring to the relative position of each target object; the second relative position is the marker with the same sorting number, referring to the relative position of each marker.

[0188] Optionally, the detection submodule is specifically used for:

[0189] The first corner point among the corner points of each target object is selected as the first coordinate origin; wherein, the first corner point is the corner point of the target object with a predetermined sorting number at the target position;

[0190] The second corner point among the corner points of each calibrator is selected as the second coordinate origin; wherein, the second corner point is the corner point of the calibrator with a predetermined sorting number at the target position;

[0191] Based on the coordinate difference between the first coordinate origin and the second coordinate origin, the corner coordinates of each target object are offset to obtain the offset corner coordinates of each target object; wherein, the offset is used to make the first coordinate origin coincide with the second coordinate origin.

[0192] For each sorting number, based on the corner coordinates of the target object after offset processing and the corner coordinates of the calibrator at that sorting number, it is detected whether the positional difference between the target object and the calibrator at that sorting number exceeds a predetermined difference range.

[0193] Optionally, the processing steps of the teaching process include:

[0194] Acquire point cloud images containing various calibration objects collected by the acquisition device;

[0195] According to the predetermined image analysis method, the acquired point cloud image is analyzed to obtain the coordinates of multiple corner points of each calibration object and the sorting number corresponding to each calibration object, and the coordinates of multiple corner points of each calibration object and the sorting number corresponding to each calibration object are saved.

[0196] In response to the robotic arm moving to a position where it can simultaneously grasp each calibration object, the pose of the robotic arm is obtained and used as a teaching pose;

[0197] The transformation relationship between the coordinates of multiple corner points of each calibration object and the teaching pose is constructed to complete the teaching process.

[0198] Optionally, the transformation relationship between the coordinates of multiple corner points of each calibration object and the teaching pose includes:

[0199] Using a point fitting algorithm, the coordinates of multiple corner points of each calibration object are fitted to obtain the coordinates of the reference point.

[0200] Establish the transformation relationship between the reference point coordinates and the teaching pose to complete the teaching process.

[0201] Optionally, the step of locating object regions in any point cloud image to be analyzed to obtain multiple point cloud regions includes:

[0202] Based on the point cloud contour library, contour localization is performed on any point cloud image to be analyzed to obtain the multiple point cloud regions.

[0203] Optionally, the step of calculating the corner coordinates of multiple corner points of each object simultaneously grasped by the robotic arm in a single operation, based on the point cloud region of each selected object and predetermined object size information, includes:

[0204] For each selected object's point cloud region, a 3D model of the object is performed for that point cloud region according to the predetermined object size information.

[0205] After 3D modeling, the corner coordinates of multiple corner points of each object simultaneously grasped by the robotic arm are obtained.

[0206] In the technical solution of this application, the acquisition, storage, use, processing, transmission, provision and disclosure of user personal information are all carried out with the user's authorization.

[0207] This application also provides an electronic device, such as... Figure 6 As shown, it includes:

[0208] Memory 601 is used to store computer programs;

[0209] The processor 602, when executing the program stored in the memory 601, implements any of the above-mentioned grasping pose determination methods.

[0210] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 602, the communication interface, and the memory 601 communicating with each other via the communication bus.

[0211] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0212] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0213] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0214] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0215] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements any of the above-described grasping pose determination methods.

[0216] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the grasping pose determination methods described in the above embodiments.

[0217] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0218] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0219] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0220] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A method for determining the grasping pose, characterized in that, The method includes: Acquire a target point cloud image containing point cloud data of each target object, obtained by a data acquisition device; wherein, each target object is an object to be simultaneously grasped by a robotic arm in a single operation. According to a predetermined image analysis method, the target point cloud image is analyzed to obtain the coordinates of multiple corner points of each target object and the corresponding sorting number of each target object. Based on the coordinates of multiple corner points and their corresponding sorting numbers for each target object, as well as the coordinates of multiple corner points and their corresponding sorting numbers for each calibration object, the system identifies whether any objects in each target object are improperly placed. Each calibration object is a group of objects whose relative positions allow the robotic arm to grasp them simultaneously in a single operation. The corner point coordinates and corresponding sorting numbers of each calibration object are obtained by analyzing the point cloud image containing each calibration object acquired by the acquisition device using a predetermined image analysis method. In response to the existence of at least one non-compliant target object placement, an alarm message is output to characterize the non-compliant object placement. In response to the trigger condition that the placement position has been adjusted, a new target point cloud image containing point cloud data of each target object is acquired by the acquisition device, and the step of analyzing the target point cloud image according to a predetermined image analysis method is returned. In response to the absence of any non-compliant placement of target objects, the pose information of the robotic arm for simultaneously grasping each target object in a single operation is determined based on the coordinates of multiple corner points of each target object.

2. The method according to claim 1, characterized in that, The predetermined image analysis method includes: For any point cloud image to be analyzed, object region localization is performed to obtain multiple point cloud regions; According to the predetermined sorting rules, the obtained point cloud regions are sorted to obtain the sorting sequence number corresponding to each point cloud region. According to the selection rules for the objects to be grasped that match the sorting rules, based on the sorting number corresponding to each point cloud region, the point cloud region and the corresponding sorting number of each object that is grasped simultaneously by the robotic arm in a single operation are selected in the point cloud image to be analyzed. Based on the point cloud region of each selected object and the predetermined object size information, the corner coordinates of multiple corner points of each object simultaneously grasped by the robotic arm are calculated.

3. The method according to claim 1, characterized in that, Each calibration object is an object set up during the teaching process; the teaching process is a process for processing the conversion relationship between the coordinates of multiple corner points of each calibration object and the teaching pose of the robotic arm. The process of determining the pose information of the robotic arm for simultaneously grasping multiple target objects in a single operation, based on the coordinates of multiple corner points of each target object, includes: Based on the coordinates of multiple corner points of each target object and the transformation relationship, the pose information of the robotic arm for simultaneously grasping each target object in a single operation is determined.

4. The method according to claim 1, characterized in that, Before the output is used to characterize the alarm information for non-compliant object placement, it also includes: Determine the sorting number corresponding to at least one target object to obtain the target sorting number; The output is used to characterize alarm information for non-compliant object placement, including: Output alarm information indicating that the placement of objects with the target sorting number is non-compliant.

5. The method according to any one of claims 1-4, characterized in that, The method of identifying whether there are any improperly placed objects among the target objects based on the coordinates of multiple corner points of each target object and its corresponding sorting number, as well as the coordinates of multiple corner points of each calibrated object and its corresponding sorting number, includes: Based on the coordinates of multiple corner points of each target object and the coordinates of multiple corner points of each calibration object, it is determined whether the target objects and calibration objects with the same sorting number meet the position difference condition. If there are target objects and markers with the same sorting number that meet the position difference condition, the target object with the same sorting number is identified as an object with non-compliant placement. The position difference condition includes whether the first relative position of the target object with the same sorting number and the second relative position of the marker with the same sorting number exceed a predetermined position difference threshold. The first relative position is the target object with the same sorting number, referring to the relative position of each target object; the second relative position is the marker with the same sorting number, referring to the relative position of each marker.

6. The method according to claim 5, characterized in that, The method of detecting whether target objects and calibration objects with the same sorting number satisfy the positional difference condition based on multiple corner coordinates of each target object and multiple corner coordinates of each calibration object includes: The first corner point among the corner points of each target object is selected as the first coordinate origin; wherein, the first corner point is the corner point of the target object with a predetermined sorting number at the target position; The second corner point among the corner points of each calibrator is selected as the second coordinate origin; wherein, the second corner point is the corner point of the calibrator with a predetermined sorting number at the target position; Based on the coordinate difference between the first coordinate origin and the second coordinate origin, the corner coordinates of each target object are offset to obtain the offset corner coordinates of each target object; wherein, the offset is used to make the first coordinate origin coincide with the second coordinate origin. For each sorting number, based on the corner coordinates of the target object after offset processing and the corner coordinates of the calibrator at that sorting number, it is detected whether the positional difference between the target object and the calibrator at that sorting number exceeds a predetermined difference range.

7. The method according to claim 3, characterized in that, The processing steps of the teaching process include: Acquire point cloud images containing various calibration objects collected by the acquisition device; According to the predetermined image analysis method, the acquired point cloud image is analyzed to obtain the coordinates of multiple corner points of each calibration object and the sorting number corresponding to each calibration object, and the coordinates of multiple corner points of each calibration object and the sorting number corresponding to each calibration object are saved. In response to the robotic arm moving to a position where it can simultaneously grasp each calibration object, the pose of the robotic arm is obtained and used as a teaching pose; The transformation relationship between the coordinates of multiple corner points of each calibration object and the teaching pose is constructed to complete the teaching process.

8. The method according to claim 7, characterized in that, The transformation relationship between the coordinates of multiple corner points of each calibration object and the teaching pose is constructed, including: Using a point fitting algorithm, the coordinates of multiple corner points of each calibration object are fitted to obtain the coordinates of the reference point. Establish the transformation relationship between the reference point coordinates and the teaching pose to complete the teaching process.

9. The method according to claim 2, characterized in that, The process of locating object regions in any point cloud image to be analyzed to obtain multiple point cloud regions includes: Based on the point cloud contour library, contour localization is performed on any point cloud image to be analyzed to obtain the multiple point cloud regions.

10. The method according to claim 2, characterized in that, Based on the point cloud region of each selected object and the predetermined object size information, the corner coordinates of multiple corner points of each object simultaneously grasped by the robotic arm in a single operation are calculated, including: For each selected object's point cloud region, a 3D model of the object is performed for that point cloud region according to the predetermined object size information. After 3D modeling, the corner coordinates of multiple corner points of each object simultaneously grasped by the robotic arm are obtained.

11. A grasping pose determination device, characterized in that, The device includes: The acquisition module is used to acquire a target point cloud image containing point cloud data of each target object, which is obtained by the acquisition device; wherein, each target object is an object that the robotic arm is to grasp simultaneously in a single operation. The analysis module is used to analyze the target point cloud image according to a predetermined image analysis method to obtain the coordinates of multiple corner points of each target object and the sorting number corresponding to each target object. The identification module is used to identify whether there are any improperly placed objects among the target objects based on the coordinates of multiple corner points and their corresponding sorting numbers of each target object, as well as the coordinates of multiple corner points and their corresponding sorting numbers of each calibration object. Each calibration object is a group of objects whose relative positions allow the robotic arm to grasp them simultaneously in a single operation. The corner point coordinates and corresponding sorting numbers of each calibration object are obtained by analyzing the point cloud image containing each calibration object acquired by the acquisition device based on the predetermined image analysis method. The output module is used to output alarm information to characterize the non-compliant placement of objects in response to the existence of at least one non-compliant target object, and to acquire a new target point cloud image containing point cloud data of each target object collected by the acquisition device in response to the trigger condition that the placement position has been adjusted, and to return the step of analyzing the target point cloud image according to the predetermined image analysis method. The first determining module is used to determine the pose information of the robotic arm for simultaneously grasping each target object in a single operation, based on the coordinates of multiple corner points of each target object, in response to the absence of any non-compliant placement of any target object.

12. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-10.

14. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-10.

Images