Methods, devices and electronic equipment for determining the pose of grasped objects

Abstract

This application relates to a method, apparatus, electronic device, and readable storage medium for determining the pose of a grasped object, and relates to the field of vision control technology. The method includes: obtaining the rotation angle of a target plane relative to a target coordinate plane in a camera coordinate system; wherein the target plane is obtained based on the end point cloud data of the grasped object's end, and the end point cloud data is obtained based on the initial point cloud data of the grasped object and the positioning result of the end; obtaining a rotation matrix corresponding to the rotation angle; calculating the offset corresponding to the target coordinate axis based on the rotation matrix; wherein the target coordinate axis is perpendicular to the target coordinate plane; and obtaining the pose of the grasped object in the camera coordinate system based on the rotation angle of the target coordinate plane and the offset corresponding to the target coordinate axis.

Description

Technical Field

[0001] This application relates to the field of vision control technology, and more specifically, to a method, apparatus, electronic device, and readable storage medium for determining the pose of a grasped object. Background Art

[0002] In virtual reality (VR) products such as head-mounted displays (HMDs), the printed circuit board (PCB) motherboard layout includes 14+ flexible printed circuit (FPC) slots. Manual insertion and removal suffers from low efficiency, high defect rates, and high labor and quality costs. In this context, using robotic arms to automatically assemble the FPCs in VR products can not only effectively improve insertion and removal efficiency and reduce defect rates, but also save significant labor costs.

[0003] Robotic arms need to obtain the accurate pose of objects to grasp them. In reality, VR products are generally complex in shape and large in size. For the flexible flat cables used to connect circuit boards in VR products, due to the high degree of freedom at the ends of the cables, it is often impossible to constrain the cables to a stable posture through tooling design in some cases. In other cases, the flexible flat cables are integrated with other components in the VR product, for example, the flexible flat cables are fixed to other components. During the process of grasping the flexible flat cables, the lack of accurate pose information makes it very easy for the robotic arm to tear the cables, causing damage.

[0004] Therefore, the above situation places higher demands on the accuracy of determining the pose of flexible flatbed cables in current industrial production. Summary of the Invention

[0005] One objective of this application is to provide a new technical solution for determining the pose of a grasped object, thereby solving the technical problem of low accuracy in the pose of the grasped object obtained in the prior art.

[0006] According to a first aspect of this application, a method for determining the pose of a grasped object is provided, comprising: obtaining a rotation angle of a target plane relative to a target coordinate plane in a camera coordinate system; wherein the target plane is obtained based on end point cloud data of the grasped object's end, and the end point cloud data is obtained based on initial point cloud data of the grasped object and the positioning result of the end; obtaining a rotation matrix corresponding to the rotation angle; calculating the offset corresponding to the target coordinate axis based on the rotation matrix; wherein the target coordinate axis is perpendicular to the target coordinate plane; and obtaining the pose of the grasped object in the camera coordinate system based on the rotation angle of the target coordinate plane and the offset corresponding to the target coordinate axis.

[0007] Optionally, before obtaining the rotation angle of the target plane relative to the target coordinate plane in the camera coordinate system, the method further includes: obtaining the initial point cloud data of the grasped object; and, based on the initial point cloud data and a preset spatial range model, extracting the first point cloud data located within the spatial range model as the end point cloud data.

[0008] Optionally, after extracting the first point cloud data located within the spatial range model as the end point cloud data based on the initial point cloud data and the preset spatial range model, the method further includes: vertically projecting the first point cloud data to obtain a first projected image; extracting a target projected image located within the planar range model based on the first projected image and the preset planar range model; obtaining the second point cloud data corresponding to the target projected image, and using the second point cloud data as the end point cloud data.

[0009] Optionally, the offset corresponding to the target coordinate axis is calculated based on the rotation matrix, including: determining the target rotation matrix based on the first rotation matrix corresponding to the first rotation angle and the second rotation matrix corresponding to the second rotation angle; wherein the first rotation angle is the rotation angle of the target plane relative to the first target coordinate plane in the camera coordinate system, and the second rotation angle is the rotation angle of the target plane relative to the second target coordinate plane in the camera coordinate system; obtaining the target rotation angle based on the target rotation matrix; obtaining the center point coordinates of the end point based on the target rotation angle; and determining the offset corresponding to the target coordinate axis based on the inverse matrix of the target rotation matrix and the center point coordinates.

[0010] Optionally, the target rotation angle is obtained based on the target rotation matrix, including: obtaining the product of the target rotation matrix and the third point cloud data of the target plane to obtain the fourth point cloud data; wherein the third point cloud data is obtained by performing planar fitting on the end point cloud data; the fourth point cloud data is vertically projected to obtain a second projected image; the target rotation angle is obtained by obtaining the target angle between the first side in the second projected image and the set coordinate axis; wherein the length of the first side is greater than or equal to the length of other sides in the second projected image, and the angle between the set coordinate axis and the first side is less than or equal to the angle between the set coordinate axis and any other side.

[0011] Optionally, the end is flat or approximately flat. The target rotation angle is obtained according to the target rotation matrix, including: rotating the second projected image around a rotation coordinate axis by the target rotation angle to obtain a rotated image; wherein the rotation coordinate axis is perpendicular to the second projected image; obtaining the planar coordinates of the center point of the rotated image; determining the height coordinates of the end in the direction of the rotation coordinate axis according to the target plane and the target rotation matrix; and obtaining the coordinates of the center point of the end according to the planar coordinates and the height coordinates.

[0012] According to a second aspect of this application, a pose determination device for a grasped object is also provided, comprising: a first acquisition module, configured to acquire the rotation angle of a target plane relative to a target coordinate plane in a camera coordinate system; wherein the target plane is obtained based on end point cloud data of the grasped object's end, and the end point cloud data is obtained based on initial point cloud data of the grasped object and the positioning result of the end; an acquisition module, configured to acquire a rotation matrix corresponding to the rotation angle; a calculation module, configured to calculate the offset corresponding to the target coordinate axis based on the rotation matrix; wherein the target coordinate axis is perpendicular to the target coordinate plane; and a processing module, configured to acquire the pose of the grasped object in the camera coordinate system based on the rotation angle of the target coordinate plane and the offset corresponding to the target coordinate axis.

[0013] Optionally, the device further includes: a second acquisition module, used to acquire initial point cloud data of the object to be grasped before the first acquisition module acquires the rotation angle of the target plane relative to the target coordinate plane in the camera coordinate system; and a first interception module, used to intercept the first point cloud data located within the spatial range model as the end point cloud data based on the initial point cloud data and a preset spatial range model.

[0014] Optionally, the device further includes: a first projection module, used to vertically project the first point cloud data after the first interception module intercepts the first point cloud data located within the spatial range model as the end point cloud data based on the initial point cloud data and the preset spatial range model, to obtain a first projected image; a second interception module, used to intercept the target projected image located within the plane range model based on the first projected image and the preset plane range model; and a third acquisition module, used to acquire the second point cloud data corresponding to the target projected image and use the second point cloud data as the end point cloud data.

[0015] Optionally, the calculation module includes: a first determining submodule, used to determine the target rotation matrix based on the first rotation matrix corresponding to the first rotation angle and the second rotation matrix corresponding to the second rotation angle; wherein the first rotation angle is the rotation angle of the target plane relative to the first target coordinate plane in the camera coordinate system, and the second rotation angle is the rotation angle of the target plane relative to the second target coordinate plane in the camera coordinate system; a first obtaining submodule, used to obtain the target rotation angle based on the target rotation matrix; a second obtaining submodule, used to obtain the center point coordinates of the end point based on the target rotation angle; and a second determining submodule, used to determine the offset corresponding to the target coordinate axis based on the inverse matrix of the target rotation matrix and the center point coordinates.

[0016] Optionally, the first acquisition submodule is used to: acquire the product of the target rotation matrix and the third point cloud data of the target plane to obtain the fourth point cloud data; wherein the third point cloud data is obtained by performing planar fitting on the end point cloud data; perform vertical projection on the fourth point cloud data to obtain the second projection image; acquire the target rotation angle between the first side in the second projection image and the set coordinate axis as the target rotation angle; wherein the length of the first side is greater than or equal to the length of other sides in the second projection image, and the angle between the set coordinate axis and the first side is less than or equal to the angle between the set coordinate axis and any other side.

[0017] Optionally, the end is flat or approximately flat, and the second acquisition submodule is used to: rotate the second projected image around the rotation coordinate axis by the target rotation angle to obtain a rotated image; wherein the rotation coordinate axis is perpendicular to the second projected image; acquire the planar coordinates of the center point of the rotated image; determine the height coordinates of the end in the direction of the rotation coordinate axis according to the target plane and the target rotation matrix; and acquire the center point coordinates of the end according to the planar coordinates and the height coordinates.

[0018] According to a third aspect of this application, an electronic device is also provided, including a memory and a processor, the memory being used to store a computer program; the processor being used to execute the computer program to implement the method according to the first aspect of this application.

[0019] According to a fourth aspect of this application, a computer-readable storage medium is also provided, on which a computer program is stored, the computer program implementing the method according to the first aspect of this application when executed by a processor.

[0020] One beneficial effect of this application embodiment is that it can obtain the rotation angle of the target plane relative to the target coordinate plane in the camera coordinate system; wherein, the target plane is obtained based on the end point cloud data of the grasped object, and the end point cloud data is obtained based on the initial point cloud data of the grasped object and the positioning result of the end; a rotation matrix corresponding to the rotation angle is obtained; based on the rotation matrix, the offset corresponding to the target coordinate axis is calculated; wherein, the target coordinate axis is perpendicular to the target coordinate plane; based on the rotation angle of the target coordinate plane and the offset corresponding to the target coordinate axis, the pose of the grasped object in the camera coordinate system is obtained. This solves the technical problem of low accuracy of the grasped object pose obtained in the prior art, thereby avoiding situations such as inability to locate the grasped object or tearing of the grasped object during the grasping process due to the inability to obtain the accurate pose of the grasped object, bringing many conveniences to industrial production.

[0021] Other features and advantages of the embodiments of this application will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the present application and, together with their description, serve to explain the principles of the embodiments of the present application.

[0023] Figure 1 This is a flowchart of a method for determining the pose of a grasped object according to some embodiments of this application;

[0024] Figure 2 This is a flowchart of another method for determining the pose of a grasped object according to some embodiments of this application;

[0025] Figure 3 This is an illustration of initial point data obtained after scanning and grasping an object according to some embodiments of this application;

[0026] Figure 4A This is an illustration of a spatial extent model in some embodiments of this application;

[0027] Figure 4B Based on Figure 4A The spatial extent model shown is from Figure 3 An illustration of the terminal point cloud data extracted from the initial point cloud data;

[0028] Figure 5 This is a flowchart of another method for determining the pose of a grasped object according to some embodiments of this application;

[0029] Figure 6A Yes Figure 4B The illustration shows the projected image obtained by vertically projecting the end point cloud data shown.

[0030] Figure 6B This is an illustration of a planar range model in some embodiments of this application;

[0031] Figure 7 These are illustrations of the external surface of the grasped object's end and the target plane in some embodiments of this application;

[0032] Figure 8 This is a flowchart of another method for determining the pose of a grasped object according to some embodiments of this application;

[0033] Figure 9 This is a flowchart of another method for determining the pose of a grasped object according to some embodiments of this application;

[0034] Figure 10 This is a schematic diagram of the third point cloud data of the target plane and the fourth point cloud data obtained from the third point cloud data and the target rotation matrix in some embodiments of this application;

[0035] Figure 11AThis is a schematic diagram of the first side of the set coordinate axis and the second projected profile in some embodiments of this application;

[0036] Figure 11B yes Figure 11A A schematic diagram of the rotated image corresponding to the second projected contour in the middle;

[0037] Figure 12 This is a functional structure block diagram of a device for determining the pose of a grasped object according to some embodiments of this application;

[0038] Figure 13 This is a schematic diagram of the hardware structure of an electronic device according to some embodiments of this application. Detailed Implementation

[0039] Various exemplary embodiments of this application will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0040] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0041] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0042] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0043] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0044] Before introducing the embodiments of this application, the principle of the robotic arm grasping the FPC will be briefly explained. Please refer to the following formula (1):

[0045]

[0046] Where T represents the transformation relationship between the two coordinate systems, b represents the reference coordinate system of the robotic arm, t represents the target object (i.e. the object to be grasped), such as FPC, e represents the end effector of the robotic arm, and c represents the camera coordinate system.

[0047] In formula (1), It represents the transformation relationship from the FPC to the robot arm's reference coordinate system. It is the relationship matrix between the robotic arm's end effector and the reference coordinate system. It refers to the transformation relationship from the camera coordinate system to the coordinate system of the robotic arm's end effector. It is the transformation relationship from the target object to the camera coordinate system.

[0048] in, Provided by the robotic arm controller, Given by the hand-eye calibration matrix, It needs to be calculated manually based on the point cloud of the FPC. This application is specifically aimed at... The calculation process is proposed.

[0049] Hereinafter, various embodiments and examples according to the present invention will be described with reference to the accompanying drawings.

[0050] <Method Example>

[0051] Figure 1 This is a flowchart of a method for determining the pose of a grasped object according to some embodiments of this application, such as... Figure 1 As shown, the method includes the following steps S110 to S140.

[0052] Step S110: Obtain the rotation angle of the target plane relative to each coordinate plane in the camera coordinate system; wherein, the target plane is obtained based on the end point cloud data of the grasped object, and the end point cloud data is obtained based on the initial point cloud data of the grasped object and the positioning result of the end.

[0053] When obtaining the number of point clouds at the end, such as Figure 2 As shown, it can be obtained through the following steps 210 to S220.

[0054] Step S210: Obtain the initial point cloud data of the object to be grasped.

[0055] A point cloud scanning device can be used to scan the object being grasped to obtain a set of point data on the surface of the object, i.e., the initial point cloud data.

[0056] For example, a point cloud scanning device is used to scan the surface of a device, and the result is as follows: Figure 3 The point cloud data shown is used as the initial point cloud data.

[0057] Step S220: Based on the initial point cloud data and the preset spatial range model, extract the first point cloud data located within the spatial range model as the end point cloud data.

[0058] After obtaining the initial point cloud data, such as Figure 4A As shown, a preset spatial range model 40 can be displayed in the image where the initial point cloud data is located. Figure 4A(Displayed in point form), the spatial extent model 40 can be a box structure. The first point cloud data within the area enclosed by the spatial extent model 40 is extracted to obtain, as shown below. Figure 4B The terminal point cloud data shown is 41.

[0059] The location and size of the spatial extent model can be fixed, and can be set by those skilled in the art according to the actual situation. This application does not limit this.

[0060] To increase the flexibility of positioning the end of the grasped object, the position and size of the spatial extent model can be variable, meaning they are adjustable. In this case, the user can modify the position and / or size of the spatial extent model through corresponding operations to change the first point cloud data enclosed by the spatial extent model.

[0061] In some embodiments, in order to further improve the accuracy of the positioning of the end of the grasped object, the end point cloud data can be redetermined based on the end point cloud data obtained in step S220 with higher precision.

[0062] After performing step S220, as Figure 5 As shown, steps S510 to S530 can also be performed.

[0063] Step S510: Vertically project the first point cloud data to obtain the first projected image.

[0064] like Figure 6A As shown, for Figure 4B The terminal point cloud data 40 shown is vertically projected to obtain the first projected image 60.

[0065] Step S520: Based on the first projected image and the preset planar range model, extract the target projected image located within the planar range model.

[0066] like Figure 6B As shown, a preset planar range model 61 can be displayed on the first projected image 60. The planar range model 61 can be a rectangular frame. In this case, the target projected image located within the planar range model 61 is captured.

[0067] The position and size of the planar range model can be fixed, and can be set by those skilled in the art according to the actual situation. This application does not limit this.

[0068] To increase the flexibility of positioning the end of the grasped object, the position and size of the planar extent model can be variable, meaning they are adjustable. In this case, the user can modify the position and / or size of the planar extent model through corresponding operations to change the projected image of the target circled by the planar extent model.

[0069] Step S530: Obtain the second point cloud data corresponding to the target projection image, and use the second point cloud data as the end point cloud data.

[0070] The target projection image can be mapped onto the first point cloud data to obtain the second point cloud data.

[0071] Specifically, there is a correspondence between the first point cloud data and the first projected image. This correspondence means that the target image of any target region in the first projected image is obtained by projecting the target point cloud data corresponding to the target image in the first point cloud data. Mapping the target projected image into the first point cloud data means determining the second point cloud data corresponding to the target projected image in the first point cloud data according to the above correspondence, and using the second point cloud data as the end point cloud data.

[0072] Through steps S510 to S530, the end of the grasped object can be further precisely located, improving the accuracy of the end of the grasped object location, so as to reduce the amount of data to be processed in subsequent steps (corresponding to steps S120 and S130) (in actual cases, the processing time of the overall processing scheme of this application embodiment can be shortened to 0.1s to 0.5s), thereby improving processing efficiency.

[0073] The end-effector point cloud data is a set of point data representing the surface of the object's end. In reality, the end-effector surface is often not an ideal plane; it may have one or more problems such as unevenness, creases in the middle, or multiple bends. In such cases, a plane can be fitted based on the end-effector point cloud data, and the fitted plane can be used as the target plane.

[0074] For example, such as Figure 7 As shown, the appearance surface S1 of the grasped object represented by the end point cloud data has problems such as uneven surface and creases in the middle of the surface; the target plane S2 obtained by performing plane fitting based on the end point cloud data is the ideal plane obtained after removing the above problems from the appearance surface S1 of the grasped object.

[0075] The process of performing plane fitting can involve processing the terminal point cloud data using a preset plane fitting algorithm. The preset plane fitting algorithm can be set by those skilled in the art according to the actual situation, for example, it can be a point cloud fitting algorithm from a Point Cloud Library (PCL).

[0076] The camera coordinate system includes the X-axis, Y-axis, and Z-axis, which are perpendicular to each other. The coordinate planes in the camera coordinate system include the XOY plane, the XOZ plane, and the YOZ plane.

[0077] When obtaining the rotation angle of the target plane relative to each coordinate plane in the camera coordinate system, it can be obtained in the following way.

[0078] With the coordinate plane being the XOY plane, the target normal vector n0 = (a, b, c) of the target plane and the first normal vector n2 = (0, 0, c) of the XOY plane are obtained. The first rotation angle of the target plane relative to the XOY plane in the camera coordinate system can be calculated using the following formula:

[0079] First rotation angle R z =arccos[|n0*n1| / (|n0|*|n1|)].

[0080] With the coordinate plane being the XOZ plane, the target normal vector n0 = (a, b, c) of the target plane and the second normal vector n2 = (0, b, 0) of the XOZ plane are obtained. The second rotation angle of the target plane relative to the XOZ plane in the camera coordinate system can be calculated using the following formula:

[0081] Second rotation angle R y =arccos[|n0*n2| / (|n0|*|n2|)].

[0082] With the coordinate plane being the YOZ plane, the target normal vector n0 = (a, b, c) of the target plane and the third normal vector n3 = (a, 0, 0) of the YOZ plane are obtained. The third rotation angle of the target plane relative to the YOZ plane in the camera coordinate system is calculated using the following formula:

[0083] Third rotation angle R x =arccos[|n0*n3| / (|n0|*|n3|)].

[0084] Step S120: Obtain the rotation matrix corresponding to the rotation angle.

[0085] The rotation matrix corresponding to the rotation angle can be calculated using the following formula.

[0086]

[0087] Step S130: Calculate the offset corresponding to the target coordinate axis based on the rotation matrix; wherein the target coordinate axis is perpendicular to the target coordinate plane.

[0088] In some embodiments, such as Figure 8 As shown, the execution process of this step may include the following steps S810 to S840.

[0089] Step S810: Determine the target rotation matrix based on the first rotation matrix corresponding to the first rotation angle and the second rotation matrix corresponding to the second rotation angle; wherein, the first rotation angle is the rotation angle of the target plane relative to the first target coordinate plane in the camera coordinate system, and the second rotation angle is the rotation angle of the target plane relative to the second target coordinate plane in the camera coordinate system.

[0090] For example, if the first rotation angle is Rx and the second rotation angle is Ry, the target rotation matrix M0 can be calculated using the following formula:

[0091]

[0092] Of course, it is understood that the above description only uses the example of the first rotation angle being Rx and the second rotation angle being Ry to illustrate the acquisition of the target rotation matrix M0. In the embodiments of this application, the first rotation angle and the second rotation angle include, but are not limited to, the examples listed above. For example, the first rotation angle and the second rotation angle can also be Rx and Rz, or the first rotation angle and the second rotation angle can also be Ry and Rz, respectively.

[0093] Step S820: Obtain the target rotation angle based on the target rotation matrix.

[0094] In some embodiments, such as Figure 9 As shown, the process of obtaining the target rotation angle based on the target rotation matrix includes the following steps S910 to S930.

[0095] Step S910: Obtain the product of the target rotation matrix and the third point cloud data of the target plane to obtain the fourth point cloud data; wherein, the third point cloud data is obtained by performing plane fitting on the end point cloud data.

[0096] The target plane is obtained by performing plane fitting on the terminal point cloud data, and the number of point clouds corresponding to the target plane is the third point cloud data C0.

[0097] Taking the target rotation matrix M0 in the above embodiment as an example, the fourth point cloud data C1 can be calculated using the following formula: C1 = M0 * C0. In this case, the fourth point cloud data C1 is parallel to the XOY plane, which is equivalent to rotating the target plane to be parallel to the XOY plane. Figure 10As shown, the third point cloud data C0 is rotated to be parallel to the XOY plane to obtain the fourth point cloud data C1.

[0098] Step S920: Perform vertical projection on the fourth point cloud data to obtain the second projected image.

[0099] For example, the fourth point cloud data is projected onto the plane (i.e., the XOY plane) in the camera coordinate system at z=0 to obtain the second projected image, thus achieving the purpose of affine transformation of the point cloud data into a two-dimensional image.

[0100] Step S930: Obtain the target angle between the first side in the second projected image and the set coordinate axis, and use the target angle as the target rotation angle; wherein, the length of the first side is greater than or equal to the length of other sides in the second projected image, and the angle between the set coordinate axis and the first side is less than or equal to the angle between the set coordinate axis and any other side.

[0101] For example, such as Figure 11A As shown, the first side is the longer side in the second projected contour. With the first rotation angle being Rx and the second rotation angle being Ry, the coordinate axis is set to the Y-axis. In this case, the target angle α between the first side L1 in the second projected image and the Y-axis is obtained, and this target angle is used as the target rotation angle α.

[0102] It should be noted that since the angle between the set coordinate axis and the first side is less than or equal to the angle between the set coordinate axis and any other side, the robotic arm can achieve the purpose of obtaining the posture of the grasped object in the camera coordinate system by rotating at the minimum angle during the above rotation process, thereby reducing the range of motion of the robotic arm and preventing the robotic arm from colliding with other objects as much as possible.

[0103] Step S830: Obtain the coordinates of the center point of the end based on the target rotation angle.

[0104] The end of the object being grasped is flat or nearly flat. A second projected image can be rotated around a rotation axis to obtain a rotated image; the rotation axis is perpendicular to the second projected image. Then, the planar coordinates of the center point of the rotated image are obtained. Based on the target plane and the target rotation matrix, the height coordinates of the end point along the rotation axis are determined. Finally, based on the aforementioned planar and height coordinates, the coordinates of the center point of the end point are obtained.

[0105] The outline of the second projected image is filled, and the filled image is rotated around the Z-axis (which is perpendicular to the second projected image) by a target rotation angle α to obtain a rotated image. The first side L1 of the rotated image is parallel to the Y-axis (e.g., ...). Figure 11B As shown). In this case, obtain the planar coordinates (x, y) of the center point of the rotated image. c ,y cObtain the height coordinate z of any point P0 in the target plane. c =M0*P0, where P is the coordinate of the center point of the end. c0 =(x c ,y c , z c ).

[0106] Step S840: Determine the offset corresponding to the target coordinate axis based on the inverse matrix of the target rotation matrix and the coordinates of the center point.

[0107] Multiply the center point coordinates of the end point with the inverse of the target rotation matrix, and use the coordinate value corresponding to each coordinate axis in the result of the multiplication as the offset corresponding to that coordinate axis.

[0108] For example, P c0 Multiplying by the inverse of M0 yields P. c1 (x c1 ,y c1 , z c1 ). x c1 The offset corresponding to the x-axis, y c1 The offset corresponding to the y-axis and z-axis c1 This is the offset corresponding to the z-axis.

[0109] Step S140: Obtain the pose of the grasped object in the camera coordinate system based on the rotation angle and offset corresponding to the target coordinate axis.

[0110] The object is rotated around the target coordinate axis by the corresponding rotation angle, and the object is moved along the target coordinate axis by the corresponding offset to obtain the pose of the object in the camera coordinate system.

[0111] One beneficial effect of this application embodiment is that it can obtain the rotation angle of the target plane relative to the target coordinate plane in the camera coordinate system; wherein, the target plane is obtained based on the end point cloud data of the grasped object, and the end point cloud data is obtained based on the initial point cloud data of the grasped object and the positioning result of the end; a rotation matrix corresponding to the rotation angle is obtained; based on the rotation matrix, the offset corresponding to the target coordinate axis is calculated; wherein, the target coordinate axis is perpendicular to the target coordinate plane; based on the rotation angle of the target coordinate plane and the offset corresponding to the target coordinate axis, the pose of the grasped object in the camera coordinate system is obtained. This solves the technical problem of low accuracy of the grasped object pose obtained in the prior art, thereby avoiding situations such as inability to locate the grasped object or tearing of the grasped object during the grasping process due to the inability to obtain the accurate pose of the grasped object, bringing many conveniences to industrial production.

[0112] The embodiments of this application can combine three-dimensional point cloud data and two-dimensional images to determine the pose of the grasped object in the camera coordinate system, thereby effectively improving the accuracy and efficiency of the processing results.

[0113] For example, if the object to be grasped is a flexible flat cable, and the high degree of freedom at the end of the flexible flat cable makes it difficult to determine its pose, or if the flexible flat cable is integrated with other components in the VR product, resulting in low pose accuracy, this embodiment can obtain the accurate pose of the end of the flexible flat cable (i.e., the position to be grasped when grasping the flexible flat cable) in the camera coordinate system based on the end point cloud data of the end of the flexible flat cable. This allows for accurate positioning of the end of the flexible flat cable. In some cases, this embodiment can shorten the processing time for determining the pose to 0.1s to 0.5s, further reducing the impact of the degree of freedom at the end of the flexible flat cable on the result of determining its pose. Grasping the flexible flat cable based on its accurate pose can effectively avoid tearing and other damage to the cable.

[0114] <Equipment Example>

[0115] Figure 12 This is a functional structural block diagram of a device for determining the pose of a grasped object according to some embodiments of this application. For example... Figure 12 As shown, the pose determination device 1200 for grasping objects includes a first acquisition module 1210, an acquisition module 1220, a calculation module 1230, and a processing module 1240.

[0116] The first acquisition module 1210 is used to acquire the rotation angle of the target plane relative to the target coordinate plane in the camera coordinate system; wherein, the target plane is obtained based on the end point cloud data of the end of the grasped object, and the end point cloud data is obtained based on the initial point cloud data of the grasped object and the positioning result of the end.

[0117] Module 1220 is used to obtain the rotation matrix corresponding to the rotation angle.

[0118] The calculation module 1230 is used to calculate the offset corresponding to the target coordinate axis based on the rotation matrix; wherein the target coordinate axis is perpendicular to the target coordinate plane.

[0119] The processing module 1240 is used to obtain the pose of the grasped object in the camera coordinate system based on the rotation angle of the target coordinate plane and the offset corresponding to the target coordinate axis.

[0120] Optionally, the device further includes: a second acquisition module, used to acquire initial point cloud data of the object to be grasped before the first acquisition module acquires the rotation angle of the target plane relative to the target coordinate plane in the camera coordinate system; and a first interception module, used to intercept the first point cloud data located within the spatial range model as the end point cloud data based on the initial point cloud data and a preset spatial range model.

[0121] Optionally, the device further includes: a first projection module, used to vertically project the first point cloud data after the first interception module intercepts the first point cloud data located within the spatial range model as the end point cloud data based on the initial point cloud data and the preset spatial range model, to obtain a first projected image; a second interception module, used to intercept the target projected image located within the plane range model based on the first projected image and the preset plane range model; and a third acquisition module, used to acquire the second point cloud data corresponding to the target projected image and use the second point cloud data as the end point cloud data.

[0122] Optionally, the calculation module includes: a first determining submodule, used to determine the target rotation matrix based on the first rotation matrix corresponding to the first rotation angle and the second rotation matrix corresponding to the second rotation angle; wherein the first rotation angle is the rotation angle of the target plane relative to the first target coordinate plane in the camera coordinate system, and the second rotation angle is the rotation angle of the target plane relative to the second target coordinate plane in the camera coordinate system; a first obtaining submodule, used to obtain the target rotation angle based on the target rotation matrix; a second obtaining submodule, used to obtain the center point coordinates of the end point based on the target rotation angle; and a second determining submodule, used to determine the offset corresponding to the target coordinate axis based on the inverse matrix of the target rotation matrix and the center point coordinates.

[0123] Optionally, the first acquisition submodule is used to: acquire the product of the target rotation matrix and the third point cloud data of the target plane to obtain the fourth point cloud data; wherein the third point cloud data is obtained by performing planar fitting on the end point cloud data; perform vertical projection on the fourth point cloud data to obtain the second projection image; acquire the target rotation angle between the first side in the second projection image and the set coordinate axis as the target rotation angle; wherein the length of the first side is greater than or equal to the length of other sides in the second projection image, and the angle between the set coordinate axis and the first side is less than or equal to the angle between the set coordinate axis and any other side.

[0124] Optionally, the end is flat or approximately flat, and the second acquisition submodule is used to: rotate the second projected image around the rotation coordinate axis by the target rotation angle to obtain a rotated image; wherein the rotation coordinate axis is perpendicular to the second projected image; acquire the planar coordinates of the center point of the rotated image; determine the height coordinates of the end in the direction of the rotation coordinate axis according to the target plane and the target rotation matrix; and acquire the center point coordinates of the end according to the planar coordinates and the height coordinates.

[0125] The electronic device 1200 can be a robotic arm or an industrial robot.

[0126] Therefore, in this embodiment, the rotation angle of the target plane relative to the target coordinate plane in the camera coordinate system can be obtained. The target plane is obtained based on the end point cloud data of the grasped object, which is obtained based on the initial point cloud data of the grasped object and the positioning result of the end. A rotation matrix corresponding to the rotation angle is obtained. Based on the rotation matrix, the offset corresponding to the target coordinate axis is calculated. The target coordinate axis is perpendicular to the target coordinate plane. Based on the rotation angle of the target coordinate plane and the offset corresponding to the target coordinate axis, the pose of the grasped object in the camera coordinate system is obtained. This solves the technical problem of low accuracy in the pose of the grasped object obtained in the prior art, thus avoiding situations such as inability to locate the grasped object or tearing of the grasped object during the grasping process due to the inability to obtain the accurate pose of the grasped object, bringing many conveniences to industrial production.

[0127] Figure 13 This is a schematic diagram of the hardware structure of an electronic device according to some embodiments.

[0128] like Figure 13 As shown, the electronic device 1300 includes a processor 1310 and a memory 1320, the memory 1320 for storing an executable computer program, and the processor 1310 for executing methods as described in any of the above method embodiments under the control of the computer program.

[0129] The electronic device 1300 can be a robotic arm or an industrial robot.

[0130] Each module of the above electronic device 1200 can be implemented by the processor 1310 in this embodiment executing the computer program stored in the memory 1320, or it can be implemented by other circuit structures, which are not limited here.

[0131] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0132] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0133] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0134] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0135] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0136] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0137] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0139] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A method for determining the pose of a grasped object, characterized in that, include: Obtain the rotation angle of the target plane relative to the target coordinate plane in the camera coordinate system; wherein, the target plane is obtained based on the end point cloud data of the grasped object, the end point cloud data is obtained based on the initial point cloud data of the grasped object and the positioning result of the end of the grasped object, the target coordinate plane includes a first target coordinate plane and a second target coordinate plane determined according to the X-axis, Y-axis and Z-axis of the camera coordinate system, and the rotation angle includes a first rotation angle of the target plane relative to the first target coordinate plane and a second rotation angle of the target plane relative to the second target coordinate plane; Obtain the first rotation matrix corresponding to the first rotation angle and the second rotation matrix corresponding to the second rotation angle; Determine the target rotation matrix based on the first rotation matrix and the second rotation matrix; The product of the target rotation matrix and the third point cloud data of the target plane is obtained to obtain the fourth point cloud data; wherein, the third point cloud data is obtained by performing plane fitting on the end point cloud data; The fourth point cloud data is vertically projected to obtain a second projected image; The target rotation angle is obtained by taking the angle between the first side in the second projected image and the target of the set coordinate axis; wherein the length of the first side is greater than or equal to the length of the other sides in the second projected image, and the angle between the set coordinate axis and the first side is less than or equal to the angle between the set coordinate axis and any other side. The coordinates of the center point of the end of the grasped object are obtained based on the target rotation angle. Based on the inverse of the target rotation matrix and the center point coordinates, determine the offset of the center point coordinates of the end of the grasped object on the target coordinate axis; wherein, the target coordinate axis is perpendicular to the target coordinate plane; The object is rotated around the target coordinate axis by the corresponding rotation angle, and the object is moved along the target coordinate axis by the corresponding offset to obtain the pose of the object in the camera coordinate system.

2. The method according to claim 1, characterized in that, Before obtaining the rotation angle of the target plane relative to the target coordinate plane in the camera coordinate system, the method further includes: Obtain the initial point cloud data of the grasped object; Based on the initial point cloud data and the preset spatial range model, the first point cloud data located within the spatial range model is extracted as the final point cloud data.

3. The method according to claim 2, characterized in that, The step of extracting the first point cloud data located within the spatial range model as the final point cloud data based on the initial point cloud data and the preset spatial range model includes: The first point cloud data is vertically projected to obtain a first projected image; Based on the first projected image and the preset planar range model, a target projected image located within the planar range model is extracted; Obtain the second point cloud data corresponding to the target projection image, and use the second point cloud data as the end point cloud data.

4. The method according to claim 1, characterized in that, The end of the grasped object is flat, and obtaining the center point coordinates of the end of the grasped object based on the target rotation angle includes: The second projected image is rotated around the rotation axis by the target rotation angle to obtain a rotated image; wherein the rotation axis is perpendicular to the second projected image; Obtain the planar coordinates of the center point of the rotated image; Based on the target plane and the target rotation matrix, determine the height coordinates of the end of the grasped object in the direction of the rotation coordinate axis; Based on the planar coordinates and height coordinates, the center point coordinates of the end of the grasped object are obtained.

5. A device for determining the pose of a grasped object, characterized in that, include: The first acquisition module is used to acquire the rotation angle of the target plane relative to the target coordinate plane in the camera coordinate system; wherein, the target plane is obtained based on the end point cloud data of the grasped object, the end point cloud data is obtained based on the initial point cloud data of the grasped object and the positioning result of the end of the grasped object, the target coordinate plane includes a first target coordinate plane and a second target coordinate plane determined according to the X-axis, Y-axis and Z-axis of the camera coordinate system, and the rotation angle includes a first rotation angle of the target plane relative to the first target coordinate plane and a second rotation angle of the target plane relative to the second target coordinate plane; The module is used to obtain the first rotation matrix corresponding to the first rotation angle and the second rotation matrix corresponding to the second rotation angle; The calculation module is configured to: determine a target rotation matrix based on the first rotation matrix and the second rotation matrix; obtain a fourth point cloud data by multiplying the target rotation matrix and the third point cloud data of the target plane; wherein the third point cloud data is obtained by plane fitting of the end point cloud data; perform vertical projection on the fourth point cloud data to obtain a second projection image; obtain the target angle between the first side in the second projection image and a set coordinate axis as the target rotation angle; wherein the length of the first side is greater than or equal to the length of other sides in the second projection image, and the angle between the set coordinate axis and the first side is less than or equal to the angle between the set coordinate axis and any other side; obtain the center point coordinates of the end of the grasped object based on the target rotation angle; and determine the offset of the center point coordinates of the end of the grasped object on the target coordinate axis based on the inverse matrix of the target rotation matrix and the center point coordinates; wherein the target coordinate axis is perpendicular to the target coordinate plane. The processing module is used to rotate the grasped object around the target coordinate axis by a corresponding rotation angle, and move the grasped object along the target coordinate axis by a corresponding offset, so as to obtain the pose of the grasped object in the camera coordinate system.

6. An electronic device comprising a memory and a processor, the memory being configured to store a computer program; the processor being configured to execute the computer program to implement the method according to any one of claims 1-4.

7. A computer-readable storage medium storing a computer program thereon, the computer program implementing the method according to any one of claims 1-4 when executed by a processor.

Images