Method and apparatus for controlling a robot

Abstract

A robot control method and device are disclosed. The method comprises: obtaining a first point cloud sequence of a target object in a coordinate system of a depth camera, wherein the first point cloud sequence comprises point clouds of the target object at multiple angles; obtaining a pose sequence of an end of a robot arm in a base coordinate system; determining a first transformation matrix of each point cloud in the first point cloud sequence to a predetermined point cloud, wherein the predetermined point cloud is a first obtained point cloud in the first point cloud sequence; obtaining a second transformation matrix from the coordinate system of the depth camera to the coordinate system of the end of the robot arm based on the first transformation matrices corresponding to a predetermined number of point clouds in the first point cloud sequence and the pose sequence; obtaining a second point cloud sequence of the end of the robot arm in the base coordinate based on the first point cloud sequence, the second transformation matrix and the pose sequence; and determining a third point cloud for controlling the movement of the end of the robot arm based on the second point cloud sequence.

Description

Technical Field

[0001] This disclosure generally relates to the field of robotics, and more specifically, to methods and apparatus for controlling robots. Background Technology

[0002] Robots have a wide range of applications, including industrial manufacturing, logistics and warehousing, healthcare, service industries, agriculture, and the military. Regardless of the application, accurate visual positioning is crucial for the navigation and manipulation of the robot's end effector, especially in the field of assistive medical care. Currently, existing methods for hand-eye alignment in robots suffer from the following shortcomings:

[0003] 1. Cumbersome Operation: Traditional hand-eye calibration requires the use of specialized calibration tools (such as checkerboard patterns, circular targets, etc.) to calibrate the depth camera and the robotic arm end effector before multi-view photography of the object. This process is time-consuming and labor-intensive. If robots are used in the field of assistive medical care, the cumbersome operation of this hand-eye calibration process is not conducive to the rapid deployment of surgery.

[0004] 2. Poor flexibility: Once the relative position of the depth camera and the robotic arm changes (e.g., due to equipment adjustment or accidental collision), hand-eye calibration needs to be re-performed, which reduces the flexibility and practicality of the system.

[0005] 3. High random bias: In the process of stitching together point clouds of a target object from multiple perspectives, the traditional incremental registration method usually uses a point cloud as a reference, resulting in high random bias, which affects the final positioning accuracy. If the robot is used in the field of assisted medical care, it will be difficult to meet the requirements of high-precision medical surgery. Summary of the Invention

[0006] The embodiments of this disclosure provide a robot control method and apparatus that can solve at least one of the above-mentioned problems.

[0007] In one general aspect, a robot control method is provided, comprising: acquiring a first point cloud sequence of a target object in the coordinate system of a depth camera, wherein the first point cloud sequence contains point clouds of the target object at multiple angles; acquiring a pose sequence of the robot's end effector in a base coordinate system; determining a first transformation matrix from each point cloud in the first point cloud sequence to a predetermined point cloud, wherein the predetermined point cloud is the first point cloud acquired in the first point cloud sequence; obtaining a second transformation matrix from the coordinate system of the depth camera to the coordinate system of the robot end effector based on the first transformation matrix and the pose sequence corresponding to a predetermined number of point clouds in the first point cloud sequence; obtaining a second point cloud sequence of the robot end effector in the base coordinate system based on the first point cloud sequence, the second transformation matrix, and the pose sequence; and determining a third point cloud for controlling the motion of the robot end effector based on the second point cloud sequence.

[0008] Optionally, based on the first transformation matrix corresponding to a predetermined number of point clouds in the first point cloud sequence and the pose sequence, a second transformation matrix is ​​obtained from the coordinate system of the depth camera to the coordinate system of the robotic arm end effector, including: obtaining the poses corresponding to a predetermined number of point clouds from the pose sequence; determining a first parameter based on the poses corresponding to each pair of point clouds in the predetermined number of point clouds; determining a second parameter based on the first transformation matrix corresponding to each pair of point clouds in the predetermined number of point clouds; and obtaining the second transformation matrix from the coordinate system of the depth camera to the coordinate system of the robotic arm end effector based on the first parameter and the second parameter.

[0009] Optionally, based on the first point cloud sequence, the second transformation matrix, and the pose sequence, a second point cloud sequence in base coordinates is obtained for the robotic arm end effector, including: for each point cloud in the first point cloud sequence, the following processing is performed: multiplying the pose corresponding to the current point cloud, the second transformation matrix, and the current point cloud to obtain the second point cloud corresponding to the current point cloud; and obtaining the second point cloud sequence based on the second point cloud corresponding to each point cloud in the first point cloud sequence.

[0010] Optionally, based on the second point cloud sequence, determining the third point cloud for controlling the end effector motion of the robotic arm includes: for each second point cloud in the second point cloud sequence, performing the following processing: obtaining the third transformation matrix from the second point cloud to the target point cloud; processing the third transformation matrix by interpolation to obtain a fourth transformation matrix that transforms the second point cloud to the intermediate coordinate system; transforming the second point cloud and the target point cloud to the intermediate coordinate system based on the fourth transformation matrix; merging the second point cloud and the target point cloud transformed to the intermediate coordinate system to obtain the updated target point cloud; and after processing all the second point clouds in the second point cloud sequence, determining the updated target point cloud as the third point cloud.

[0011] Optionally, the third transformation matrix is ​​processed by interpolation to obtain a fourth transformation matrix that transforms the second point cloud to the intermediate coordinate system. This includes: converting the rotation information in the third transformation matrix into quaternions and performing spherical linear interpolation between the quaternions and unit quaternions to obtain a rotation matrix; performing linear interpolation on the position information in the third transformation matrix to obtain a displacement vector; and obtaining the fourth transformation matrix that transforms the second point cloud to the intermediate coordinate system based on the rotation matrix and the displacement vector.

[0012] Optionally, based on the fourth transformation matrix, the second point cloud and the target point cloud are transformed to the intermediate coordinate system, including: transforming the second point cloud to the intermediate coordinate system by multiplying the fourth transformation matrix and the second point cloud; and transforming the target point cloud to the intermediate coordinate system by multiplying the inverse of the fourth transformation matrix and the third transformation matrix with the target point cloud.

[0013] In another general aspect, a robot control device is provided, comprising: a first acquisition unit configured to acquire a first point cloud sequence of a target object in the coordinate system of a depth camera, wherein the first point cloud sequence includes point clouds of the target object at multiple angles; a second acquisition unit configured to acquire a pose sequence of the robot's end effector in a base coordinate system; a first determination unit configured to determine a first transformation matrix from each point cloud in the first point cloud sequence to a predetermined point cloud, wherein the predetermined point cloud is the first point cloud acquired in the first point cloud sequence; a third acquisition unit configured to obtain a second transformation matrix from the coordinate system of the depth camera to the coordinate system of the robot's end effector based on the first transformation matrix and the pose sequence corresponding to a predetermined number of point clouds in the first point cloud sequence; a fourth acquisition unit configured to obtain a second point cloud sequence of the robot's end effector in the base coordinate system based on the first point cloud sequence, the second transformation matrix, and the pose sequence; and a second determination unit configured to determine a third point cloud for controlling the movement of the robot's end effector based on the second point cloud sequence.

[0014] Optionally, the third acquisition unit is further configured to acquire poses corresponding to a predetermined number of point clouds from the pose sequence; determine a first parameter based on the poses corresponding to each pair of point clouds in the predetermined number of point clouds; determine a second parameter based on the first transformation matrix corresponding to each pair of point clouds in the predetermined number of point clouds; and obtain a second transformation matrix from the coordinate system of the depth camera to the coordinate system of the robotic arm end effector based on the first parameter and the second parameter.

[0015] Optionally, the fourth acquisition unit is further configured to perform the following processing for each point cloud in the first point cloud sequence: multiply the pose corresponding to the current point cloud, the second transformation matrix, and the current point cloud to obtain the second point cloud corresponding to the current point cloud; and obtain the second point cloud sequence based on the second point cloud corresponding to each point cloud in the first point cloud sequence.

[0016] Optionally, the second determining unit is further configured to perform the following processing for each second point cloud in the second point cloud sequence: obtain a third transformation matrix from the second point cloud to the target point cloud; process the third transformation matrix by interpolation to obtain a fourth transformation matrix that transforms the second point cloud to the intermediate coordinate system; transform the second point cloud and the target point cloud to the intermediate coordinate system based on the fourth transformation matrix; merge the second point cloud and the target point cloud transformed to the intermediate coordinate system to obtain the updated target point cloud; and after processing all the second point clouds in the second point cloud sequence, determine the updated target point cloud as the third point cloud.

[0017] Optionally, the second determining unit is further configured to convert the rotation information in the third transformation matrix into quaternions and perform spherical linear interpolation between the quaternions and the unit quaternions to obtain the rotation matrix; perform linear interpolation on the position information in the third transformation matrix to obtain the displacement vector; and obtain a fourth transformation matrix that transforms the second point cloud to the intermediate coordinate system based on the rotation matrix and the displacement vector.

[0018] Optionally, the second determining unit is further configured to transform the second point cloud into an intermediate coordinate system by multiplying the fourth transformation matrix and the second point cloud; and to transform the target point cloud into an intermediate coordinate system by multiplying the inverse of the fourth transformation matrix and the third transformation matrix with the target point cloud.

[0019] In another general aspect, a computer-readable storage medium is provided for storing instructions, wherein when the instructions are executed by at least one computing device, they cause at least one computing device to perform a control method for any of the robots described above.

[0020] In another general aspect, a system is provided that includes at least one computing device and at least one storage device for storing instructions, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to perform a control method for any of the robots described above.

[0021] In another general aspect, a computer program product is provided, including computer instructions that, when executed by a processor, implement a control method for any of the robots described above.

[0022] According to the robot control method and apparatus of the embodiments of this disclosure, the third point cloud of the end effector of the robotic arm can be acquired in real time by utilizing the geometric features of the target object itself (i.e., the first point cloud sequence and its corresponding transformation matrix), so that the hand-eye calibration of the depth camera and the end effector of the robotic arm can be completed automatically without pre-calibration, saving the tedious calibration process; moreover, this disclosure does not rely on external calibration tools and does not require pre-calibration, so it can adapt to changes in the position of the depth camera or the robotic arm, and has higher flexibility; furthermore, in point cloud registration, this disclosure uses the first point cloud acquired in the first point cloud sequence as the only reference point cloud, instead of using the stitched result as the point cloud, which can overcome the error caused by the stitched result to a certain extent.

[0023] Further aspects and / or advantages of the general concept of this disclosure will be set forth in part in the description which follows, and in part will be clear from the description or may be learned by practice of the general concept of this disclosure. Attached Figure Description

[0024] The above and other objects and features of the embodiments of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings illustrating the embodiments, wherein:

[0025] Figure 1 This is a flowchart illustrating a robot control method according to an embodiment of the present disclosure;

[0026] Figure 2 This is a system flowchart illustrating a robot control method according to an embodiment of the present disclosure;

[0027] Figure 3 This is a block diagram illustrating the control device of the robot disclosed herein. Detailed Implementation

[0028] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0029] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided only to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein, which will become clear upon understanding the disclosure of this application.

[0030] As used herein, the term “and / or” includes any one of the associated listed items and any combination of any two or more.

[0031] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts should not be limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as the first component, first assembly, first region, first layer, or first part may also be referred to as the second component, second assembly, second region, second layer, or second part.

[0032] In the specification, when an element (such as a layer, region, or substrate) is described as being "on" another element, "connected to," or "bonded to" another element, the element may be directly "on" another element, directly "connected to," or "bonded to" the other element, or one or more other elements may be present in between. Conversely, when an element is described as being "directly on" another element, "directly connected to," or "directly bonded to" another element, no other elements may be present in between.

[0033] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. The terms “comprising,” “including,” and “having” indicate the presence of the described features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0034] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains upon understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formalistic manner.

[0035] Furthermore, in the description of the examples, detailed descriptions of well-known related structures or functions will be omitted when it is believed that such detailed descriptions would lead to a vague interpretation of this disclosure.

[0036] To facilitate understanding of this disclosure, a simple explanation of robot hand-eye calibration is provided here, namely, the process of determining the transformation relationship between the coordinate system of the depth camera and the coordinate system of the end effector of the robotic arm.

[0037] The robot control method and apparatus of this disclosure will be described in detail below with reference to the accompanying drawings.

[0038] This disclosure proposes a method for controlling a robot. Figure 1 This is a flowchart illustrating a robot control method according to an embodiment of the present disclosure. (Refer to...) Figure 1 The robot control method includes the following steps:

[0039] In step S101, the first point cloud sequence of the target object in the coordinate system of the depth camera is obtained, wherein the first point cloud sequence contains point clouds of the target object from multiple angles.

[0040] As an example, the robot's robotic arm can pull a depth camera to capture images of a target object from multiple perspectives and record the 3D point cloud of the target object from each perspective. The first point cloud sequence is obtained by integrating 3D point clouds from each viewpoint. The aforementioned depth camera can be replaced with other 3D sensors. The aforementioned 3D point cloud is the same as the aforementioned point cloud. The coordinate system indicating the depth camera. N Indicates the number of viewpoints.

[0041] As an example, the depth camera described above can be mounted at the end of a robot's robotic arm, such as on an end effector (e.g., a mechanical gripper, a welding tool, etc.). This mounting method is very useful in scenarios requiring high-precision operations, such as precision assembly tasks and precision surgical tasks. However, this disclosure does not limit the mounting location of the depth camera.

[0042] In step S102, the pose sequence of the robot's end effector in the base coordinate system is obtained;

[0043] As an example, a robot's robotic arm can pull a depth camera to capture images of a target object from multiple perspectives and record the transformation matrix from the coordinate system of the robotic arm's end effector to the base coordinate system from each perspective. That is, the pose of the robotic arm end effector in the base coordinate system corresponding to each point cloud. Therefore, a pose sequence can be obtained based on all poses. The above Indicator coordinate system, the above The coordinate system indicates the end effector of the robotic arm, where T represents the transformation matrix and N represents the number of viewpoints.

[0044] It should be noted that a pose sequence refers to a sequence of the position and orientation of a robotic arm's end effector arranged in chronological order or a specific order. Position information typically describes the end effector's coordinates in space, such as in a three-dimensional Cartesian coordinate system; orientation information typically describes the end effector's orientation, often represented by Euler angles (such as roll, pitch, and yaw) or rotation matrices and quaternions. In other words, a pose sequence contains both position and orientation information.

[0045] A base coordinate system is an important concept in robot programming and operation. It is a Cartesian coordinate system fixed in the robot's working environment, used to define the robot's position and orientation. The base coordinate system is usually defined at a fixed position on the robot's base or in the robot's workspace, serving as a global reference frame for the entire robot system. All robot movements and operations are described relative to this reference frame.

[0046] In step S103, a first transformation matrix from each point cloud in the first point cloud sequence to a predetermined point cloud is determined, wherein the predetermined point cloud is the first point cloud acquired in the first point cloud sequence.

[0047] As an example, a point cloud registration method can be used to determine the location of each point cloud in the first point cloud sequence. To the first point cloud acquired Transformation matrix ,in, The form can be 4× The matrix, where 4 represents 4-dimensional homogeneous coordinates. This indicates the number of point clouds.

[0048] It should be noted that point cloud registration is the process of aligning point cloud data acquired from different viewpoints to place them in a unified coordinate system; incremental registration is a method of progressively registering newly acquired point clouds with existing point clouds. The transformation matrix is ​​used to describe geometric transformations in three-dimensional space. Its dimension can be 4×4, and subsequent transformations of the three-dimensional point cloud can be achieved through matrix operations. This transformation matrix can include several parts: a 3×3 submatrix in the upper left corner to describe rotation transformations; a 3×1 column vector in the upper right corner to describe translation transformations; and the last row, usually [0, 0, 0, 1], to enable the transformation matrix to handle homogeneous coordinates. In three-dimensional space, a point cloud can be represented as (x, y, z). To facilitate simultaneous handling of rotation and translation, it needs to be extended to homogeneous coordinates (x, y, z, 1).

[0049] As an example, this embodiment uses an adjusted incremental registration method, where the reference point cloud for each point cloud is the first point cloud acquired in the first point cloud sequence, rather than the point cloud after each stitching. This can overcome the error caused by the stitching result to a certain extent.

[0050] As an example, the above registration method can employ the ICP (Iterative Closest Point) algorithm, or feature-based registration methods such as keypoint matching, distance transformation, or deep learning methods to improve the robustness and accuracy of registration. This disclosure does not limit the scope of the method.

[0051] In step S104, based on the first transformation matrix and pose sequence corresponding to a predetermined number of point clouds in the first point cloud sequence, the second transformation matrix from the coordinate system of the depth camera to the coordinate system of the end effector of the robotic arm is obtained.

[0052] As an example, the first transformation matrix and pose sequence corresponding to a predetermined number of point clouds in the first point cloud sequence can be input into the hand-eye calibration model to obtain the second transformation matrix from the coordinate system of the depth camera to the coordinate system of the robotic arm end effector. This model can be represented as follows: ,in, The first parameter is associated with the first transformation matrix. Let X be the second parameter related to the pose sequence, and let X be the second transformation matrix to be solved.

[0053] The following is based on and One method for obtaining the second transformation matrix, detailing the process:

[0054] According to embodiments of this disclosure, a second transformation matrix from the coordinate system of the depth camera to the coordinate system of the robotic arm end effector can be obtained based on a first transformation matrix and pose sequence corresponding to a predetermined number of point clouds in a first point cloud sequence, as follows: From the pose sequence, obtain the poses corresponding to a predetermined number of point clouds; determine a first parameter based on the poses corresponding to each pair of point clouds in the predetermined number of point clouds; determine a second parameter based on the first transformation matrices corresponding to each pair of point clouds in the predetermined number of point clouds; and obtain the second transformation matrix from the coordinate system of the depth camera to the coordinate system of the robotic arm end effector based on the first and second parameters. Through this embodiment, by utilizing the first transformation matrix and poses corresponding to a predetermined number of point clouds in the first point cloud sequence, relatively accurate parameters can be determined, thereby obtaining an accurate second transformation matrix.

[0055] As an example, the aforementioned predetermined quantity can be set as needed. The predetermined quantity of point clouds can be a portion of the point clouds in the first point cloud sequence or all of the point clouds in the first point cloud sequence. This disclosure does not limit this.

[0056] As an example, the above It can be represented as The above It can be represented as This disclosure does not impose any limitations on this matter.

[0057] To facilitate understanding the above and The following is a brief description. and The above represents the process of obtaining:

[0058] For any pair of point clouds in the first point cloud sequence and Since the point cloud in the base coordinate system can be obtained through the transformation matrix from the coordinate system of the depth camera to the coordinate system of the robotic arm's end effector and the transformation matrix (i.e., pose) from the coordinate system of the robotic arm's end effector to the base coordinate system, the two are aligned after the transformation.

[0059] (1)

[0060] in, This represents the transformation matrix from the coordinate system of the robotic arm's end effector to the base coordinate system. This represents the transformation matrix from the coordinate system of the depth camera to the coordinate system of the robotic arm's end effector.

[0061] Then, by using an incremental point cloud registration method (such as the ICP algorithm), we can obtain the point cloud registration from any point cloud. To the first point cloud acquired Transformation matrix Therefore:

[0062] (2)

[0063] Simplifying the above formulas (1) and (2), we can obtain:

[0064] (3)

[0065] Furthermore, the above formula (3) can be rearranged into a hand-eye calibration model. ,in, , , .

[0066] It should be noted that the simplification process of the above formulas (1) and (2) is as follows:

[0067] Formula (2) can be transformed into:

[0068] (4)

[0069] Substituting formula (4) into formula (1) yields:

[0070] (5)

[0071] By eliminating the common variables on both sides of formula (5), we can obtain:

[0072] (6)

[0073] After rearranging formula (6), we can obtain the above formula (3).

[0074] In step S105, based on the first point cloud sequence, the second transformation matrix, and the pose sequence, the second point cloud sequence of the robotic arm end effector in the base coordinates is obtained.

[0075] According to embodiments of this disclosure, a second point cloud sequence in base coordinates for the robotic arm end effector can be obtained based on a first point cloud sequence, a second transformation matrix, and a pose sequence in the following manner: For each point cloud in the first point cloud sequence, the following processing is performed: the pose corresponding to the current point cloud, the second transformation matrix, and the current point cloud are multiplied together to obtain the second point cloud corresponding to the current point cloud; based on the second point cloud corresponding to each point cloud in the first point cloud sequence, a second point cloud sequence is obtained.

[0076] As an example, you can And the solution is obtained Point cloud applied to each coordinate system of the depth camera This yields the point cloud of the robotic arm end effector in the base coordinate system corresponding to each point cloud. That is, the second point mentioned above, can be specifically represented as:

[0077] (7)

[0078] After obtaining the second point cloud corresponding to each point cloud. Then, the corresponding second point cloud sequence can be obtained. .

[0079] In step S106, a third point cloud is determined based on the second point cloud sequence to control the movement of the robotic arm's end effector.

[0080] As an example, the second point cloud sequence can be merged to obtain the third point cloud; or the second point cloud sequence can be processed as needed to obtain the third point cloud, and this disclosure does not limit this.

[0081] The above embodiments of this disclosure establish the correspondence between the geometric features of the target object itself through the registration of multi-view point clouds, namely the first point cloud sequence and its corresponding first transformation matrix. Based on the above information and the hand-eye calibration model, the transformation matrix from the coordinate system of the depth camera to the coordinate system of the robotic arm end effector is obtained. This not only avoids pre-calibration, but also unifies the solution of the transformation matrix from the coordinate system of the depth camera to the coordinate system of the robotic arm end effector with the registration of multi-view point clouds, so that the hand-eye calibration of the depth camera and the robotic arm end effector can be completed automatically, and the position of the depth camera or the robotic arm end effector can also be adapted to changes.

[0082] However, in the registration of multi-view point clouds, due to factors such as sensor noise, calibration errors, and approximations in the registration algorithm, these point clouds may not be accurately aligned in the base coordinate system. For example, traditional incremental registration methods typically register newly added point clouds onto the synthesized point cloud from the previous registration, resulting in significant random errors. To reduce random errors, this disclosure proposes an error correction method. In each incremental registration step, not only are the newly added point clouds adjusted, but the already synthesized point clouds are also adjusted through interpolation, moving both to an intermediate coordinate system. This method ensures overall optimization of the point clouds and reduces random errors.

[0083] According to embodiments of this disclosure, the determination of a third point cloud for controlling the end effector motion of a robotic arm based on a second point cloud sequence can be achieved as follows: For each second point cloud in the second point cloud sequence, the following processing is performed: obtaining a third transformation matrix from the second point cloud to the target point cloud; processing the third transformation matrix by interpolation to obtain a fourth transformation matrix that transforms the second point cloud to an intermediate coordinate system; transforming the second point cloud and the target point cloud to the intermediate coordinate system based on the fourth transformation matrix; merging the second point cloud and the target point cloud transformed to the intermediate coordinate system to obtain an updated target point cloud; and after processing all the second point clouds in the second point cloud sequence, determining the updated target point cloud as the third point cloud.

[0084] In this embodiment, in each step of incremental registration, not only is the newly added point cloud (i.e., the second point cloud in the processing) adjusted, but the already synthesized point cloud (target point cloud) is also adjusted by interpolation, so that both are moved to an intermediate coordinate system, ensuring the overall optimization of the point cloud and reducing cumulative errors.

[0085] As an example, before processing each second point cloud, the target point cloud can be initialized. For instance, the first second point cloud in the second point cloud sequence can be set as the target point cloud. Then, registration is performed step by step for each second point cloud, with the result being the first... The second point cloud For example:

[0086] First, a second point cloud is calculated using a point cloud registration algorithm (such as ICP). To the target point cloud The transformation matrix, i.e., the third transformation matrix, can be expressed as: Secondly, the third transformation matrix is ​​processed by interpolation to obtain the fourth transformation matrix that transforms the second point cloud to the intermediate coordinate system. Then, using the fourth transformation matrix, the second point cloud and the target point cloud can be transformed to the intermediate coordinate system, and the second point cloud transformed to the intermediate coordinate system can be... and target point cloud Merge them into a new target point cloud for the next registration step. .

[0087] After processing all the second point clouds in the second point cloud sequence, update the target point cloud. The third point cloud is identified, and the movement of the robot's end effector is then controlled based on this third point cloud. If used in the field of assistive medical care, surgical navigation and positioning can be performed through this third point cloud.

[0088] As an example, the above interpolation method can be spherical linear interpolation (Slerp), simple linear interpolation, or other interpolation methods, such as spherical Bezier curve interpolation, quaternion averaging, etc. The interpolation strategy can be adjusted according to specific needs, and this disclosure does not limit it.

[0089] As an example, the error correction process in this embodiment can also use a global optimization method to replace incremental registration, and optimize all point clouds as a whole to further improve the stitching accuracy. This disclosure does not limit this.

[0090] According to embodiments of this disclosure, the third transformation matrix is ​​processed by interpolation to obtain a fourth transformation matrix that transforms the second point cloud to the intermediate coordinate system. This can be achieved as follows: for the rotation information in the third transformation matrix, the rotation information is converted into quaternions, and spherical linear interpolation is performed between the quaternions and the unit quaternions to obtain the rotation matrix; for the position information in the third transformation matrix, linear interpolation is performed to obtain the displacement vector; based on the rotation matrix and the displacement vector, the fourth transformation matrix that transforms the second point cloud to the intermediate coordinate system is obtained.

[0091] In this embodiment, spherical linear interpolation using rotation information in quaternion form can conveniently and quickly obtain the rotation matrix; similarly, linear interpolation of position information can also conveniently and quickly obtain the displacement vector, and thus conveniently and quickly obtain the fourth transformation matrix.

[0092] As an example, the rotation information mentioned above can be obtained from... arrive The rotational transformation, the above position information can be from arrive Translation transformations are not limited in this disclosure.

[0093] For rotation information, the rotation matrix transforming the second point cloud to the intermediate coordinate system can be obtained through spherical linear interpolation. Specifically, the rotation matrix is... Convert to quaternion ,exist and unit quaternion Spherical linear interpolation between them can be expressed as follows:

[0094] (8)

[0095] get Then, Transform the matrix into its form to obtain the corresponding rotation matrix. .

[0096] For the location information, the displacement vector from the second point cloud to the intermediate coordinate system can be obtained through linear interpolation, and can be represented as follows:

[0097] (9)

[0098] After obtaining the corresponding rotation matrix and translation vector, the fourth transformation matrix can be obtained, which can be represented as follows: .

[0099] According to embodiments of this disclosure, the second point cloud and the target point cloud can be transformed into an intermediate coordinate system based on the fourth transformation matrix in the following manner: the second point cloud is transformed into an intermediate coordinate system by multiplying the fourth transformation matrix and the second point cloud; the target point cloud is transformed into an intermediate coordinate system by multiplying the inverse of the fourth transformation matrix and the third transformation matrix with the target point cloud.

[0100] As an example, after obtaining the fourth transformation matrix, the second point cloud and the target point cloud can be transformed to the intermediate coordinate system based on this fourth transformation matrix. Specifically:

[0101] For the second node, the second point cloud can be transformed to the intermediate coordinate system by multiplying the fourth transformation matrix and the second point cloud, as shown below:

[0102] (10)

[0103] For the target node, the target point cloud can be calculated using the fourth transformation matrix. Transformation matrix to intermediate coordinate system , means as follows:

[0104] (11)

[0105] After obtaining the above transformation matrix, the target point cloud is transformed using this transformation matrix. Transforming to the intermediate coordinate system, it can be represented as follows:

[0106] (12)

[0107] In summary, traditional hand-eye calibration methods require the use of specialized calibration tools to pre-calibrate the depth camera and the end effector of the robotic arm before multi-view shooting of the target object. This process is cumbersome, lacks flexibility, and the errors are prone to accumulate in subsequent multi-view shooting.

[0108] This disclosure provides a real-time hand-eye calibration and error correction method based on multi-view point clouds, specifically as follows: Figure 2 As shown, firstly, the robot's robotic arm can guide a 3D sensor to capture images of the target object from multiple perspectives, and record the 3D point cloud of the target object from each perspective and the transformation matrix from the coordinate system of the robotic arm's end effector to the base coordinate system. Secondly, real-time hand-eye calibration is performed, that is, based on the transformation matrix from the coordinate system of the robotic arm's end effector to the base coordinate system and the 3D point cloud from multiple perspectives, the transformation matrix from the coordinate system of the depth camera to the coordinate system of the robotic arm's end effector and the multi-view 3D point cloud of the robotic arm's end effector in the base coordinate system are obtained (i.e., the second point cloud sequence mentioned above). After obtaining the multi-view 3D point cloud of the robotic arm's end effector in the base coordinate system, error correction of the multi-view point cloud is performed to obtain the error-corrected 3D point cloud (i.e., the updated target point cloud mentioned above).

[0109] As can be seen, this disclosure utilizes the geometric features of the target object itself to automatically transform the coordinate system between the depth camera and the coordinate system of the robotic arm end effector during multi-view shooting, thereby obtaining a third point cloud for controlling the robotic arm end effector. This eliminates the cumbersome calibration process. Moreover, since it does not rely on external calibration tools and does not require pre-calibration, this disclosure can adapt to changes in the position of the depth camera or the robotic arm end effector, exhibiting greater flexibility. Furthermore, this disclosure can also introduce an interpolation method during point cloud stitching to simultaneously adjust newly added point clouds and existing point clouds, moving them to an intermediate coordinate system, reducing accumulated errors, and ensuring high-precision positioning of the target object.

[0110] It should be noted that this disclosure can be applied to the field of robot-assisted medical care, especially surgical navigation and visual positioning technology, which can not only simplify surgical preparation and shorten surgical time, but also meet the requirements of high-precision medical surgery; this disclosure can also be applied to other fields, and this disclosure does not limit them.

[0111] The surgical navigation described above can be applied to surgeries involving intracranial drug delivery / implantation systems in large animals, as well as to other surgeries requiring precise positioning, such as spinal surgery and joint replacement. This disclosure does not limit its application in these areas.

[0112] Figure 3 This is a block diagram illustrating the control device of the robot disclosed herein, such as... Figure 3 As shown, the device includes a first acquisition unit 30, a second acquisition unit 32, a first determination unit 34, a third acquisition unit 36, a fourth acquisition unit 38, and a second determination unit 310.

[0113] The first acquisition unit 30 is configured to acquire a first point cloud sequence of the target object in the coordinate system of the depth camera, wherein the first point cloud sequence includes point clouds of the target object at multiple angles; the second acquisition unit 32 is configured to acquire a pose sequence of the robot's end effector in the base coordinate system; the first determination unit 34 is configured to determine a first transformation matrix from each point cloud in the first point cloud sequence to a predetermined point cloud, wherein the predetermined point cloud is the first point cloud acquired in the first point cloud sequence; the third acquisition unit 36 ​​is configured to obtain a second transformation matrix from the coordinate system of the depth camera to the coordinate system of the robot's end effector based on the first transformation matrix and pose sequence corresponding to a predetermined number of point clouds in the first point cloud sequence; the fourth acquisition unit 38 is configured to obtain a second point cloud sequence of the robot's end effector in the base coordinate system based on the first point cloud sequence, the second transformation matrix, and the pose sequence; and the second determination unit 310 is configured to determine a third point cloud for controlling the movement of the robot's end effector based on the second point cloud sequence.

[0114] According to an embodiment of this disclosure, the third acquisition unit 36 ​​is further configured to acquire poses corresponding to a predetermined number of point clouds from the pose sequence; determine a first parameter based on the poses corresponding to each pair of point clouds in the predetermined number of point clouds; determine a second parameter based on the first transformation matrix corresponding to each pair of point clouds in the predetermined number of point clouds; and obtain a second transformation matrix from the coordinate system of the depth camera to the coordinate system of the robotic arm end effector based on the first parameter and the second parameter.

[0115] According to an embodiment of this disclosure, the fourth acquisition unit 38 is further configured to perform the following processing for each point cloud in the first point cloud sequence: multiply the pose corresponding to the current point cloud, the second transformation matrix, and the current point cloud to obtain the second point cloud corresponding to the current point cloud; and obtain the second point cloud sequence based on the second point cloud corresponding to each point cloud in the first point cloud sequence.

[0116] According to an embodiment of this disclosure, the second determining unit 310 is further configured to perform the following processing for each second point cloud in the second point cloud sequence: obtaining a third transformation matrix from the second point cloud to the target point cloud; processing the third transformation matrix by interpolation to obtain a fourth transformation matrix that transforms the second point cloud to an intermediate coordinate system; transforming the second point cloud and the target point cloud to the intermediate coordinate system based on the fourth transformation matrix; merging the second point cloud and the target point cloud transformed to the intermediate coordinate system to obtain an updated target point cloud; and determining the updated target point cloud as the third point cloud after processing all the second point clouds in the second point cloud sequence.

[0117] According to an embodiment of this disclosure, the second determining unit 310 is further configured to convert the rotation information in the third transformation matrix into quaternions and perform spherical linear interpolation between the quaternions and the unit quaternions to obtain a rotation matrix; perform linear interpolation on the position information in the third transformation matrix to obtain a displacement vector; and obtain a fourth transformation matrix that transforms the second point cloud to the intermediate coordinate system based on the rotation matrix and the displacement vector.

[0118] According to an embodiment of this disclosure, the second determining unit 310 is further configured to transform the second point cloud to an intermediate coordinate system by multiplying the fourth transformation matrix and the second point cloud; and to transform the target point cloud to an intermediate coordinate system by multiplying the inverse of the fourth transformation matrix and the third transformation matrix with the target point cloud.

[0119] According to embodiments of the present disclosure, a computer-readable storage medium for storing instructions is provided, wherein when the instructions are executed by at least one computing device, they cause at least one computing device to perform a robot control method as described in any of the above embodiments.

[0120] According to embodiments of the present disclosure, a system is provided that includes at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by at least one computing device, cause at least one computing device to perform a robot control method as described in any of the above embodiments.

[0121] According to embodiments of this disclosure, a computer program product is provided, including computer instructions that, when executed by a processor, implement a control method for any of the robots described above.

[0122] While some embodiments of this disclosure have been shown and described, those skilled in the art will understand that modifications may be made to these embodiments without departing from the principles and spirit of this disclosure, which are defined by the claims and their equivalents.

Claims

1. A method for controlling a robot, characterized in that, include: Obtain the first point cloud sequence of the target object in the coordinate system of the depth camera, wherein the first point cloud sequence contains point clouds of the target object from multiple angles; Obtain the pose sequence of the robot's end effector in the base coordinate system; Determine a first transformation matrix from each point cloud in the first point cloud sequence to a predetermined point cloud, wherein the predetermined point cloud is the first point cloud acquired in the first point cloud sequence; Based on the first transformation matrix corresponding to a predetermined number of point clouds in the first point cloud sequence and the pose sequence, a second transformation matrix is ​​obtained from the coordinate system of the depth camera to the coordinate system of the robotic arm end effector. Based on the first point cloud sequence, the second transformation matrix, and the pose sequence, the second point cloud sequence of the robotic arm end effector in the base coordinates is obtained; Based on the second point cloud sequence, a third point cloud is determined to control the movement of the robotic arm's end effector.

2. The control method as described in claim 1, characterized in that, The process of obtaining a second transformation matrix from the coordinate system of the depth camera to the coordinate system of the robotic arm end effector, based on a first transformation matrix corresponding to a predetermined number of point clouds in the first point cloud sequence and the pose sequence, includes: From the pose sequence, obtain the poses corresponding to the predetermined number of point clouds; Based on the poses of each pair of point clouds in the predetermined number of point clouds, the first parameter is determined. The second parameter is determined based on the first transformation matrix corresponding to each pair of point clouds in the predetermined number of point clouds. Based on the first parameter and the second parameter, a second transformation matrix is ​​obtained from the coordinate system of the depth camera to the coordinate system of the robotic arm end effector.

3. The control method as described in claim 1, characterized in that, The step of obtaining the second point cloud sequence of the robotic arm end effector in base coordinates based on the first point cloud sequence, the second transformation matrix, and the pose sequence includes: For each point cloud in the first point cloud sequence, the following process is performed: the pose corresponding to the current point cloud, the second transformation matrix, and the current point cloud are multiplied together to obtain the second point cloud corresponding to the current point cloud. The second point cloud sequence is obtained based on the second point cloud corresponding to each point cloud in the first point cloud sequence.

4. The control method as described in claim 1, characterized in that, The step of determining the third point cloud for controlling the movement of the robotic arm end effector based on the second point cloud sequence includes: For each second point cloud in the second point cloud sequence, perform the following processing: Obtain the third transformation matrix from the second point cloud to the target point cloud; The third transformation matrix is ​​processed by interpolation to obtain a fourth transformation matrix that transforms the second point cloud to the intermediate coordinate system. Based on the fourth transformation matrix, the second point cloud and the target point cloud are transformed to the intermediate coordinate system; The second point cloud and the target point cloud, transformed to the intermediate coordinate system, are merged to obtain the updated target point cloud. After processing all the second point clouds in the second point cloud sequence, the updated target point cloud is determined as the third point cloud.

5. The control method as described in claim 4, characterized in that, The step of processing the third transformation matrix through interpolation to obtain the fourth transformation matrix that transforms the second point cloud to the intermediate coordinate system includes: For the rotation information in the third transformation matrix, the rotation information is converted into quaternions, and spherical linear interpolation is performed between the quaternions and the unit quaternion to obtain the rotation matrix; Linear interpolation is performed on the position information in the third transformation matrix to obtain the displacement vector; Based on the rotation matrix and the displacement vector, a fourth transformation matrix is ​​obtained to transform the second point cloud into the intermediate coordinate system.

6. The control method as described in claim 4, characterized in that, The step of transforming the second point cloud and the target point cloud to the intermediate coordinate system based on the fourth transformation matrix includes: By multiplying the fourth transformation matrix and the second point cloud, the second point cloud is transformed into the intermediate coordinate system; The target point cloud is transformed into the intermediate coordinate system by multiplying the fourth transformation matrix, the inverse of the third transformation matrix, and the target point cloud.

7. A control device for a robot, characterized in that, include: The first acquisition unit is configured to acquire a first point cloud sequence of the target object in the coordinate system of the depth camera, wherein the first point cloud sequence contains point clouds of the target object at multiple angles. The second acquisition unit is configured to acquire the pose sequence of the robot's end effector in the base coordinate system; The first determining unit is configured to determine a first transformation matrix from each point cloud in the first point cloud sequence to a predetermined point cloud, wherein the predetermined point cloud is the first point cloud acquired in the first point cloud sequence. The third acquisition unit is configured to obtain a second transformation matrix from the coordinate system of the depth camera to the coordinate system of the end effector of the robotic arm based on the first transformation matrix corresponding to a predetermined number of point clouds in the first point cloud sequence and the pose sequence. The fourth acquisition unit is configured to obtain a second point cloud sequence of the robotic arm end effector in base coordinates based on the first point cloud sequence, the second transformation matrix, and the pose sequence. The second determining unit is configured to determine a third point cloud for controlling the movement of the robotic arm end effector based on the second point cloud sequence.

8. A computer-readable storage medium for storing instructions, characterized in that, When the instructions are executed by at least one computing device, the at least one computing device is caused to perform the robot control method as described in any one of claims 1 to 6.

9. A system comprising at least one computing device and at least one storage device for storing instructions, characterized in that, When the instruction is executed by the at least one computing device, it causes the at least one computing device to perform the robot control method as described in any one of claims 1 to 6.

10. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the robot control method as described in any one of claims 1 to 6.

Images