Robot control method and device

Abstract

The invention discloses a robot control method and device, and the method comprises the steps: obtaining a first point cloud sequence of a target object in a coordinate system of a depth camera, and enabling the first point cloud sequence to comprise point clouds of the target object at a plurality of angles; acquiring a pose sequence of the tail end of a mechanical arm of the robot under the base coordinate system; a first transformation matrix from each point cloud in the first point cloud sequence to a predetermined point cloud is determined, and the predetermined point cloud is the first obtained point cloud in the first point cloud sequence; obtaining a second transformation matrix from the coordinate system of the depth camera to the coordinate system of the tail end of the mechanical arm based on a first transformation matrix corresponding to a predetermined number of point clouds in the first point cloud sequence and the pose sequence; based on the first point cloud sequence, the second transformation matrix and the pose sequence, a second point cloud sequence of the tail end of the mechanical arm under the base coordinate is obtained; and based on the second point cloud sequence, third point cloud for controlling the tail end of the mechanical arm to move is determined.

Description

Technical Field

[0001] The present disclosure generally relates to the field of robots, and more particularly, to a control method and apparatus for robots. Background Art

[0002] Robots are widely used in a wide range of fields, such as industrial manufacturing, logistics and warehousing, medical, service, agricultural, military, etc. However, no matter which field they are used in, accurate visual positioning is crucial for the navigation and operation of the robot's end arm, especially in the field of auxiliary medical treatment. At present, the hand-eye calibration method for robots has the following main defects: 1. Complicated operation: Traditional hand-eye calibration requires the use of special calibration tools (such as chessboards, circular targets, etc.) to complete the calibration between the depth camera and the end of the robotic arm before shooting the object from multiple perspectives. This process is time-consuming and laborious. If the robot is used in the field of assisted medical treatment, the cumbersome operation of the hand-eye calibration process is not conducive to the rapid deployment of surgery.

[0003] 2. Poor flexibility: Once the relative position of the depth camera and the robotic arm changes (such as equipment adjustment or accidental collision), hand-eye calibration needs to be performed again, which reduces the flexibility and practicality of the system.

[0004] 3. High random deviation: In the process of stitching multi-view point clouds of the target object, the traditional incremental registration method usually uses a point cloud as a reference, resulting in high random deviation and affecting the final positioning accuracy. If the robot is used in the field of auxiliary medical care, it is difficult to meet the requirements of high-precision medical surgery. Summary of the invention

[0005] The embodiments of the present disclosure provide a robot control method and device, which can solve at least one of the above problems.

[0006] In a general aspect, a robot control method is provided, comprising: obtaining a first point cloud sequence of a target object in a coordinate system of a depth camera, wherein the first point cloud sequence includes point clouds of the target object at multiple angles; obtaining a pose sequence of a robotic arm end of the robot in a base coordinate system; determining a first transformation matrix from each point cloud in the first point cloud sequence to a predetermined point cloud, wherein the predetermined point cloud is the first point cloud obtained in the first point cloud sequence; obtaining a second transformation matrix from the coordinate system of the depth camera to the coordinate system of the robotic arm end based on the first transformation matrix and the pose sequence corresponding to a predetermined number of point clouds in the first point cloud sequence; obtaining a second point cloud sequence of the robotic arm end in the base coordinate system based on the first point cloud sequence, the second transformation matrix and the pose sequence; and determining a third point cloud for controlling the movement of the robotic arm end based on the second point cloud sequence.

[0007] Optionally, based on the first transformation matrix and posture sequence corresponding to a predetermined number of point clouds in the first point cloud sequence, a second transformation matrix from the coordinate system of the depth camera to the coordinate system of the end of the robotic arm is obtained, including: obtaining the postures corresponding to a predetermined number of point clouds from the posture sequence; determining the first parameter based on the postures corresponding to each pair of point clouds in the predetermined number of point clouds; determining the second parameter based on the first transformation matrix corresponding to each pair of point clouds in the predetermined number of point clouds; and obtaining the second transformation matrix from the coordinate system of the depth camera to the coordinate system of the end of the robotic arm based on the first parameter and the second parameter.

[0008] Optionally, based on the first point cloud sequence, the second transformation matrix and the posture sequence, a second point cloud sequence of the end of the robotic arm in the base coordinates is obtained, including: for each point cloud in the first point cloud sequence, the following processing is performed: the posture corresponding to the current point cloud, the second transformation matrix and the current point cloud are multiplied to obtain the second point cloud corresponding to the current point cloud; based on the second point cloud corresponding to each point cloud in the first point cloud sequence, a second point cloud sequence is obtained.

[0009] Optionally, based on the second point cloud sequence, a third point cloud for controlling the movement of the end of the robotic arm is determined, including: for each second point cloud in the second point cloud sequence, performing the following processing: obtaining a third transformation matrix from the second point cloud to the target point cloud; processing the third transformation matrix by interpolation to obtain a fourth transformation matrix for transforming the second point cloud to an intermediate coordinate system; based on the fourth transformation matrix, transforming the second point cloud and the target point cloud to the intermediate coordinate system; merging the second point cloud and the target point cloud transformed to the intermediate coordinate system to obtain an updated target point cloud; after processing all second point clouds in the second point cloud sequence, determining the updated target point cloud as the third point cloud.

[0010] Optionally, the third transformation matrix is ​​processed by interpolation to obtain a fourth transformation matrix that transforms the second point cloud into the intermediate coordinate system, including: for the rotation information in the third transformation matrix, the rotation information is converted into a quaternion, and spherical linear interpolation is performed between the quaternion and the unit quaternion to obtain a rotation matrix; for the position information in the third transformation matrix, linear interpolation is performed to obtain a displacement vector; based on the rotation matrix and the displacement vector, a fourth transformation matrix that transforms the second point cloud into the intermediate coordinate system is obtained.

[0011] Optionally, based on the fourth transformation matrix, the second point cloud and the target point cloud are transformed into an intermediate coordinate system, including: transforming the second point cloud into the intermediate coordinate system by multiplying the fourth transformation matrix and the second point cloud; transforming the target point cloud into the intermediate coordinate system by multiplying the fourth transformation matrix, the inverse of the third transformation matrix and the target point cloud. In another general aspect, a control device for a robot is provided, comprising: a first acquisition unit, configured to acquire a first point cloud sequence of a target object in a coordinate system of a depth camera, wherein the first point cloud sequence includes point clouds of the target object at multiple angles; a second acquisition unit, configured to acquire a pose sequence of a robotic arm end of the robot in a base coordinate system; a first determination unit, configured to determine a first transformation matrix from each point cloud in the first point cloud sequence to a predetermined point cloud, wherein the predetermined point cloud is the first point cloud acquired in the first point cloud sequence; a third acquisition unit, configured to obtain a second transformation matrix from the coordinate system of the depth camera to the coordinate system of the robotic arm end based on the first transformation matrix and the pose sequence corresponding to a predetermined number of point clouds in the first point cloud sequence; a fourth acquisition unit, configured to obtain a second point cloud sequence of the robotic arm end in the base coordinate system based on the first point cloud sequence, the second transformation matrix and the pose sequence; and a second determination unit, configured to determine a third point cloud for controlling the movement of the robotic arm end based on the second point cloud sequence.

[0012] Optionally, the third acquisition unit is also configured to obtain poses corresponding to a predetermined number of point clouds from the pose sequence; determine a first parameter based on the poses corresponding to each pair of point clouds in the predetermined number of point clouds; determine a second parameter based on the first transformation matrix corresponding to each pair of point clouds in the predetermined number of point clouds; and obtain a second transformation matrix from the coordinate system of the depth camera to the coordinate system of the end of the robotic arm based on the first parameter and the second parameter.

[0013] Optionally, the fourth acquisition unit is also configured to perform the following processing for each point cloud in the first point cloud sequence: multiply the posture corresponding to the current point cloud, the second transformation matrix and the current point cloud to obtain a second point cloud corresponding to the current point cloud; based on the second point cloud corresponding to each point cloud in the first point cloud sequence, obtain a second point cloud sequence.

[0014] Optionally, the second determination unit is further configured to perform the following processing for each second point cloud in the second point cloud sequence: obtain a third transformation matrix from the second point cloud to the target point cloud; process the third transformation matrix by interpolation to obtain a fourth transformation matrix that transforms the second point cloud to the intermediate coordinate system; based on the fourth transformation matrix, transform the second point cloud and the target point cloud to the intermediate coordinate system; merge the second point cloud and the target point cloud transformed to the intermediate coordinate system to obtain an updated target point cloud; after processing all second point clouds in the second point cloud sequence, determine the updated target point cloud as the third point cloud.

[0015] Optionally, the second determination unit is further configured to convert the rotation information in the third transformation matrix into a quaternion, and perform spherical linear interpolation between the quaternion and the unit quaternion to obtain a rotation matrix; perform linear interpolation on the position information in the third transformation matrix to obtain a displacement vector; and based on the rotation matrix and the displacement vector, obtain a fourth transformation matrix that transforms the second point cloud into the intermediate coordinate system.

[0016] Optionally, the second determination unit is further configured to transform the second point cloud to the intermediate coordinate system by multiplying the fourth transformation matrix and the second point cloud; and transform the target point cloud to the intermediate coordinate system by multiplying the fourth transformation matrix, the inverse of the third transformation matrix and the target point cloud.

[0017] In another general aspect, a computer-readable storage medium storing instructions is provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to perform any robot control method as described above.

[0018] In another general aspect, a system is provided comprising at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to perform any robot control method as described above.

[0019] In another general aspect, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement any robot control method as described above.

[0020] According to the control method and device of the robot of the embodiment of the present disclosure, the geometric features of the target object itself (i.e., the first point cloud sequence and its corresponding transformation matrix) can be used to acquire the third point cloud of the end of the robotic arm in real time, so that the hand-eye calibration of the depth camera and the end of the robotic arm can be automatically completed without pre-calibration, thus eliminating the cumbersome calibration process; moreover, the present disclosure does not rely on external calibration tools and does not require pre-calibration, so that it can adapt to changes in the position of the depth camera or the robotic arm and has higher flexibility; furthermore, the present disclosure uses the first point cloud acquired in the first point cloud sequence as the only reference point cloud in the point cloud registration, rather than using the stitching result as the point cloud, which can overcome the error caused by the stitching result to a certain extent.

[0021] Additional aspects and / or advantages of the present general inventive concept will be set forth in part in the following description and in part will be apparent from the description or may be learned through practice of the present general inventive concept. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other objects and features of the embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings showing the embodiments, in which: Figure 1 is a flowchart showing a method for controlling a robot according to an embodiment of the present disclosure; Figure 2 is a system flow chart showing a method for controlling a robot according to an embodiment of the present disclosure; Figure 3 is a block diagram showing a control device of the robot of the present disclosure. DETAILED DESCRIPTION

[0023] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of features known in the art may be omitted.

[0024] The features described herein can be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, which will be clear after understanding the disclosure of the present application.

[0025] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0026] Although terms such as "first", "second", and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are only used to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Therefore, without departing from the teachings of the examples described herein, the first member, first component, first region, first layer, or first portion referred to in the examples may also be referred to as the second member, second component, second region, second layer, or second portion.

[0027] In the specification, when an element (such as a layer, a region, or a substrate) is described as being “on”, “connected to”, or “coupled to” another element, the element may be directly “on”, “connected to”, or “coupled to” another element, or one or more other elements may be present therebetween. Conversely, when an element is described as being “directly on”, “directly connected to”, or “directly coupled to” another element, there may be no other elements present therebetween.

[0028] The terms used herein are only used to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the described features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0029] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which the present disclosure belongs after understanding the present disclosure. Unless explicitly defined as such herein, terms (such as those defined in general dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal manner.

[0030] Furthermore, in the description of examples, when it is considered that a detailed description of a well-known related structure or function would cause vague interpretation of the present disclosure, such a detailed description will be omitted.

[0031] To facilitate understanding of the present disclosure, a brief explanation of the hand-eye calibration of the robot is given here, that is, the hand-eye calibration is the process of determining the conversion relationship between the coordinate system of the depth camera and the coordinate system of the end of the robotic arm.

[0032] The control method and device of the robot disclosed in the present invention are described in detail below with reference to the accompanying drawings.

[0033] The present invention provides a robot control method. Figure 1 is a flowchart showing a control method of a robot according to an embodiment of the present disclosure. Figure 1 , the robot control method includes the following steps: In step S101, a first point cloud sequence of a target object in a coordinate system of a depth camera is obtained, wherein the first point cloud sequence includes point clouds of the target object at multiple angles.

[0034] As an example, the robot's mechanical arm can pull the depth camera to shoot the target object from multiple perspectives and record the 3D point cloud of the target object at each perspective. , the first point cloud sequence is obtained by integrating the 3D point clouds under each viewing angle, wherein the above-mentioned depth camera can also be replaced by other 3D sensors, the above-mentioned 3D point cloud is the above-mentioned point cloud, the above-mentioned Indicates the coordinate system of the depth camera, N Indicates the number of viewing angles.

[0035] As an example, the above-mentioned depth camera can be installed at the end of the robot's mechanical arm, such as on the end effector (such as a mechanical gripper, a welding tool, etc.). This installation method is very useful in scenarios requiring high-precision operations, such as precision assembly tasks, precision surgical tasks, etc. However, the installation position of the depth camera is not limited in this disclosure.

[0036] In step S102, a position and posture sequence of the end of the robot's mechanical arm in the base coordinate system is obtained; As an example, the robot's mechanical arm can pull the depth camera to shoot the target object from multiple perspectives, and record the transformation matrix from the coordinate system of the end of the mechanical arm to the base coordinate system at each perspective. , that is, the position and posture of the end of the robotic arm corresponding to each point cloud in the base coordinate system , and then based on all the poses, the pose sequence can be obtained , the above Indicates the base coordinate system, the above Indicates the coordinate system of the end of the robot arm, T represents the transformation matrix, and N represents the number of viewing angles.

[0037] It should be noted that the pose sequence refers to a sequence of the position and attitude of the end of the robot arm arranged in chronological order or a certain specific order, where the position information usually describes the coordinate position of the end of the robot arm in space, such as the coordinates in a three-dimensional rectangular coordinate system; the attitude information usually describes the orientation of the end of the robot arm, which is usually represented by Euler angles (such as roll angle, pitch angle, yaw angle) or rotation matrix, quaternion, etc. In other words, the pose sequence contains position information and attitude information.

[0038] The base coordinate system is an important concept in robot programming and operation. It is a Cartesian coordinate system fixed in the robot's working environment, used to define the robot's position and posture. The base coordinate system is usually defined at a fixed position in the robot base or robot workspace. It serves as the global reference frame of the entire robot system. All robot movements and operations are described relative to this reference frame.

[0039] In step S103, a first transformation matrix from each point cloud in the first point cloud sequence to a predetermined point cloud is determined, wherein the predetermined point cloud is the first point cloud acquired in the first point cloud sequence.

[0040] As an example, a point cloud registration method can be used to determine the first point cloud sequence for each point cloud To the first acquired point cloud The transformation matrix ,in, The form can be 4× , 4 represents 4-dimensional homogeneous coordinates, Indicates the number of point clouds.

[0041] It should be noted that point cloud registration is the process of aligning point cloud data acquired from different perspectives so that they are in a unified coordinate system; incremental registration is a method of gradually aligning newly acquired point clouds with existing point clouds. The transformation matrix is ​​used to describe geometric transformations in three-dimensional space. The dimension can be 4×4. The transformation of three-dimensional point clouds can be achieved through matrix operations. The transformation matrix can include several parts: the 3×3 submatrix in the upper left corner is used to describe rotation transformations; the 3×1 column vector in the upper right corner is used to describe translation transformations; the last row is usually [0, 0, 0, 1], so that the transformation matrix can handle homogeneous coordinates. In three-dimensional space, point clouds can be represented as (x, y, z). In order to facilitate the simultaneous processing of rotation and translation, it needs to be expanded to homogeneous coordinates (x, y, z, 1).

[0042] As an example, this embodiment adopts an adjusted incremental registration method, that is, the reference point cloud of each point cloud is the first point cloud acquired in the first point cloud sequence, rather than the point cloud after each splicing, which can overcome the error caused by the splicing result to a certain extent.

[0043] As an example, the above-mentioned registration method can adopt the ICP (Iterative Closest Point) algorithm, and can also adopt a feature-based registration method, such as using key point matching, distance transformation or deep learning methods, to improve the robustness and accuracy of the registration, which is not limited in this disclosure.

[0044] In step S104, based on the first transformation matrix and the pose sequence corresponding to a predetermined number of point clouds in the first point cloud sequence, a second transformation matrix from the coordinate system of the depth camera to the coordinate system of the end of the robotic arm is obtained.

[0045] As an example, the first transformation matrix and the pose sequence corresponding to a predetermined number of point clouds in the first point cloud sequence can be input into the hand-eye calibration model to obtain the second transformation matrix from the coordinate system of the depth camera to the coordinate system of the end of the robotic arm. The model can be expressed as ,in, is the first parameter associated with the first transformation matrix, is the second parameter related to the pose sequence, and X is the second transformation matrix to be solved.

[0046] The following is based on and A method of obtaining the second transformation matrix is ​​described in detail: According to an embodiment of the present disclosure, the second transformation matrix from the coordinate system of the depth camera to the coordinate system of the end of the robotic arm can be obtained based on the first transformation matrix and posture sequence corresponding to a predetermined number of point clouds in the first point cloud sequence in the following manner: obtaining the postures corresponding to a predetermined number of point clouds from the posture sequence; determining the first parameter based on the postures corresponding to each of the two point clouds in the predetermined number of point clouds; determining the second parameter based on the first transformation matrix corresponding to each of the two point clouds in the predetermined number of point clouds; obtaining the second transformation matrix from the coordinate system of the depth camera to the coordinate system of the end of the robotic arm based on the first parameter and the second parameter. Through this embodiment, using the first transformation matrix and posture corresponding to the predetermined number of point clouds in the first point cloud sequence, relatively accurate parameters can be determined, and then an accurate second transformation matrix can be obtained.

[0047] As an example, the above-mentioned predetermined number can be set as needed, and the above-mentioned predetermined number of point clouds can be part of the point clouds in the first point cloud sequence, or can be all the point clouds in the first point cloud sequence, which is not limited in the present disclosure.

[0048] As an example, the above It can be expressed as , the above It can be expressed as , which is not limited in the present disclosure.

[0049] To facilitate understanding of the above and , the following is a brief description and The above representation acquisition process: For any pair of point clouds in the first point cloud sequence and , since the point cloud of the end of the robotic arm corresponding to each point cloud can be obtained through the transformation matrix from the coordinate system of the depth camera to the coordinate system of the end of the robotic arm and the transformation matrix (i.e., posture) from the coordinate system of the end of the robotic arm to the base coordinate system, the two are aligned after the transformation, that is: (1) in, Represents the transformation matrix from the coordinate system of the end of the robot to the base coordinate system, Represents the transformation matrix from the coordinate system of the depth camera to the coordinate system of the end of the robotic arm.

[0050] Then, using the incremental point cloud registration method (such as ICP algorithm), we can get To the first acquired point cloud The transformation matrix , so we have: (2) Arranging the above formulas (1) and (2) yields: (3) Furthermore, the above formula (3) can be organized into a hand-eye calibration model ,in, , , .

[0051] It should be noted that the arrangement process of the above formulas (1) and (2) is as follows: Formula (2) can be converted into: (4) Substituting formula (4) into formula (1) yields: (5) Eliminating the common variables on both sides of formula (5), we can obtain: (6) After rearranging formula (6), we can get the above formula (3).

[0052] In step S105, based on the first point cloud sequence, the second transformation matrix and the pose sequence, a second point cloud sequence of the end of the robot arm in the base coordinates is obtained.

[0053] According to an embodiment of the present disclosure, a second point cloud sequence at the end of the robot arm in base coordinates can be obtained based on the first point cloud sequence, the second transformation matrix and the posture sequence in the following manner: for each point cloud in the first point cloud sequence, the following processing is performed: the posture corresponding to the current point cloud, the second transformation matrix and the current point cloud are multiplied to obtain the second point cloud corresponding to the current point cloud; based on the second point cloud corresponding to each point cloud in the first point cloud sequence, a second point cloud sequence is obtained. As an example, you can And solve for Applied to each point cloud in the coordinate system of the depth camera , get the point cloud of the end of the robotic arm corresponding to each point cloud in the base coordinate system , which is the second point cloud mentioned above, can be specifically expressed as: (7) After getting the second point cloud corresponding to each point cloud After that, the corresponding second point cloud sequence can be obtained .

[0054] In step S106, based on the second point cloud sequence, a third point cloud for controlling the movement of the end of the robot arm is determined.

[0055] As an example, the second point cloud sequences may be merged to obtain a third point cloud; or the second point cloud sequences may be processed as required to obtain a third point cloud, which is not limited in the present disclosure.

[0056] The above-mentioned embodiments of the present disclosure establish a correspondence between the geometric features of the target object itself, i.e., the above-mentioned first point cloud sequence and its corresponding first transformation matrix, through the registration of multi-view point clouds, and obtain the transformation matrix from the coordinate system of the depth camera to the coordinate system of the end of the robotic arm based on the above-mentioned information and the hand-eye calibration model. This not only avoids pre-calibration, but also unifies the solution of the transformation matrix from the coordinate system of the depth camera to the coordinate system of the end of the robotic arm with the registration of multi-view point clouds, so that the hand-eye calibration of the depth camera and the end of the robotic arm can be automatically completed, and it can also adapt to changes in the position of the depth camera or the robotic arm.

[0057] However, in the registration of multi-view point clouds, due to factors such as sensor noise, calibration errors, and the approximation of the registration algorithm, these point clouds may not be accurately aligned in the base coordinate system. For example, the traditional incremental registration method usually registers the newly added point cloud to the synthetic point cloud after the previous registration, which has a large random error. In order to reduce the random error, the present disclosure proposes an error correction method, that is, in each step of incremental registration, not only the newly added point cloud is adjusted, but also the synthesized point cloud is adjusted by interpolation, so that both are moved to an intermediate coordinate system. This method ensures the overall optimization of the point cloud and reduces the random error.

[0058] According to an embodiment of the present disclosure, determining a third point cloud for controlling the movement of the end of the robotic arm based on the second point cloud sequence can be achieved in the following manner: for each second point cloud in the second point cloud sequence, performing the following processing: obtaining a third transformation matrix from the second point cloud to the target point cloud; processing the third transformation matrix by interpolation to obtain a fourth transformation matrix for transforming the second point cloud to an intermediate coordinate system; based on the fourth transformation matrix, transforming the second point cloud and the target point cloud to the intermediate coordinate system; merging the second point cloud and the target point cloud transformed to the intermediate coordinate system to obtain an updated target point cloud; after processing all second point clouds in the second point cloud sequence, determining the updated target point cloud as the third point cloud.

[0059] Through this embodiment, in each step of incremental registration, not only the newly added point cloud (i.e., the second point cloud in the execution process) is adjusted, but also the synthesized point cloud (target point cloud) is adjusted by interpolation so that both are moved to an intermediate coordinate system, thereby ensuring the overall optimization of the point cloud and reducing the cumulative error.

[0060] As an example, before processing each second point cloud, the target point cloud may be initialized, for example, the first second point cloud in the second point cloud sequence may be set as the target point cloud, that is, Then, each second point cloud is gradually registered. Second point cloud For example: First, use a point cloud registration algorithm (such as ICP) to calculate the second point cloud To target point cloud The transformation matrix, that is, the third transformation matrix, can be expressed as ; Secondly, the third transformation matrix is ​​processed by interpolation to obtain the fourth transformation matrix that transforms the second point cloud into the intermediate coordinate system Then, using the fourth transformation matrix, the second point cloud and the target point cloud can be transformed into the intermediate coordinate system, and the second point cloud transformed into the intermediate coordinate system and target point cloud Merge as the new target point cloud for the next registration step, i.e. .

[0061] After all the second point clouds in the second point cloud sequence are processed, the updated target point cloud It is determined as the third point cloud, and then the movement of the robot's mechanical arm end is controlled based on the third point cloud. If it is used in the auxiliary medical field, surgical navigation positioning can be performed through the third point cloud.

[0062] As an example, the above interpolation method can be spherical linear interpolation (Slerp), simple linear interpolation, or other interpolation methods, such as spherical Bezier curve interpolation, quaternion averaging, etc. The interpolation strategy can be adjusted according to specific needs, and the present disclosure does not limit this.

[0063] As an example, the error correction process of this embodiment may also use a global optimization method instead of incremental registration to optimize all point clouds as a whole to further improve the stitching accuracy, which is not limited in this disclosure.

[0064] According to an embodiment of the present disclosure, the third transformation matrix is ​​processed by interpolation to obtain a fourth transformation matrix that transforms the second point cloud into an intermediate coordinate system. This can be achieved in the following manner: for the rotation information in the third transformation matrix, the rotation information is converted into a quaternion, and spherical linear interpolation is performed between the quaternion and the unit quaternion to obtain a rotation matrix; for the position information in the third transformation matrix, linear interpolation is performed to obtain a displacement vector; based on the rotation matrix and the displacement vector, a fourth transformation matrix that transforms the second point cloud into the intermediate coordinate system is obtained.

[0065] Through this embodiment, the rotation information in the form of quaternion is used to perform spherical linear interpolation, so that the rotation matrix can be obtained conveniently and quickly; and the position information is directly linearly interpolated, so that the displacement vector can be obtained conveniently and quickly, and then the fourth transformation matrix can be obtained conveniently and quickly.

[0066] As an example, the above rotation information can be obtained from arrive The above position information can be obtained from arrive The present disclosure does not limit the translation transformation.

[0067] For the rotation information, the rotation matrix of the second point cloud transformed to the intermediate coordinate system can be obtained by spherical linear interpolation. Specifically, Convert to quaternion ,exist and the unit quaternion Spherical linear interpolation is performed between them, which can be expressed as follows: (8) get After that, Transformation matrix form, that is, the corresponding rotation matrix .

[0068] For the position information, the displacement vector of the second point cloud transformed to the intermediate coordinate system can be obtained by linear interpolation, which can be expressed as follows: (9) After obtaining the corresponding rotation matrix and displacement vector, the fourth transformation matrix can be obtained, which can be expressed as .

[0069] According to an embodiment of the present disclosure, based on the fourth transformation matrix, the second point cloud and the target point cloud are transformed into an intermediate coordinate system, which can be achieved in the following manner: the second point cloud is transformed into the intermediate coordinate system by multiplying the fourth transformation matrix and the second point cloud; the target point cloud is transformed into the intermediate coordinate system by multiplying the fourth transformation matrix, the inverse of the third transformation matrix and the target point cloud.

[0070] As an example, after obtaining the fourth transformation matrix, the second point cloud and the target point cloud can be transformed into the intermediate coordinate system based on the fourth transformation matrix. Specifically: For the second node, the second point cloud can be transformed into the intermediate coordinate system by multiplying the fourth transformation matrix with the second point cloud, which can be expressed as follows: (10) For the target node, the target point cloud can be calculated by the fourth transformation matrix Transformation matrix to the intermediate coordinate system , which is expressed as follows: (11) After obtaining the above transformation matrix, the target point cloud is transformed using the transformation matrix Transformed to the intermediate coordinate system, it can be expressed as follows: (12) In summary, the traditional hand-eye calibration method requires the use of a dedicated calibration tool to pre-calibrate the depth camera and the end of the robotic arm before multi-view shooting of the target object. The operation is cumbersome, the flexibility is poor, and the errors are easily accumulated in the subsequent multi-view shooting.

[0071] The present disclosure provides a real-time hand-eye calibration and error correction method based on multi-view point cloud, specifically Figure 2 As shown, first, the robot's robotic arm can pull the 3D sensor to shoot the target object at multiple perspectives, and record the 3D point cloud of the target object at each perspective and the transformation matrix from the coordinate system of the robotic arm end to the base coordinate system; secondly, real-time hand-eye calibration is performed, that is, based on the transformation matrix and 3D point cloud from the coordinate system of the robotic arm end to the base coordinate system at multiple perspectives, the transformation matrix from the coordinate system of the depth camera to the coordinate system of the robotic arm end and the multi-perspective 3D point cloud of the robotic arm end in the base coordinate system (that is, the above-mentioned second point cloud sequence) are obtained; after obtaining the multi-perspective 3D point cloud of the robotic arm end in the base coordinate system, error correction of the multi-perspective point cloud is performed to obtain the 3D point cloud after error correction (that is, the above-mentioned updated target point cloud).

[0072] It can be seen that the present disclosure utilizes the geometric features of the target object itself to automatically complete the conversion between the coordinate system of the depth camera and the coordinate system of the end of the robotic arm during multi-view shooting to obtain a third point cloud for controlling the end of the robotic arm, thereby eliminating the cumbersome calibration process. Moreover, since it does not rely on external calibration tools and does not require pre-calibration, the present disclosure can adapt to changes in the position of the depth camera or the end of the robotic arm and has higher flexibility. Furthermore, the present disclosure can also introduce an interpolation method when stitching point clouds, adjust newly added point clouds and existing point clouds at the same time, move them to an intermediate coordinate system, reduce cumulative errors, and ensure high-precision positioning of the target object.

[0073] It should be noted that the present disclosure can be applied to the field of robot-assisted medical treatment, especially surgical navigation and visual positioning technology, which can not only simplify surgical preparation and shorten operation time, but also meet the requirements of high-precision medical surgery; the present disclosure can also be applied to other fields, which are not limited by the present disclosure.

[0074] The above-mentioned surgical navigation can be applied to the surgery of intracranial drug delivery / implantation system in large animals, and can also be applied to other surgeries requiring precise positioning, such as spinal surgery, joint replacement, etc., which is not limited by the present disclosure.

[0075] Figure 3 is a block diagram showing a control device of a robot disclosed in the present invention, such as Figure 3 As shown, the apparatus includes a first acquiring unit 30 , a second acquiring unit 32 , a first determining unit 34 , a third acquiring unit 36 ​​, a fourth acquiring unit 38 and a second determining unit 310 .

[0076] The first acquisition unit 30 is configured to acquire a first point cloud sequence of the target object in the coordinate system of the depth camera, wherein the first point cloud sequence includes point clouds at multiple angles of the target object; the second acquisition unit 32 is configured to acquire a pose sequence of the end of the robot's robotic arm in the base coordinate system; the first determination unit 34 is configured to determine a first transformation matrix from each point cloud in the first point cloud sequence to a predetermined point cloud, wherein the predetermined point cloud is the first point cloud acquired in the first point cloud sequence; the third acquisition unit 36 ​​is configured to obtain a second transformation matrix from the coordinate system of the depth camera to the coordinate system of the end of the robotic arm based on the first transformation matrix and pose sequence corresponding to a predetermined number of point clouds in the first point cloud sequence; the fourth acquisition unit 38 is configured to obtain a second point cloud sequence of the end of the robotic arm in the base coordinate system based on the first point cloud sequence, the second transformation matrix and the pose sequence; the second determination unit 310 is configured to determine a third point cloud for controlling the movement of the end of the robotic arm based on the second point cloud sequence.

[0077] According to an embodiment of the present disclosure, the third acquisition unit 36 ​​is also configured to obtain poses corresponding to a predetermined number of point clouds from the pose sequence; determine a first parameter based on the poses corresponding to each pair of point clouds in the predetermined number of point clouds; determine a second parameter based on the first transformation matrix corresponding to each pair of point clouds in the predetermined number of point clouds; and obtain a second transformation matrix from the coordinate system of the depth camera to the coordinate system of the end of the robotic arm based on the first parameter and the second parameter.

[0078] According to an embodiment of the present disclosure, the fourth acquisition unit 38 is also configured to perform the following processing for each point cloud in the first point cloud sequence: multiply the posture corresponding to the current point cloud, the second transformation matrix and the current point cloud to obtain the second point cloud corresponding to the current point cloud; based on the second point cloud corresponding to each point cloud in the first point cloud sequence, obtain the second point cloud sequence.

[0079] According to an embodiment of the present disclosure, the second determination unit 310 is further configured to perform the following processing for each second point cloud in the second point cloud sequence: obtain a third transformation matrix from the second point cloud to the target point cloud; process the third transformation matrix by interpolation to obtain a fourth transformation matrix that transforms the second point cloud to the intermediate coordinate system; based on the fourth transformation matrix, transform the second point cloud and the target point cloud to the intermediate coordinate system; merge the second point cloud and the target point cloud transformed to the intermediate coordinate system to obtain an updated target point cloud; after processing all second point clouds in the second point cloud sequence, determine the updated target point cloud as the third point cloud.

[0080] According to an embodiment of the present disclosure, the second determination unit 310 is further configured to convert the rotation information in the third transformation matrix into a quaternion, and perform spherical linear interpolation between the quaternion and the unit quaternion to obtain a rotation matrix; perform linear interpolation on the position information in the third transformation matrix to obtain a displacement vector; and based on the rotation matrix and the displacement vector, obtain a fourth transformation matrix that transforms the second point cloud into the intermediate coordinate system.

[0081] According to an embodiment of the present disclosure, the second determination unit 310 is also configured to transform the second point cloud to the intermediate coordinate system by multiplying the fourth transformation matrix and the second point cloud; and transform the target point cloud to the intermediate coordinate system by multiplying the fourth transformation matrix, the inverse of the third transformation matrix and the target point cloud.

[0082] According to an embodiment of the present disclosure, a computer-readable storage medium storing instructions is provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to execute a robot control method as described in any of the above embodiments.

[0083] According to an embodiment of the present disclosure, a system is provided, comprising at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, prompt the at least one computing device to execute a robot control method as described in any of the above embodiments.

[0084] According to an embodiment of the present disclosure, a computer program product is provided, including computer instructions, which, when executed by a processor, implement any of the above-mentioned robot control methods.

[0085] Although some embodiments of the present disclosure have been shown and described, it will be appreciated by those skilled in the art that modifications may be made to the embodiments without departing from the principles and spirit of the present disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. A robot control method, characterized in that: include: Acquire a first point cloud sequence of the target object in the coordinate system of the depth camera, wherein the first point cloud sequence includes point clouds of the target object at multiple angles; Obtain the position and posture sequence of the robot's manipulator end in the base coordinate system; Determine a first transformation matrix from each point cloud in the first point cloud sequence to a predetermined point cloud, wherein the predetermined point cloud is the first point cloud acquired in the first point cloud sequence; Based on a first transformation matrix corresponding to a predetermined number of point clouds in the first point cloud sequence and the pose sequence, obtaining a second transformation matrix from the coordinate system of the depth camera to the coordinate system of the end of the robotic arm; Based on the first point cloud sequence, the second transformation matrix and the pose sequence, obtaining a second point cloud sequence of the end of the robotic arm in base coordinates; Based on the second point cloud sequence, a third point cloud for controlling the movement of the end of the robot arm is determined.

2. The control method according to claim 1, characterized in that: The step of obtaining a second transformation matrix from the coordinate system of the depth camera to the coordinate system of the end of the robotic arm based on the first transformation matrix corresponding to a predetermined number of point clouds in the first point cloud sequence and the pose sequence comprises: From the pose sequence, obtaining poses corresponding to the predetermined number of point clouds; Determining a first parameter based on the poses corresponding to each of the two point clouds in the predetermined number of point clouds; Determine a second parameter based on first transformation matrices corresponding to each of two point clouds in the predetermined number of point clouds; Based on the first parameter and the second parameter, a second transformation matrix from the coordinate system of the depth camera to the coordinate system of the end of the robotic arm is obtained.

3. The control method according to claim 1, characterized in that: The step of obtaining a second point cloud sequence of the end of the robotic arm in base coordinates based on the first point cloud sequence, the second transformation matrix and the pose sequence comprises: For each point cloud in the first point cloud sequence, performing the following processing: multiplying the posture corresponding to the current point cloud, the second transformation matrix, and the current point cloud to obtain a second point cloud corresponding to the current point cloud; Based on the second point cloud corresponding to each point cloud in the first point cloud sequence, the second point cloud sequence is obtained.

4. The control method according to claim 1, characterized in that: The step of determining a third point cloud for controlling the movement of the end of the robotic arm based on the second point cloud sequence includes: For each second point cloud in the second point cloud sequence, the following processing is performed: Obtaining a third transformation matrix from the second point cloud to the target point cloud; Processing the third transformation matrix by interpolation to obtain a fourth transformation matrix for transforming the second point cloud into an intermediate coordinate system; Based on the fourth transformation matrix, transform the second point cloud and the target point cloud into the intermediate coordinate system; Merging the second point cloud transformed into the intermediate coordinate system with the target point cloud to obtain an updated target point cloud; After all second point clouds in the second point cloud sequence are processed, the updated target point cloud is determined as the third point cloud.

5. The control method according to claim 4, characterized in that: The processing of the third transformation matrix by interpolation to obtain a fourth transformation matrix for transforming the second point cloud into an intermediate coordinate system includes: For the rotation information in the third transformation matrix, convert the rotation information into a quaternion, and perform spherical linear interpolation between the quaternion and the unit quaternion to obtain a rotation matrix; Performing linear interpolation on the position information in the third transformation matrix to obtain a displacement vector; Based on the rotation matrix and the displacement vector, a fourth transformation matrix is ​​obtained for transforming the second point cloud into an intermediate coordinate system.

6. The control method according to claim 4, characterized in that: The transforming the second point cloud and the target point cloud to the intermediate coordinate system based on the fourth transformation matrix includes: Transform the second point cloud to the intermediate coordinate system by multiplying the fourth transformation matrix and the second point cloud; The target point cloud is transformed into the intermediate coordinate system by multiplying the fourth transformation matrix, the inverse of the third transformation matrix and the target point cloud.

7. A robot control device, characterized in that: include: A first acquisition unit is configured to acquire a first point cloud sequence of the target object in the coordinate system of the depth camera, wherein the first point cloud sequence includes point clouds of the target object at multiple angles; A second acquisition unit is configured to acquire a position and posture sequence of the end of the robot's mechanical arm in a base coordinate system; A first determining unit is configured to determine a first transformation matrix from each point cloud in the first point cloud sequence to a predetermined point cloud, wherein the predetermined point cloud is a first point cloud acquired in the first point cloud sequence; A third acquisition unit is configured to obtain a second transformation matrix from the coordinate system of the depth camera to the coordinate system of the end of the robotic arm based on the first transformation matrix corresponding to a predetermined number of point clouds in the first point cloud sequence and the pose sequence; a fourth acquisition unit, configured to obtain a second point cloud sequence of the end of the robotic arm in base coordinates based on the first point cloud sequence, the second transformation matrix and the pose sequence; The second determining unit is configured to determine a third point cloud for controlling the movement of the end of the robot arm based on the second point cloud sequence.

8. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by at least one computing device, the at least one computing device is prompted to execute the robot control method according to any one of claims 1 to 6.

9. A system comprising at least one computing device and at least one storage device storing instructions, characterized in that: When the instructions are executed by the at least one computing device, the at least one computing device is prompted to execute the robot control method according to any one of claims 1 to 6.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the robot control method according to any one of claims 1 to 6 is implemented.

Images