Dual-depth camera extrinsic parameter calibration method and device, robot, and storage medium

Abstract

The embodiment of the application discloses a double-depth camera external parameter calibration method and device, a robot and a storage medium, relates to the technical field of calibration, and comprises the following steps: acquiring first point cloud data collected by a first depth camera from a preset scene and second point cloud data collected by a second depth camera from the preset scene; determining a first transformation matrix for transforming the first point cloud data from a first coordinate system to a machine coordinate system, and calibrating the first transformation matrix to the first depth camera; determining a second transformation matrix for transforming the second point cloud data from a second coordinate system to the machine coordinate system, and calibrating the second transformation matrix to the second depth camera; determining first transformed point cloud data based on the first transformation matrix, and determining second transformed point cloud data based on the second transformation matrix; determining a third transformation matrix for aligning the first transformed point cloud data and the second transformed point cloud data, and calibrating the third transformation matrix to the second depth camera, thereby improving the precision and accuracy of external parameter calibration.

Description

Technical Field

[0001] This application relates to the field of calibration technology, and in particular to a method, apparatus, robot, and storage medium for calibrating extrinsic parameters of a dual depth camera. Background Technology

[0002] During autonomous navigation of a robot, point cloud data of the surrounding environment can be acquired using a depth camera. However, the coordinates of this point cloud data are based on the coordinate system of the depth camera itself. Therefore, it is necessary to calibrate the extrinsic parameters of the depth camera to accurately convert the point cloud data in the camera coordinate system to the robot coordinate system. This allows for the construction of map data of the surrounding environment, enabling precise positioning of the robot and providing a foundation for the robot's path planning.

[0003] Currently, due to the small field of view of depth cameras and the tendency of a single depth camera to degrade when probing flat areas such as walls, dual depth cameras can be used to determine the robot's position data. However, when calibrating the extrinsic parameters of dual depth cameras, the errors caused by factors such as the installation position, installation angle, and measurement depth of the depth cameras are not taken into account, which reduces the accuracy of the extrinsic parameter calibration and leads to low positioning accuracy of the robot. Summary of the Invention

[0004] This application provides a method, apparatus, robot, and storage medium for extrinsic parameter calibration of a dual depth camera, which realizes the extrinsic parameter calibration function of a dual depth camera to solve the problem of low accuracy of extrinsic parameter calibration in the prior art.

[0005] In a first aspect, embodiments of this application provide a dual-depth camera extrinsic parameter calibration method, applied to a robot, the robot being equipped with a first depth camera and a second depth camera, the method comprising:

[0006] Acquire the first point cloud data obtained by the first depth camera capturing a preset scene, and acquire the second point cloud data obtained by the second depth camera capturing a preset scene;

[0007] Determine the transformation matrix to transform the first point cloud data from the first coordinate system to the machine coordinate system, obtain the first transformation matrix, and calibrate the first transformation matrix to the first depth camera. The first coordinate system is the coordinate system of the first depth camera, and the machine coordinate system is the coordinate system of the robot.

[0008] Determine the transformation matrix to transform the second point cloud data from the second coordinate system to the machine coordinate system, obtain the second transformation matrix, and calibrate the second transformation matrix to the second depth camera, with the second coordinate system being the coordinate system of the second depth camera;

[0009] The first point cloud data is transformed from the first coordinate system to the machine coordinate system based on the first transformation matrix to obtain the first transformed point cloud data, and the second point cloud data is transformed from the second coordinate system to the machine coordinate system based on the second transformation matrix to obtain the second transformed point cloud data.

[0010] A transformation matrix is ​​determined to align the first transformed point cloud data with the second transformed point cloud data, resulting in a third transformation matrix, which is then calibrated to the second depth camera.

[0011] Secondly, embodiments of this application provide a dual-depth camera extrinsic parameter calibration device, applied to a robot, the robot being equipped with a first depth camera and a second depth camera, the device comprising:

[0012] The acquisition module is used to acquire the first point cloud data obtained by the first depth camera in a preset scene, and to acquire the second point cloud data obtained by the second depth camera in a preset scene.

[0013] The first determining module is used to determine the transformation matrix for transforming the first point cloud data from the first coordinate system to the machine coordinate system, obtain the first transformation matrix, and calibrate the first transformation matrix to the first depth camera. The first coordinate system is the coordinate system of the first depth camera, and the machine coordinate system is the coordinate system of the robot.

[0014] The second determining module is used to determine the transformation matrix for transforming the second point cloud data from the second coordinate system to the machine coordinate system, obtain the second transformation matrix, and calibrate the second transformation matrix to the second depth camera, wherein the second coordinate system is the coordinate system of the second depth camera;

[0015] The transformation module is used to transform the first point cloud data from the first coordinate system to the machine coordinate system based on the first transformation matrix to obtain the first transformed point cloud data, and to transform the second point cloud data from the second coordinate system to the machine coordinate system based on the second transformation matrix to obtain the second transformed point cloud data.

[0016] The third determining module is used to determine the transformation matrix that aligns the first transformed point cloud data with the second transformed point cloud data, obtain the third transformation matrix, and calibrate the third transformation matrix to the second depth camera.

[0017] Thirdly, embodiments of this application provide a robot, which includes:

[0018] At least one processor; and a memory communicatively connected to the at least one processor;

[0019] The memory stores a computer program that can be executed by at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the dual-depth camera extrinsic calibration method of any embodiment of this application.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the dual-depth camera extrinsic parameter calibration method as described in any embodiment of this application.

[0021] In this embodiment, by determining a first transformation matrix to transform the first point cloud data from the first coordinate system to the machine coordinate system, and a second transformation matrix to transform the second point cloud data from the second coordinate system to the machine coordinate system, the first and second point cloud data can be transformed into the same machine coordinate system, thereby determining the precise position of the depth camera in three-dimensional space. Then, by further determining a third transformation matrix to align the first and second transformed point cloud data, the relative position and attitude relationship between the first and second depth cameras can be determined more accurately. Furthermore, it can effectively account for errors caused by factors such as the installation position, installation angle, and measurement depth of the depth cameras, thereby reducing extrinsic parameter errors caused by installation and measurement factors, thus improving the accuracy and precision of extrinsic parameter calibration. Simultaneously, the collaborative work of the two depth cameras and precise extrinsic parameter calibration can effectively reduce positioning errors caused by the limitations of a single camera (such as planar detection degradation and a small field of view) or inaccurate extrinsic parameters, enabling the robot to better adapt to complex and changing environments, thereby enhancing the robot's perception stability and improving its positioning accuracy and precision. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating a dual-depth camera extrinsic parameter calibration method provided in an embodiment of this application;

[0024] Figure 2 This is another flowchart illustrating the dual-depth camera extrinsic parameter calibration method provided in this application embodiment;

[0025] Figure 3 This is an example diagram of a preset scenario provided in an embodiment of this application;

[0026] Figure 4 This is a schematic diagram of a dual-depth camera extrinsic parameter calibration device provided in an embodiment of this application;

[0027] Figure 5 This is a schematic diagram of the structure of a robot provided in an embodiment of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0029] It should be noted that the terms "first," "second," "target," and "original," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein. Furthermore, the terms "comprising," "having," and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Figure 1 This is a flowchart illustrating a dual-depth camera extrinsic parameter calibration method provided in this application embodiment. This embodiment can be applied to scenarios requiring extrinsic parameter calibration of dual-depth cameras. The dual-depth camera extrinsic parameter calibration method provided in this embodiment can be executed by a dual-depth camera extrinsic parameter calibration device provided in this application embodiment. This device can be implemented through software and / or hardware. In a specific embodiment, the dual-depth camera extrinsic parameter calibration device can be integrated into a robot, for example, a robotic vacuum cleaner. The executing entity of this method can be a robot.

[0031] In one specific implementation, the robot is equipped with a first depth camera and a second depth camera, which are two camera devices used to acquire depth information. The first depth camera and the second depth camera are installed at different positions on the robot, so that they can observe the surrounding environment from different positions and angles, thereby providing data support for subsequent external parameter calibration and robot localization and perception tasks. For example, the first depth camera and the second depth camera can be TOF (Time-of-Flight) cameras.

[0032] See Figure 1 The dual-depth camera extrinsic parameter calibration method in this embodiment includes, but is not limited to, the following steps:

[0033] S110. Obtain the first point cloud data obtained by the first depth camera in the preset scene, and obtain the second point cloud data obtained by the second depth camera in the preset scene.

[0034] The preset scene is a specific environmental area specifically set up for the depth camera to collect data during the extrinsic parameter calibration process. The preset scene can include various objects, planes, and geometrically structured objects, etc., with the aim of enabling the depth camera to acquire representative point cloud data in order to more accurately calibrate the depth camera's extrinsic parameters. For example, the preset scene can be an indoor area with simple geometrically shaped objects placed there, or an outdoor local space with specific texture and depth variations.

[0035] The first point cloud data is a dataset collected by the first depth camera from a preset scene. It consists of a large number of three-dimensional spatial points, each containing spatial location information, used to represent the three-dimensional structure and depth information of the preset scene from the perspective of the first depth camera. The second point cloud data is a dataset collected by the second depth camera from a preset scene. It consists of a large number of three-dimensional spatial points, each containing spatial location information, used to represent the three-dimensional structure and depth information of the preset scene from the perspective of the second depth camera.

[0036] Specifically, in order to ensure that the first point cloud data and the second point cloud data are acquired at the same time as the preset scene, a synchronization triggering mechanism can be established. That is, it can be achieved through hardware synchronization (such as using the same trigger signal source to connect the first depth camera and the second depth camera) or software synchronization (such as setting synchronization instructions in the data acquisition program).

[0037] When the synchronization triggering mechanism is triggered, the robot can simultaneously start the first depth camera and the second depth camera, and set the first depth camera and the second depth camera to collect data at the same frame rate; then, the first depth camera can start collecting data of the preset scene and process it through its built-in sensors and algorithms to obtain the first point cloud data. At the same time, the second depth camera can start collecting data of the same preset scene and process it through its built-in sensors and algorithms to obtain the second point cloud data.

[0038] The robot can then acquire first point cloud data from the first depth camera and second point cloud data from the second depth camera. It can then perform preprocessing on the first and second point cloud data, such as noise reduction and filtering, to improve data quality.

[0039] S120. Determine the transformation matrix for transforming the first point cloud data from the first coordinate system to the machine coordinate system, obtain the first transformation matrix, and calibrate the first transformation matrix to the first depth camera.

[0040] The first coordinate system is the coordinate system of the first depth camera, that is, the coordinate system of the first depth camera itself, and it is a three-dimensional rectangular coordinate system used to describe the spatial position relationship of the first point cloud data from the perspective of the first depth camera itself; the origin position and coordinate axis direction of the first coordinate system are determined by the internal parameters and installation method of the first depth camera.

[0041] The machine coordinate system is the robot's coordinate system, defined on the robot's body. It is used to describe the robot's position and posture in space, as well as the relative positional relationships of the robot's various components (including the first depth camera and the second depth camera). It provides a common reference framework for unifying the data from the first depth camera and the second depth camera, enabling all data to be processed and fused within the robot's overall spatial position description system.

[0042] The first transformation matrix is ​​a transformation matrix that transforms the first point cloud data from the first coordinate system to the machine coordinate system, and is used to define the relationship between the first coordinate system and the machine coordinate system. The first transformation matrix may include the rotation and translation information required to transform the first point cloud data in the first coordinate system to the machine coordinate system.

[0043] Specifically, after obtaining the first point cloud data, the transformation matrix for transforming the first point cloud data from the first coordinate system to the machine coordinate system can be determined. For example, in a preset scenario, feature points of a calibration object with obvious geometric features (such as the corner points of a chessboard calibration board or the vertices of a cube) can be obtained. At this time, there are multiple feature points. Then, the coordinates of each feature point in the first coordinate system are obtained from the first point cloud data. The coordinates of each feature point in the machine coordinate system are determined by the robot's kinematic model or other measurement methods. Then, feature point matching is performed to match the feature points in the first point cloud data with the feature points in the machine coordinate system to obtain matching point pairs. Based on the coordinates of each point in the matching point pair in the first coordinate system and the coordinates in the machine coordinate system, the transformation matrix from the first coordinate system to the machine coordinate system is calculated to obtain the first transformation matrix. The first transformation matrix can include a rotation matrix and a translation vector to describe the relative position and orientation between the first coordinate system and the machine coordinate system.

[0044] Then, the first transformation matrix can be calibrated to the first depth camera. That is, the first transformation matrix can be sent to the first depth camera, and the first depth camera can receive the first transformation matrix sent by the robot and write the first transformation matrix into the configuration file of the first depth camera to store the new extrinsic parameter information.

[0045] S130. Determine the transformation matrix for transforming the second point cloud data from the second coordinate system to the machine coordinate system, obtain the second transformation matrix, and calibrate the second transformation matrix to the second depth camera.

[0046] The second coordinate system is the coordinate system of the second depth camera, specifically a three-dimensional Cartesian coordinate system used to describe the spatial positional relationship of the second point cloud data from the camera's own perspective. The origin and axis orientation of the second coordinate system are determined by the internal parameters and installation method of the second depth camera. It should be noted that due to differences in camera installation position and angle, the origin and axis orientation of the second coordinate system differ from those of the first coordinate system.

[0047] The second transformation matrix is ​​a transformation matrix that transforms the second point cloud data from the second coordinate system to the machine coordinate system, and is used to define the relationship between the second coordinate system and the machine coordinate system. The second transformation matrix may include the rotation and translation information required to transform the second point cloud data in the second coordinate system to the machine coordinate system.

[0048] Specifically, after obtaining the second point cloud data, the transformation matrix for transforming the second point cloud data from the second coordinate system to the machine coordinate system can be determined. For example, in a preset scenario, feature points of a calibration object with obvious geometric features can be obtained. Then, the coordinates of each feature point in the second coordinate system are obtained from the second point cloud data. The coordinates of each feature point in the machine coordinate system are determined by the robot's kinematic model or other measurement methods. Then, feature point matching is performed to match the feature points in the second point cloud data with the feature points in the machine coordinate system to obtain matching point pairs. Based on the coordinates of each point in the matching point pair in the second coordinate system and the coordinates in the machine coordinate system, the transformation matrix from the second coordinate system to the machine coordinate system is calculated to obtain the second transformation matrix. The second transformation matrix can include rotation matrices and translation vectors to describe the relative position and orientation between the second coordinate system and the machine coordinate system.

[0049] Then, the second transformation matrix can be calibrated to the second depth camera. That is, the second transformation matrix can be sent to the second depth camera, and the second depth camera can receive the second transformation matrix sent by the robot and write the second transformation matrix into the configuration file of the second depth camera to store the new extrinsic parameter information.

[0050] S140. Based on the first transformation matrix, transform the first point cloud data from the first coordinate system to the machine coordinate system to obtain the first transformed point cloud data, and based on the second transformation matrix, transform the second point cloud data from the second coordinate system to the machine coordinate system to obtain the second transformed point cloud data.

[0051] The first transformed point cloud data is the point cloud data in the machine coordinate system after the first point cloud data has been transformed by the first transformation matrix. Its purpose is to unify the data acquired by the first depth camera into the machine coordinate system, providing a data foundation for subsequent alignment operations. The second transformed point cloud data is the point cloud data in the machine coordinate system after the second point cloud data has been transformed by the second transformation matrix. Its purpose is to unify the data acquired by the second depth camera into the machine coordinate system, providing a data foundation for subsequent alignment operations.

[0052] Specifically, after obtaining the first transformation matrix and the second transformation matrix, the first point cloud data can be transformed from the first coordinate system to the machine coordinate system based on the first transformation matrix. That is, for each point in the first point cloud data, the product of the coordinates of each point and the first transformation matrix can be calculated by matrix multiplication to obtain the coordinates of the transformed point corresponding to each point. At this time, the coordinates of the transformed point are in the machine coordinate system. The transformed points corresponding to each point are then assembled into the first transformed point cloud data to transform the coordinates of each point in the first point cloud data from the position in the first coordinate system to the corresponding position in the machine coordinate system, thereby realizing the conversion of data between different coordinate systems. Then, the second point cloud data can be transformed from the second coordinate system to the machine coordinate system based on the second transformation matrix. That is, for each point in the second point cloud data, the product of the coordinates of each point and the second transformation matrix can be calculated by matrix multiplication to obtain the coordinates of the transformed point corresponding to each point. At this time, the coordinates of the transformed point are in the machine coordinate system. The transformed points corresponding to each point are then assembled into the second transformed point cloud data to transform the coordinates of each point in the second point cloud data from the position in the second coordinate system to the corresponding position in the machine coordinate system, thereby realizing the conversion of data between different coordinate systems.

[0053] S150. Determine the transformation matrix that aligns the first transformed point cloud data with the second transformed point cloud data, obtain the third transformation matrix, and calibrate the third transformation matrix to the second depth camera.

[0054] The third transformation matrix is ​​a transformation matrix that aligns the first transformed point cloud data with the second transformed point cloud data; the third transformation matrix may include information such as rotation and translation.

[0055] Specifically, even if the first and second transformed point cloud data have been transformed into the machine coordinate system, they may still have differences in position and orientation due to factors such as camera installation errors and measurement errors. Therefore, it is necessary to further determine the transformation matrix to align the first and second transformed point cloud data.

[0056] For example, representative key points can be extracted from the first and second transformed point cloud data, and feature descriptors corresponding to the key points can be calculated. The feature descriptors are vectors that characterize the local geometric features around the key points. Then, using the calculated feature descriptors, the key points of the first and second transformed point cloud data are matched to obtain matching point pairs. Based on the matching point pairs, the transformation matrix that aligns the second and first transformed point cloud data is determined, resulting in the third transformation matrix. The purpose of the third transformation matrix is ​​to find an optimal spatial transformation so that the second and first transformed point cloud data can be accurately aligned in space.

[0057] Next, the third transformation matrix can be calibrated to the second depth camera. That is, the third transformation matrix can be sent to the second depth camera, which can then receive the third transformation matrix sent by the robot and write it into its configuration file to store the new extrinsic parameter information. By calibrating the third transformation matrix to the second depth camera, the extrinsic parameter relationship between the second and first depth cameras can be further corrected, thereby achieving more accurate dual-depth camera extrinsic parameter calibration.

[0058] The technical solution of this application embodiment, by determining a first transformation matrix to transform the first point cloud data from the first coordinate system to the machine coordinate system, and a second transformation matrix to transform the second point cloud data from the second coordinate system to the machine coordinate system, can transform the first point cloud data and the second point cloud data into the same machine coordinate system, thereby determining the precise position of the depth camera in three-dimensional space. Then, by further determining a third transformation matrix to align the first transformed point cloud data and the second transformed point cloud data, the relative position and attitude relationship between the first depth camera and the second depth camera can be determined more accurately. It can also effectively take into account the errors caused by factors such as the installation position, installation angle, and measurement depth of the depth camera, thereby reducing the external parameter errors caused by installation and measurement factors, and thus improving the accuracy and precision of external parameter calibration. At the same time, the collaborative work of the two depth cameras and the precise external parameter calibration can effectively reduce the positioning errors caused by the limitations of a single camera (such as the degradation of planar detection and small field of view) or inaccurate external parameters, enabling the robot to better adapt to complex and changing environments, thereby enhancing the robot's perception stability and improving the robot's positioning accuracy and precision.

[0059] The following further describes a dual-depth camera extrinsic parameter calibration method provided by an embodiment of this application. Figure 2 This is another schematic flowchart illustrating the dual-depth camera extrinsic parameter calibration method provided in this application. This application's embodiment is an optimization based on the above embodiments. See also... Figure 2The method in this embodiment includes, but is not limited to, the following steps:

[0060] S201. Obtain the first point cloud data obtained by the first depth camera in the preset scene, and obtain the second point cloud data obtained by the second depth camera in the preset scene.

[0061] Optional, such as Figure 3 The diagram shown is an example of a preset scenario provided in an embodiment of this application. Figure 3 The preset scene includes a calibration object with a clear geometric structure, namely a calibration object composed of four planar plates that are perpendicular to each other and combined at a certain angle; Figure 3 The first depth camera is installed on the front part of the robot, and the second depth camera is installed on the side part of the robot.

[0062] Optionally, the first point cloud data may include left plane point cloud data and right plane point cloud data. The left plane point cloud data is the set of point clouds of the planes that make up the calibration object and are located to the left of the first depth camera; the right plane point cloud data is the set of point clouds of the planes that make up the calibration object and are located to the right of the first depth camera.

[0063] S202. Determine the plane intersection vector based on the left plane point cloud data and the right plane point cloud data.

[0064] Among them, the planar intersection vector is used to describe the intersection line between the left-plane point cloud data and the right-plane point cloud data.

[0065] Specifically, after obtaining the first point cloud data, the left plane point cloud data and the right plane point cloud data can be obtained from the first point cloud data. Then, plane fitting is performed on the left plane point cloud data to obtain the left plane equation, and plane fitting is performed on the right plane point cloud data to obtain the right plane equation. Then, the normal vector of the left plane is determined based on the left plane equation to obtain the left plane normal vector, and the normal vector of the right plane is determined based on the right plane equation to obtain the right plane normal vector. Finally, the cross product of the left plane normal vector and the right plane normal vector is performed to obtain the plane intersection vector.

[0066] S203. Determine the transformation matrix that transforms the plane intersection vector to be parallel to the vertical axis of the machine coordinate system, obtain the vertical axis transformation matrix, and transform the plane intersection vector based on the vertical axis transformation matrix to obtain the transformed intersection vector.

[0067] The vertical axis transformation matrix is ​​a transformation matrix that transforms the plane intersection vector to a position parallel to the vertical axis (i.e., the z-axis) of the machine coordinate system; the vertical axis transformation matrix may include rotation transformation information.

[0068] The transformed intersection vector is the vector obtained by transforming the planar intersection vector through the vertical axis transformation matrix. That is, the direction of the transformed intersection vector is parallel to the vertical axis of the machine coordinate system.

[0069] Specifically, the transformation matrix that transforms the planar intersection vector to be parallel to the vertical axis of the machine coordinate system is determined, resulting in the vertical axis transformation matrix, including Sa1-Sa3:

[0070] Sa1: Perform the cross product of the plane intersection vector and the vertical unit vector of the machine coordinate system to obtain the first rotation axis.

[0071] The first rotation axis is the axis around which the plane intersection vector is rotated during the process of transforming the plane intersection vector to be parallel to the vertical axis of the machine coordinate system.

[0072] Specifically, after obtaining the plane intersection vector, in order to transform the plane intersection vector to be parallel to the vertical axis of the machine coordinate system, the plane intersection vector and the unit vector of the vertical axis of the machine coordinate system can be cross-producted, and the result of the cross product is determined as the first rotation axis.

[0073] Sa2. Perform a dot product of the plane intersection vector and the vertical axis unit vector to obtain the first rotation angle.

[0074] Wherein, the first rotation angle is the angle by which the plane intersection vector needs to rotate around the first rotation axis, so that the plane intersection vector is parallel to the vertical axis of the machine coordinate system.

[0075] Specifically, after obtaining the first axis of rotation, the dot product of the plane intersection vector and the vertical axis unit vector can be performed, and the first rotation angle can be determined based on the dot product result. The formula for calculating the first rotation angle θ1 is as follows:

[0076]

[0077] Where l represents the plane intersection vector, n z |l| represents the vertical unit vector of the machine coordinate system, |n| represents the magnitude of the plane intersection vector, and |n| represents the vertical unit vector of the machine coordinate system. z | represents the magnitude of the unit vector along the vertical axis of the machine coordinate system.

[0078] Sa3. Determine the vertical axis transformation matrix based on the first rotation axis and the first rotation angle.

[0079] Specifically, after obtaining the first rotation angle, the antisymmetric matrix of the first rotation axis can be determined, and the vertical axis transformation matrix, i.e., the vertical axis transformation matrix T, can be determined based on the antisymmetric matrix of the first rotation axis and the first rotation angle. 竖轴 The calculation formula is:

[0080] T 竖轴=I+sin(θ1)K1+(1-cos(θ1))K1 2

[0081] Where K1 represents the antisymmetric matrix of the first rotation axis, and I represents the 3×3 identity matrix.

[0082] In this embodiment, the vertical axis transformation matrix determined by the plane intersection vector and the vertical axis unit vector of the machine coordinate system can improve the accuracy and efficiency of determining the vertical axis transformation matrix. This accurately transforms the plane intersection vector to be parallel to the vertical axis of the machine coordinate system and reduces the accumulation of direction errors in subsequent calibration processes. This helps to align the vertical axis of the first coordinate system with the vertical axis of the machine coordinate system and corrects the roll and pitch angles, providing data support for the subsequent determination of the first transformation matrix.

[0083] After determining the vertical axis transformation matrix, the plane intersection vector can be transformed based on the vertical axis transformation matrix. That is, the transformed intersection vector can be obtained by calculating the product of the plane intersection vector and the vertical axis transformation matrix through matrix multiplication. At this time, the transformed intersection vector is parallel to the vertical axis of the machine coordinate system, thereby aligning the vertical axis of the first coordinate system and the vertical axis of the machine coordinate system to a certain extent, which facilitates the subsequent external parameter calibration.

[0084] S204. Determine the intersection point of the transformed intersection vector with the horizontal plane of the machine coordinate system to obtain the planar intersection point, and determine the transformation matrix to transform the planar intersection point to the horizontal axis of the machine coordinate system to obtain the horizontal axis transformation matrix.

[0085] In this context, the horizontal plane of the machine coordinate system is the plane formed by the horizontal and vertical axes of the machine coordinate system, specifically the plane consisting of the x-axis, y-axis, and origin. The plane intersection point is the point where the transformed intersection vector intersects the horizontal plane of the machine coordinate system. The horizontal transformation matrix is ​​the transformation matrix that transforms the plane intersection point onto the horizontal axis (x-axis) of the machine coordinate system; the horizontal transformation matrix can include rotation transformation information.

[0086] Specifically, since the transformation intersection vector is parallel to the vertical axis of the machine coordinate system, that is, the transformation intersection vector is perpendicular to the horizontal plane of the machine coordinate system, the intersection point of the transformation intersection vector and the horizontal plane of the machine coordinate system can be determined as the plane intersection point after obtaining the transformation intersection vector.

[0087] To further refine the alignment between the first coordinate system and the machine coordinate system, the plane intersection point can be transformed onto the horizontal axis of the machine coordinate system. This requires determining the transformation matrix that transforms the plane intersection point onto the horizontal axis of the machine coordinate system, thus obtaining the horizontal axis transformation matrix.

[0088] Furthermore, the transformation matrix that transforms the plane intersection points to the horizontal axis of the machine coordinate system is determined, resulting in the horizontal axis transformation matrix, including Sb1-Sb2:

[0089] Sb1. Determine the second rotation angle based on the intersection of the planes and the unit vector of the horizontal axis of the machine coordinate system.

[0090] The second rotation angle is the angle by which the plane intersection point needs to be rotated so that the plane intersection point is transformed onto the horizontal axis of the machine coordinate system. It is used to measure the degree of deviation between the position of the plane intersection point and the horizontal axis of the machine coordinate system.

[0091] Specifically, the second rotation angle can be determined by multiplying the coordinate vectors corresponding to the intersection points of the planes with the unit vector of the horizontal axis, and then by calculating the result of the dot product. The formula for calculating the second rotation angle θ2 is as follows:

[0092]

[0093] Where P1 represents the coordinate vector corresponding to the intersection point of the planes, and n x |P1| represents the unit vector of the horizontal axis in the machine coordinate system, |P1| represents the magnitude of the coordinate vector corresponding to the intersection point of the planes, and |n| represents the horizontal axis of the machine coordinate system. x | represents the magnitude of the unit vector along the horizontal axis of the machine coordinate system.

[0094] Sb2. Determine the horizontal axis transformation matrix based on the second rotation angle and the vertical axis unit vector of the machine coordinate system.

[0095] Specifically, during the transformation of the planar intersection point to the horizontal axis of the machine coordinate system, the rotation axis of the planar intersection point is the vertical axis of the machine coordinate system. Therefore, after obtaining the second rotation angle, the antisymmetric matrix of the vertical axis unit vector can be determined, and the horizontal axis transformation matrix, i.e., the horizontal axis transformation matrix T, can be determined based on the antisymmetric matrix of the vertical axis unit vector and the second rotation angle. 横轴 The calculation formula is:

[0096] T 横轴 =I+sin(θ2)K2+(1-cos(θ2))K2 2

[0097] Where K2 represents the antisymmetric matrix of the unit vectors along the vertical axis.

[0098] In this embodiment, by using the plane intersection point, the horizontal axis unit vector and the vertical axis unit vector of the machine coordinate system, the accuracy and efficiency of determining the horizontal axis transformation matrix can be improved. This allows the plane intersection point to be precisely rotated onto the horizontal axis of the machine coordinate system, and reduces the accumulation of position errors in the subsequent calibration process. This helps to align the horizontal axis of the first coordinate system with the horizontal axis of the machine coordinate system, and corrects the yaw angle, providing data support for the subsequent determination of the first transformation matrix.

[0099] S205. Determine the first transformation matrix based on the vertical axis transformation matrix, the horizontal axis transformation matrix, and the intersection of the planes.

[0100] Specifically, the first transformation matrix, including Sc1-Sc3, is determined based on the vertical axis transformation matrix, the horizontal axis transformation matrix, and the plane intersection points:

[0101] Sc1. Transform the intersection points of the planes based on the horizontal axis transformation matrix to obtain the transformed intersection points.

[0102] The transformation intersection point is the point after the plane intersection point is transformed by the horizontal axis transformation matrix, that is, the transformation intersection point is located on the horizontal axis of the machine coordinate system.

[0103] Specifically, the coordinate vector corresponding to the intersection point of the plane can be obtained by multiplying the coordinate vector corresponding to the intersection point of the plane with the horizontal axis transformation matrix through matrix multiplication.

[0104] Sc2. The transformed intersection point is translated based on a preset distance to obtain the target intersection point.

[0105] The preset distance is a pre-set distance value used to translate the transformed intersection point. Users can adjust and set this preset distance according to actual usage needs; however, this embodiment does not impose specific limitations on it. The target intersection point is the point obtained after translating the transformed intersection point.

[0106] Specifically, since the distance between the robot's center and the plane intersection line of the calibrated object (i.e., the plane intersection line represented by the plane intersection vector in S202) is a preset distance, the horizontal axis coordinate of the transformed intersection point after transformation by the horizontal axis transformation matrix should be the preset distance. Therefore, after obtaining the transformed intersection point, the horizontal axis coordinate of the transformed intersection point can be corrected and translated based on the preset distance. That is, the difference between the preset distance and the horizontal axis coordinate of the transformed intersection point can be calculated to obtain the horizontal axis offset. Then, the horizontal axis coordinate of the transformed intersection point is updated to the horizontal axis offset to obtain the target intersection point. For example, as shown... Figure 3 As shown, the distance between the center of the robot and the plane intersection of the calibrated object is 0.56 meters (i.e., the preset distance). If the coordinates of the intersection point are changed to (0.3,0,0), then the coordinates of the target intersection point are (0.26,0,0).

[0107] Sc3. Determine the first transformation matrix based on the vertical axis transformation matrix, the horizontal axis transformation matrix, and the target intersection point.

[0108] Specifically, after obtaining the target intersection point, the transpose matrix of the coordinate vector corresponding to the target intersection point can be determined. Then, based on the vertical axis transformation matrix, the horizontal axis transformation matrix, and the transpose matrix of the coordinate vector corresponding to the target intersection point, the first transformation matrix T1 can be determined. That is, the first transformation matrix T1 can be expressed as:

[0109]

[0110] Where P0 represents the transpose of the coordinate vector corresponding to the target intersection point.

[0111] In this embodiment, by correcting and translating the transformation intersection point by a preset distance, the horizontal axis of the first coordinate system and the horizontal axis of the machine coordinate system can be further aligned. By using the vertical axis transformation matrix, the horizontal axis transformation matrix, and the target intersection point, the transformation relationship between the camera coordinate system and the machine coordinate system in different directions can be comprehensively considered, thereby improving the accuracy and efficiency of determining the first transformation matrix. This accurately transforms the first point cloud data into the machine coordinate system, providing data support for the subsequent determination of the third transformation matrix.

[0112] S206. The first transformation matrix is ​​calibrated to the first depth camera.

[0113] S207. Determine the transformation matrix for transforming the second point cloud data from the second coordinate system to the machine coordinate system, obtain the second transformation matrix, and calibrate the second transformation matrix to the second depth camera.

[0114] Specifically, the second transformation matrix can be determined using the same processing flow as S202 to S205, and the second transformation matrix can be calibrated to the second depth camera. For detailed implementation details, please refer to the description of S202 to S205.

[0115] S208. Based on the first transformation matrix, transform the first point cloud data from the first coordinate system to the machine coordinate system to obtain the first transformed point cloud data, and based on the second transformation matrix, transform the second point cloud data from the second coordinate system to the machine coordinate system to obtain the second transformed point cloud data.

[0116] S209. The first transformed point cloud data is determined as the target point cloud data, and the second transformed point cloud data is determined as the source point cloud data.

[0117] The target point cloud data is the point cloud data used as a reference standard when aligning the first transformed point cloud data and the second transformed point cloud data. The source point cloud data is the point cloud data that needs to be adjusted when aligning the first transformed point cloud data and the second transformed point cloud data.

[0118] Specifically, due to errors in the installation angle (e.g., the first depth camera is located directly in front of the robot, and the second depth camera is located to the side of the robot) and errors in the measurement depth, the first transformed point cloud data and the second transformed point cloud data are not completely aligned. In this case, further alignment operations are needed for the first transformed point cloud data and the second transformed point cloud data. Therefore, the first transformed point cloud data can be determined as the target point cloud data, and the second transformed point cloud data can be determined as the source point cloud data.

[0119] It should be noted that the target point cloud data and source point cloud data can be adjusted according to the difference in the actual installation angle. That is, if the first depth camera is located on the side of the robot and the second depth camera is located in front of the robot, then the first transformed point cloud data is determined as the source point cloud data and the second transformed point cloud data is determined as the target point cloud data.

[0120] S210. Minimize the residuals using the maximum likelihood estimation algorithm to obtain the third transformation matrix.

[0121] The residual is the distance between matching points in the source point cloud data and the target point cloud data.

[0122] Specifically, when aligning the first transformed point cloud data and the second transformed point cloud data, matching point pairs can be determined from the source point cloud data and the target point cloud data, and the distance between the two points in the matching point pair can be determined as the residual. For example, as shown... Figure 3 As shown, the calibration object formed by four planar plates perpendicular to each other and combined at a certain angle has a specific geometric relationship. Therefore, matching point pairs can be determined from the source point cloud data and the target point cloud data based on the geometric relationship of the calibration object.

[0123] Then, the maximum likelihood estimation algorithm can be used to minimize the residuals to obtain the third transformation matrix, specifically including Sd1-Sd2:

[0124] Sd1. Minimize the residuals using the maximum likelihood estimation algorithm to obtain the intermediate transformation matrix.

[0125] The intermediate transformation matrix is ​​a transformation matrix obtained by minimizing the residual through the maximum likelihood estimation algorithm. The intermediate transformation matrix can include rotation and translation information to transform the source point cloud data in the machine coordinate system so that the distance (i.e., residual) between the matching points in the source point cloud data and the target point cloud data is minimized.

[0126] Specifically, the source point cloud data is known to be A = {a1, a2, ..., a...} n The target point cloud data is B = {b1, b2, ..., b}. n Furthermore, all points within the neighborhood of the source point cloud data and the neighborhood of the target point cloud data conform to a normal distribution, i.e. in For a i The mean of the field For a i Standard deviation of the field For b i The mean of the field For b i Given the standard deviation of the field, the residuals can be expressed as:

[0127] d i =b i -T0a i

[0128] Among them, the i-th point in the source point cloud data (i.e., a) i ) and the i-th point in the target point cloud data (i.e., b) i Let ) be the points that match each other, and d be the points that match each other. i Indicates a i With b i The residuals between; T0 represents the intermediate transformation matrix.

[0129] The residuals corresponding to each pair of matching points obtained thus also conform to a normal distribution, and according to the properties of the normal distribution:

[0130]

[0131] Among them, T0 T This represents the transpose of the intermediate transformation matrix.

[0132] Then, the residuals are minimized using the maximum likelihood estimation algorithm to obtain the intermediate transformation matrix. The formula for calculating the intermediate transformation matrix is:

[0133]

[0134] Sd2. Determine the third transformation matrix based on the intermediate transformation matrix and the preset installation angle.

[0135] The preset installation angle is an angle predetermined based on the installation angles of the first depth camera and the second depth camera, that is, the relative angle between the first depth camera and the second depth camera.

[0136] Specifically, since the intermediate transformation matrix does not take into account the relative angle between the first depth camera and the second depth camera, after obtaining the intermediate transformation matrix, the vertical axis of the machine coordinate system can be used as the rotation axis, the preset installation angle can be used as the rotation angle, and the third transformation matrix can be determined based on the rotation axis and the rotation angle.

[0137] In this embodiment, minimizing the residuals using the maximum likelihood estimation algorithm improves the accuracy and efficiency of determining the intermediate transformation matrix, thereby compensating for the calibration error of a single camera. Then, combining the intermediate transformation matrix with the preset installation angle further calibrates the extrinsic parameters, effectively reducing the extrinsic parameter errors caused by the camera's installation position and angle. This improves the accuracy and efficiency of determining the third transformation matrix, making it more accurately reflect the real spatial relationships between depth cameras and between the depth camera and the robot, thus improving the accuracy of extrinsic parameter calibration.

[0138] S211. The third transformation matrix is ​​calibrated to the second depth camera.

[0139] The technical solution of this application embodiment can acquire first point cloud data obtained by a first depth camera capturing a preset scene, and acquire second point cloud data obtained by a second depth camera capturing a preset scene. Next, based on the left and right plane point cloud data, a plane intersection vector is determined. Then, a transformation matrix is ​​determined to transform the plane intersection vector to a value parallel to the vertical axis of the machine coordinate system, resulting in a vertical axis transformation matrix. The plane intersection vector is then transformed based on the vertical axis transformation matrix to obtain a transformed intersection vector. The intersection point of the transformed intersection vector with the horizontal plane of the machine coordinate system is then determined, resulting in a plane intersection point. A transformation matrix is ​​then determined to transform the plane intersection point to the horizontal axis of the machine coordinate system, resulting in a horizontal axis transformation matrix. Finally, a first transformation matrix is ​​determined based on the vertical axis transformation matrix, the horizontal axis transformation matrix, and the plane intersection point. This first transformation matrix is ​​then calibrated to the first depth camera. The plane intersection vector and the plane intersection point provide a clear geometric meaning and reference for determining the first transformation matrix, thereby improving the accuracy and efficiency of determining the first transformation matrix and reducing extrinsic parameter errors caused by inaccurate coordinate transformation. This allows for more precise calibration of the spatial position and attitude of the first depth camera relative to the machine coordinate system.

[0140] After obtaining the first and second transformed point cloud data, the first transformed point cloud data can be identified as the target point cloud data, and the second transformed point cloud data as the source point cloud data. Then, the maximum likelihood estimation algorithm is used to minimize the residuals to obtain the third transformation matrix. The third transformation matrix is ​​then calibrated to the second depth camera, realizing collaborative extrinsic parameter calibration between the two depth cameras. This effectively accounts for errors caused by factors such as the installation position, installation angle, and measurement depth of the depth cameras. Furthermore, based on a data-driven optimization process, the accuracy and efficiency of determining the third transformation matrix are further improved, and the relative position and attitude between the first and second depth cameras are determined more accurately, thereby improving the precision and accuracy of extrinsic parameter calibration. Through the first and second transformation matrices, data can be transformed from their respective camera coordinate systems to the machine coordinate system. Combined with the third transformation matrix, the data matching relationship between the first and second depth cameras is further optimized, thus providing the robot with richer and more accurate environmental perception. This enables the robot to better adapt to complex and changing environments, thereby improving the robot's positioning accuracy and precision.

[0141] Figure 4 This is a schematic diagram of a dual-depth camera extrinsic parameter calibration device provided in an embodiment of this application, with reference to... Figure 4 The dual-depth camera extrinsic calibration device may include:

[0142] The acquisition module 410 is used to acquire first point cloud data obtained by the first depth camera in a preset scene, and to acquire second point cloud data obtained by the second depth camera in a preset scene.

[0143] The first determining module 420 is used to determine the transformation matrix for transforming the first point cloud data from the first coordinate system to the machine coordinate system, obtain the first transformation matrix, and calibrate the first transformation matrix to the first depth camera, wherein the first coordinate system is the coordinate system of the first depth camera and the machine coordinate system is the coordinate system of the robot.

[0144] The second determining module 430 is used to determine the transformation matrix for transforming the second point cloud data from the second coordinate system to the machine coordinate system, obtain the second transformation matrix, and calibrate the second transformation matrix to the second depth camera, wherein the second coordinate system is the coordinate system of the second depth camera;

[0145] The transformation module 440 is used to transform the first point cloud data from the first coordinate system to the machine coordinate system based on the first transformation matrix to obtain the first transformed point cloud data, and to transform the second point cloud data from the second coordinate system to the machine coordinate system based on the second transformation matrix to obtain the second transformed point cloud data.

[0146] The third determining module 450 is used to determine the transformation matrix that aligns the first transformed point cloud data with the second transformed point cloud data, obtain the third transformation matrix, and calibrate the third transformation matrix to the second depth camera.

[0147] In one embodiment, the first point cloud data includes left plane point cloud data and right plane point cloud data. The first determining module 420 determines a transformation matrix to transform the first point cloud data from the first coordinate system to the machine coordinate system, obtaining a first transformation matrix, including: determining a plane intersection vector based on the left plane point cloud data and the right plane point cloud data; determining a transformation matrix to transform the plane intersection vector to be parallel to the vertical axis of the machine coordinate system, obtaining a vertical axis transformation matrix, and transforming the plane intersection vector based on the vertical axis transformation matrix to obtain a transformed intersection vector; determining the intersection point of the transformed intersection vector with the horizontal plane of the machine coordinate system, obtaining a plane intersection point, and determining a transformation matrix to transform the plane intersection point to be on the horizontal axis of the machine coordinate system, obtaining a horizontal axis transformation matrix; and determining the first transformation matrix based on the vertical axis transformation matrix, the horizontal axis transformation matrix, and the plane intersection point.

[0148] In one embodiment, the first determining module 420 determines a transformation matrix to transform the planar intersection vector to a form parallel to the vertical axis of the machine coordinate system, thereby obtaining a vertical axis transformation matrix. This includes: performing a cross product of the planar intersection vector and the vertical axis unit vector of the machine coordinate system to obtain a first rotation axis; performing a dot product of the planar intersection vector and the vertical axis unit vector to obtain a first rotation angle; and determining the vertical axis transformation matrix based on the first rotation axis and the first rotation angle.

[0149] In one embodiment, the first determining module 420 determines the transformation matrix that transforms the plane intersection point to the horizontal axis of the machine coordinate system, thereby obtaining the horizontal axis transformation matrix, including: determining a second rotation angle based on the plane intersection point and the horizontal axis unit vector of the machine coordinate system; and determining the horizontal axis transformation matrix based on the second rotation angle and the vertical axis unit vector of the machine coordinate system.

[0150] In one embodiment, the first determining module 420 determines a first transformation matrix based on the vertical axis transformation matrix, the horizontal axis transformation matrix, and the plane intersection point, including: transforming the plane intersection point based on the horizontal axis transformation matrix to obtain a transformed intersection point; translating the transformed intersection point based on a preset distance to obtain a target intersection point; and determining the first transformation matrix based on the vertical axis transformation matrix, the horizontal axis transformation matrix, and the target intersection point.

[0151] In one embodiment, the third determining module 450 determines a transformation matrix to align the first transformed point cloud data with the second transformed point cloud data, thereby obtaining a third transformation matrix. This includes: determining the first transformed point cloud data as the target point cloud data and the second transformed point cloud data as the source point cloud data; minimizing the residuals using a maximum likelihood estimation algorithm to obtain the third transformation matrix, where the residuals are the distances between matching points in the source point cloud data and the target point cloud data.

[0152] In one embodiment, the third determining module 450 uses the maximum likelihood estimation algorithm to minimize the residuals to obtain a third transformation matrix, including: using the maximum likelihood estimation algorithm to minimize the residuals to obtain an intermediate transformation matrix; and determining the third transformation matrix based on the intermediate transformation matrix and a preset installation angle.

[0153] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is merely an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0154] The dual-depth camera extrinsic parameter calibration device provided in this embodiment can be applied to the dual-depth camera extrinsic parameter calibration method provided in any of the above embodiments, and has the corresponding functions and beneficial effects.

[0155] Figure 5 This is a schematic diagram of the structure of a robot provided in an embodiment of this application. Figure 5 A block diagram is shown that is suitable for implementing an exemplary robot 11 according to embodiments of the present application. Figure 5 The robot 11 shown is merely an example and should not impose any limitations on the functionality and scope of use of this embodiment.

[0156] like Figure 5 As shown, robot 11 is represented in the form of a general-purpose computing electronic device. The components of robot 11 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).

[0157] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0158] Robot 11 typically includes a variety of computer-readable media. These media can be any available media that can be accessed by Robot 11, including volatile and non-volatile media, movable and non-movable media.

[0159] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Robot 11 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 5 Not shown; usually referred to as a "hard drive"). Although Figure 5 As not shown, disk drives for reading and writing to removable non-volatile disks (e.g., "floppy disks") and optical disc drives for reading and writing to removable non-volatile optical discs (e.g., CD-ROMs, DVD-ROMs, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0160] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of this application.

[0161] Robot 11 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with the robot 11, and / or with any device that enables the robot 11 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, robot 11 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20.

[0162] like Figure 5 As shown, network adapter 20 communicates with other modules of robot 11 via bus 18. It should be understood that, although... Figure 5 As not shown, other hardware and / or software modules can be used in conjunction with robot 11, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0163] The processing unit 16 executes various functional applications and page displays by running programs stored in the system memory 28, such as implementing a dual-depth camera extrinsic calibration method provided in any embodiment of this application.

[0164] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements, for example, a dual-depth camera extrinsic calibration method provided in any embodiment of this application.

[0165] Those skilled in the art will understand that the modules or steps described above in this application can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0166] Furthermore, the acquisition, storage, use, and processing of data in this application's technical solution all comply with relevant national laws and regulations.

[0167] Note that the above are merely preferred embodiments and the technical principles employed in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments. Many other equivalent embodiments may be included without departing from the inventive concept of this application, and the scope of this application is determined by the scope of the appended claims.

Claims

1. A method for calibrating extrinsic parameters of a dual-depth camera, characterized in that, Applied to a robot equipped with a first depth camera and a second depth camera, the method includes: Acquire first point cloud data obtained by the first depth camera capturing a preset scene, and acquire second point cloud data obtained by the second depth camera capturing the preset scene; A transformation matrix is ​​determined to transform the first point cloud data from the first coordinate system to the machine coordinate system, a first transformation matrix is ​​obtained, and the first transformation matrix is ​​calibrated to the first depth camera, wherein the first coordinate system is the coordinate system of the first depth camera, and the machine coordinate system is the coordinate system of the robot; Determine the transformation matrix that transforms the second point cloud data from the second coordinate system to the machine coordinate system, obtain the second transformation matrix, and calibrate the second transformation matrix to the second depth camera, wherein the second coordinate system is the coordinate system of the second depth camera; Based on the first transformation matrix, the first point cloud data is transformed from the first coordinate system to the machine coordinate system to obtain the first transformed point cloud data, and based on the second transformation matrix, the second point cloud data is transformed from the second coordinate system to the machine coordinate system to obtain the second transformed point cloud data. A transformation matrix is ​​determined to align the first transformed point cloud data with the second transformed point cloud data, resulting in a third transformation matrix, which is then calibrated to the second depth camera. The step of determining the transformation matrix for aligning the first transformed point cloud data with the second transformed point cloud data to obtain the third transformation matrix includes: The first transformed point cloud data is determined as the target point cloud data, and the second transformed point cloud data is determined as the source point cloud data; The residual is minimized using the maximum likelihood estimation algorithm to obtain an intermediate transformation matrix. The third transformation matrix is ​​determined based on the intermediate transformation matrix and the preset installation angle. The residual is the distance between matching points in the source point cloud data and the target point cloud data. The intermediate transformation matrix is ​​a transformation matrix obtained by minimizing the residual using the maximum likelihood estimation algorithm.

2. The dual-depth camera extrinsic parameter calibration method according to claim 1, characterized in that, The first point cloud data includes left plane point cloud data and right plane point cloud data. Determining the transformation matrix to transform the first point cloud data from the first coordinate system to the machine coordinate system, and obtaining the first transformation matrix, includes: Determine the plane intersection vector based on the left plane point cloud data and the right plane point cloud data; Determine the transformation matrix that transforms the planar intersection vector to be parallel to the vertical axis of the machine coordinate system, obtain the vertical axis transformation matrix, and transform the planar intersection vector based on the vertical axis transformation matrix to obtain the transformed intersection vector; Determine the intersection point of the transformed intersection vector with the horizontal plane of the machine coordinate system to obtain the planar intersection point, and determine the transformation matrix that transforms the planar intersection point to the horizontal axis of the machine coordinate system to obtain the horizontal axis transformation matrix; The first transformation matrix is ​​determined based on the vertical axis transformation matrix, the horizontal axis transformation matrix, and the plane intersection point.

3. The dual-depth camera extrinsic parameter calibration method according to claim 2, characterized in that, The step of determining the transformation matrix that transforms the planar intersection vector to be parallel to the vertical axis of the machine coordinate system, thereby obtaining the vertical axis transformation matrix, includes: The first rotation axis is obtained by performing a cross product between the plane intersection vector and the vertical unit vector of the machine coordinate system; The first rotation angle is obtained by multiplying the plane intersection vector and the vertical axis unit vector by a dot product. The vertical axis transformation matrix is ​​determined based on the first rotation axis and the first rotation angle.

4. The dual-depth camera extrinsic parameter calibration method according to claim 2, characterized in that, The step of determining the transformation matrix that transforms the plane intersection point to the horizontal axis of the machine coordinate system, and obtaining the horizontal axis transformation matrix, includes: The second rotation angle is determined based on the intersection of the planes and the unit vector of the horizontal axis of the machine coordinate system; The horizontal axis transformation matrix is ​​determined based on the second rotation angle and the vertical axis unit vector of the machine coordinate system.

5. The dual-depth camera extrinsic parameter calibration method according to claim 2, characterized in that, Determining the first transformation matrix based on the vertical axis transformation matrix, the horizontal axis transformation matrix, and the plane intersection point includes: The intersection points of the plane are transformed based on the horizontal axis transformation matrix to obtain the transformed intersection points; The target intersection point is obtained by translating the transformed intersection point based on a preset distance. The first transformation matrix is ​​determined based on the vertical axis transformation matrix, the horizontal axis transformation matrix, and the target intersection point.

6. A dual-depth camera extrinsic parameter calibration device, characterized in that, Applied to a robot, the robot being equipped with a first depth camera and a second depth camera, the device includes: The acquisition module is used to acquire first point cloud data obtained by the first depth camera from a preset scene, and to acquire second point cloud data obtained by the second depth camera from the preset scene. The first determining module is used to determine the transformation matrix for transforming the first point cloud data from the first coordinate system to the machine coordinate system, obtain the first transformation matrix, and calibrate the first transformation matrix to the first depth camera, wherein the first coordinate system is the coordinate system of the first depth camera, and the machine coordinate system is the coordinate system of the robot. The second determining module is used to determine the transformation matrix that transforms the second point cloud data from the second coordinate system to the machine coordinate system, obtain the second transformation matrix, and calibrate the second transformation matrix to the second depth camera, wherein the second coordinate system is the coordinate system of the second depth camera; The transformation module is used to transform the first point cloud data from the first coordinate system to the machine coordinate system based on the first transformation matrix to obtain first transformed point cloud data, and to transform the second point cloud data from the second coordinate system to the machine coordinate system based on the second transformation matrix to obtain second transformed point cloud data. The third determining module is used to determine the transformation matrix that aligns the first transformed point cloud data with the second transformed point cloud data, obtain the third transformation matrix, and calibrate the third transformation matrix to the second depth camera. The third determining module determines a transformation matrix to align the first transformed point cloud data with the second transformed point cloud data, resulting in a third transformation matrix, including: The first transformed point cloud data is determined as the target point cloud data, and the second transformed point cloud data is determined as the source point cloud data; The residual is minimized using the maximum likelihood estimation algorithm to obtain an intermediate transformation matrix. The third transformation matrix is ​​determined based on the intermediate transformation matrix and the preset installation angle. The residual is the distance between matching points in the source point cloud data and the target point cloud data. The intermediate transformation matrix is ​​a transformation matrix obtained by minimizing the residual using the maximum likelihood estimation algorithm.

7. A robot, characterized in that, The robot includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the dual depth camera extrinsic calibration method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the dual-depth camera extrinsic calibration method as described in any one of claims 1 to 5.

Images