Sensor calibration method, storage medium, electronic device, and program product

Abstract

The application provides a sensor calibration method, a storage medium, an electronic device and a program product, and relates to the technical field of sensor calibration. The sensor calibration method comprises the following steps: determining the coordinates of target feature points in a calibration board in a calibration board coordinate system; determining the pose of the calibration board coordinate system relative to a robot body; determining the coordinates of the target feature points in the robot body coordinate system based on the coordinates of the target feature points in the calibration board coordinate system and the pose of the calibration board coordinate system relative to the robot body; and determining the external parameter data of a first target sensor of the robot relative to the robot body based on the coordinates of the target feature points in the robot body coordinate system, wherein the first target sensor is a sensor with a fixed relative position relationship with the robot body. The application avoids dependence on professional calibration sites and high-precision equipment, simplifies the calibration process, reduces the calibration cost, and ensures the calibration accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sensor calibration, and in particular to a sensor calibration method, a storage medium, an electronic device and a program product. BACKGROUND

[0002] In the field of robots, sensor calibration aims to ensure that the robot can still maintain its accuracy after a long period of use or is affected by external factors, such as hardware replacement, environmental changes, equipment aging, etc.

[0003] However, existing sensor calibration techniques often require professional calibration sites, high-precision calibration equipment, and complex calibration procedures, such as requiring the robot to perform a figure-eight motion, resulting in high calibration costs. Therefore, there is currently a need for a simple and low-cost sensor calibration method that can ensure the accuracy of the calibration. SUMMARY

[0004] In view of this, the embodiments of the present application provide a sensor calibration method, a storage medium, an electronic device and a program product.

[0005] In a first aspect, an embodiment of the present application provides a sensor calibration method applied to a robot deployed with a sensor, the method comprising: determining coordinates of a target feature point in a calibration board in a calibration board coordinate system;

[0006] determining a pose of the calibration board coordinate system relative to a robot body; determining coordinates of the target feature point in a robot body coordinate system based on the coordinates of the target feature point in the calibration board coordinate system and the pose of the calibration board coordinate system relative to the robot body; and determining extrinsic data of a first target sensor of the robot relative to the robot body based on the coordinates of the target feature point in the robot body coordinate system, the first target sensor being a sensor with a fixed relative positional relationship with the robot body.

[0007] In combination with the first aspect, in some implementations of the first aspect, the first target sensor includes a laser radar, and the target feature point includes a laser radar feature point; and determining the extrinsic data of the first target sensor of the robot relative to the robot body based on the coordinates of the target feature point in the robot body coordinate system comprises: obtaining scanning data of the laser radar on the calibration board; determining coordinates of the laser radar feature point in a laser radar coordinate system based on the scanning data; and determining the extrinsic data of the laser radar relative to the robot body based on the coordinates of the laser radar feature point in the robot body coordinate system and the coordinates of the laser radar feature point in the laser radar coordinate system.

[0008] In some implementations of the first aspect, the first target sensor includes a target camera, and the target feature points include camera feature points. The method further includes: determining, based on the coordinates of the camera feature points in the robot body coordinate system, the extrinsic parameter data of the target camera relative to the robot body.

[0009] In some implementations of the first aspect, the robot is deployed with a second target sensor having a variable relative position relationship with the robot body. The second target sensor includes a head camera, and the target feature points include camera feature points. The method further includes: determining, based on the coordinates of the camera feature points in the robot body coordinate system, the extrinsic parameter data of the head camera relative to the head end joint.

[0010] In some implementations of the first aspect, determining, based on the coordinate data of the camera feature points in the robot body coordinate system, the extrinsic parameter data of the head camera relative to the head end joint includes:

[0011] determining the coordinates of the camera feature points in the head camera coordinate system; determining, based on the coordinates of the camera feature points in the robot body coordinate system and the coordinates of the camera feature points in the head camera coordinate system, the pose of the head camera relative to the robot body at the current position; determining the pose of the head end joint relative to the robot body; and determining, based on the pose of the head camera relative to the robot body at the current position and the pose of the head end joint relative to the robot body, the extrinsic parameter data of the head camera relative to the head end joint.

[0012] In some implementations of the first aspect, the robot is deployed with a second target sensor having a variable relative position relationship with the robot body. The second target sensor includes a hand camera, and the target feature points include camera feature points. The method further includes: obtaining target images of a calibration board captured by the hand camera at a plurality of poses of the robot arm; identifying, based on the target images, the coordinates of the camera feature points in the hand camera coordinate system; obtaining coordinates of a target position in a hand end joint coordinate system when the robot arm moves to the target position of the calibration board; and determining, based on the coordinates of the camera feature points in the hand camera coordinate system and the coordinates of the target position in the hand end joint coordinate system, the extrinsic parameter data of the hand camera relative to the hand end joint. Preferably, the hand camera includes a left hand camera and / or a right hand camera.

[0013] In some implementations of the first aspect, the determining the pose of the calibration board coordinate system relative to the robot body includes: determining a pose of the calibration board coordinate system relative to the hand camera; determining a pose of the hand end joint to the robot body; and determining the pose of the calibration board coordinate system relative to the robot body based on the pose of the calibration board coordinate system relative to the hand camera, the pose of the hand end joint to the robot body, and extrinsic parameter data of the hand camera relative to the hand end joint.

[0014] In some implementations of the first aspect, the target feature points include laser radar feature points and / or camera feature points, the laser radar feature points are determined based on the hollowed-out circles in the calibration board, and the camera feature points are determined based on the target color circles in the calibration board.

[0015] In the second aspect, an embodiment of the present application provides a sensor calibration device applied to a robot in which a sensor is deployed, the device including: a first determination module configured to determine coordinates of target feature points in a calibration board in a calibration board coordinate system; a second determination module configured to determine a pose of the calibration board coordinate system relative to a robot body; a third determination module configured to determine coordinates of the target feature points in a robot body coordinate system based on the coordinates of the target feature points in the calibration board coordinate system and the pose of the calibration board coordinate system relative to the robot body; and a fourth determination module configured to determine extrinsic parameter data of a first target sensor of the robot relative to the robot body based on the coordinates of the target feature points in the robot body coordinate system, the first target sensor being a sensor with a fixed relative positional relationship with the robot body.

[0016] In the third aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, the computer program being configured to execute the laser point cloud processing method of the first aspect.

[0017] In the fourth aspect, an embodiment of the present application provides an electronic device including: a processor; a memory configured to store processor-executable instructions; and the processor configured to execute the laser point cloud processing method of the first aspect.

[0018] In the fifth aspect, an embodiment of the present application provides a computer program product including instructions configured to cause an electronic device to implement the laser point cloud processing method of the first aspect when the instructions are executed on the electronic device.

[0019] In the present application, by determining the coordinates of the target feature points in the calibration plate coordinate system, the reference data is provided for the subsequent calibration process. Moreover, this step only needs to record the positions of the target feature points in the calibration plate when the calibration plate is made, and thus the preparation work of sensor calibration is simplified. Then, by determining the pose of the calibration plate coordinate system relative to the robot body, the sensor calibration process is associated with the spatial position of the robot body. Then, by mathematical calculation, the coordinates of the target feature points are converted from the calibration plate coordinate system to the robot body coordinate system, and this step does not need additional complex operation, thereby reducing the complexity and cost in the calibration process. Finally, based on the coordinates of the target feature points in the robot body coordinate system, the external parameter data of the first target sensor relative to the robot body is determined, and the calibration of the first target sensor is completed.

[0020] In summary, the present application provides a simple and low-cost sensor calibration method, which can be realized through mathematical calculation and coordinate conversion, avoids the dependence on professional calibration sites and high-precision equipment, simplifies the calibration process, reduces the calibration cost, and does not lose the accuracy of calibration. BRIEF DESCRIPTION OF DRAWINGS

[0021] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description of embodiments of the present application, taken in conjunction with the accompanying drawings. The drawings provided in the present application are used to provide further understanding of the embodiments of the present application, and constitute a part of the specification, and are used to explain the present application together with the embodiments of the present application, and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0022] Figure 1 Fig. 1 shows a flowchart of a sensor calibration method provided by an embodiment of the present application.

[0023] Figure 2 Fig. 2 shows a flowchart of determining the external parameter data of the first target sensor of the robot relative to the robot body provided by an embodiment of the present application.

[0024] Figure 3 Fig. 3 shows a flowchart of determining the external parameter data of the first target sensor of the robot relative to the robot body provided by another embodiment of the present application.

[0025] Figure 4 Fig. 4 shows a flowchart of determining the external parameter data of the head camera relative to the head end joint based on the coordinate data of the camera feature points in the robot body coordinate system provided by an embodiment of the present application.

[0026] Figure 5 Fig. 5 shows a flowchart of a sensor calibration method provided by another embodiment of the present application.

[0027] Figure 6 Fig. 1 shows a flowchart of a method for determining the pose of a calibration plate coordinate system relative to a robot body according to an embodiment of the present application.

[0028] Figure 7 Fig. 2 shows a schematic diagram of a calibration plate according to an embodiment of the present application.

[0029] Figure 8 Fig. 3 shows a schematic diagram of a sensor calibration device according to an embodiment of the present application.

[0030] Figure 9 Fig. 4 shows a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0032] Figure 1 Fig. 5 shows a flowchart of a sensor calibration method according to an embodiment of the present application. The method is exemplarily applied to a robot deployed with a sensor. As shown in Fig. 5, the method comprises the following steps. Figure 1

[0033] Step S110: determining the coordinates of target feature points in the calibration plate in the calibration plate coordinate system.

[0034] The target feature points in the calibration plate refer to specific points on the surface of the calibration plate, which are pre-set for the calibration process. The calibration plate coordinate system refers to a coordinate system established based on the physical position and direction of the calibration plate. Exemplarily, the origin of the calibration plate coordinate system is located at a specific point of the calibration plate, such as the center or a corner of the calibration plate, and the coordinate axes of the calibration plate coordinate system are parallel to the physical edges of the calibration plate. Based on the design parameters of the target feature points in the calibration plate and the construction rules of the calibration plate coordinate system, the coordinate values of each target feature point on the calibration plate in the calibration plate coordinate system can be determined.

[0035] Step S120: determining the pose of the calibration plate coordinate system relative to the robot body.

[0036] ​The pose of the calibration board coordinate system relative to the robot body includes a translation vector and a rotation matrix. The translation vector determines the position of the origin of the calibration board coordinate system in the robot body coordinate system, and the rotation matrix describes the rotation relationship of the calibration board coordinate system relative to the robot body coordinate system. Through the pose, the accurate spatial relationship of the calibration board relative to the robot body can be known.

[0037] In step S130, based on the coordinates of the target feature points in the calibration board coordinate system and the pose of the calibration board coordinate system relative to the robot body, the coordinates of the target feature points in the robot body coordinate system are determined.

[0038] Specifically, according to the pose of the calibration board coordinate system relative to the robot body, the coordinates of the target feature points are converted from the calibration board coordinate system to the robot body coordinate system through coordinate transformation. For example, the coordinates of the target feature points in the robot body coordinate system are wherein, represents the pose of the calibration board coordinate system relative to the robot body, P cb represents the coordinates of the target feature points in the calibration board coordinate system.

[0039] In step S140, based on the coordinates of the target feature points in the robot body coordinate system, the extrinsic data of the first target sensor of the robot relative to the robot body is determined.

[0040] The first target sensor is a sensor with a fixed relative position relationship with the robot body. That is, regardless of how the robot body moves or changes the pose, the position and direction of the first target sensor relative to the robot body are unchanged. The extrinsic data of the first target sensor relative to the robot body refers to the position and direction of the first target sensor in the robot body coordinate system, including the translation vector and the rotation matrix of the first target sensor relative to the robot body coordinate system.

[0041] Specifically, the coordinates of the target feature points in the first target sensor coordinate system are obtained, and based on the coordinates of the target feature points in the first target sensor coordinate system and the coordinates of the target feature points in the robot body coordinate system, the extrinsic data of the first target sensor relative to the robot body is determined.

[0042] Alternatively, by comparing the coordinates of the same target feature points in the first target sensor coordinate system and the robot body coordinate system, a conversion equation can be established, and then the rotation matrix and the translation vector are obtained by solving the conversion equation.

[0043] For example, it is assumed that the coordinates of the target feature points in the first target sensor coordinate system are (x b , y b , z b), the coordinates of the target feature points in the robot body coordinate system are (x r ,y r ,z r ), then the conversion equation of the two is: wherein R represents a rotation matrix, and t represents a translation vector.

[0044] Optionally, the optimal rotation matrix R and translation vector t can be found by a mathematical optimization method, such as the least square method. That is, the extrinsic parameter data of the first target sensor relative to the robot body is obtained.

[0045] In this embodiment, by determining the coordinates of the target feature points in the calibration board coordinate system, the reference data is provided for the subsequent calibration process. Moreover, this step only needs to record the positions of the target feature points in the calibration board when the calibration board is made, so that the preparation work of sensor calibration is simplified. Then, by determining the pose of the calibration board coordinate system relative to the robot body, the sensor calibration process is associated with the spatial position of the robot body. Then, by mathematical calculation, the coordinates of the target feature points are converted from the calibration board coordinate system to the robot body coordinate system, which does not need additional complex operation, and reduces the complexity and cost in the calibration process. Finally, based on the coordinates of the target feature points in the robot body coordinate system, the extrinsic parameter data of the first target sensor relative to the robot body is determined, and the calibration of the first target sensor is completed.

[0046] In summary, the present application provides a simple and low-cost sensor calibration method, which can be realized by mathematical calculation and coordinate conversion, avoids the dependence on professional calibration sites and high-precision equipment, simplifies the calibration process, reduces the calibration cost, and does not lose the accuracy of calibration.

[0047] Figure 2 Fig. 1 shows a flowchart of determining the extrinsic parameter data of the first target sensor of the robot relative to the robot body according to an embodiment of the present application. Based on the embodiment shown in Figure 1 Fig. 1, the embodiment shown in Figure 2 Fig. 2 is extended, and the differences between Figure 2 Fig. 2 and Figure 1 Fig. 1 will be mainly described below, and the same parts will not be repeated.

[0048] Firstly, in this embodiment, the first target sensor includes a laser radar, and the laser radar is exemplarily deployed in the ground of the robot. In addition, the target feature points include laser radar feature points, which are points with specific properties detected by the laser radar when scanning the calibration board, for example, the laser radar feature points are points with abnormal reflection intensity in the calibration board.

[0049] As shown in Figure 2 The first target sensor of the robot is determined relative to the robot body based on the coordinates of the target feature points in the robot body coordinate system, including the following steps.

[0050] In step S210, the scanning data of the calibration board by the laser radar is obtained.

[0051] Before the laser radar collects the scanning data of the calibration board, it is necessary to ensure that the calibration board is located within the scanning range of the laser radar, and during the scanning process, the robot and the calibration board remain relatively stationary, and the distance and angle between the calibration board and the robot are appropriate.

[0052] In addition, in order to improve the accuracy of the calibration, the laser radar needs to scan the calibration board from different positions and angles multiple times to obtain more rich information of the laser radar feature points. That is, the scanning data in the present embodiment is multiple frames of data obtained by the laser radar scanning the calibration board from different positions and angles.

[0053] In step S220, the coordinates of the laser radar feature points in the laser radar coordinate system are determined based on the scanning data.

[0054] Specifically, the scanning data contains the distance and angle information of each laser radar feature point relative to the emitter of the laser radar. Using this information, the position of each laser radar feature point in the laser radar coordinate system can be calculated.

[0055] In step S230, the extrinsic parameter data of the laser radar relative to the robot body is determined based on the coordinates of the laser radar feature points in the robot body coordinate system and the coordinates of the laser radar feature points in the laser radar coordinate system.

[0056] In one example, after obtaining the respective coordinates of the laser radar feature points in the robot body coordinate system and the laser radar coordinate system, a conversion equation is established between the two, and by solving the conversion equation, a rotation matrix and a translation vector are obtained, i.e. the extrinsic parameter data of the laser radar relative to the robot body is obtained.

[0057] In another example, based on the coordinates of each laser radar feature point in the robot body coordinate system and the laser radar coordinate system, a 3D-3D ICP algorithm is used to determine the extrinsic parameter data of the laser radar relative to the robot body.

[0058] In this embodiment, by acquiring the scanning data of the laser radar on the calibration board, the point cloud information of the calibration board in the laser radar coordinate system is collected. Then, by analyzing these scanning data, the coordinates of the laser radar feature points in the laser radar coordinate system are determined, providing accurate reference for subsequent coordinate conversion. Finally, by using the coordinates of the laser radar feature points in the robot body coordinate system and the laser radar coordinate system, the extrinsic parameter data of the laser radar relative to the robot body is determined through mathematical transformation or ICP algorithm.

[0059] In summary, the embodiment provides a high-efficiency, accurate and low-cost laser radar calibration method, which enables the robot to maintain high accuracy and high performance of the laser radar in various working environments, whether after long-term use, hardware replacement, environmental change or equipment aging.

[0060] Figure 3 The flowchart shown is a process schematic diagram for determining the extrinsic parameter data of the first target sensor of the robot relative to the robot body provided by another embodiment of the application. In this embodiment, the first target sensor includes a target camera, and the target feature points include camera feature points. Figure 1 Based on the embodiment shown, the embodiment shown extends Figure 3 Based on the embodiment shown, the embodiment shown extends Figure 3 The embodiment shown is different from Figure 1 The same parts of the embodiment shown will not be described again.

[0061] First, in this embodiment, the first target sensor includes a target camera, which is exemplarily installed in the chassis of the robot. The target feature points include camera feature points, which are feature points with obvious and distinguishable features in the image collected by the target camera. For example, the camera feature points are points with unique textures.

[0062] As Figure 3 Based on the coordinates of the target feature points in the robot body coordinate system, the extrinsic parameter data of the first target sensor of the robot relative to the robot body is determined, including the following steps.

[0063] Step S310, determining the coordinates of the camera feature points in the target camera coordinate system.

[0064] Specifically, the image of the calibration board shot by the target camera is acquired; the pixel coordinates of the camera feature points in the calibration board in the image are determined; and the pixel coordinates are converted into the coordinates in the target camera coordinate system based on the intrinsic parameter data of the target camera.

[0065] It should be noted that when shooting the calibration board, the target camera needs to ensure that the calibration board is in the field of view of the target camera and is not blocked. During shooting, the target camera should be kept stable to avoid affecting the accurate positioning of the camera feature points.

[0066] In step S320, based on the coordinates of the camera feature points in the robot body coordinate system and the coordinates of the camera feature points in the target camera coordinate system, the extrinsic parameter data of the target camera relative to the robot body is determined.

[0067] In one example, after obtaining the respective coordinates of the camera feature points in the robot body coordinate system and the target camera coordinate system, a conversion equation is established, and the rotation matrix and the translation vector are obtained by solving the conversion equation, i.e., the extrinsic parameter data of the target camera relative to the robot body is obtained.

[0068] In another example, based on the coordinates of the camera feature points in the robot body coordinate system and the coordinates of the camera feature points in the target camera coordinate system, the extrinsic parameter data of the target camera relative to the robot body is calculated by using the 3D-2D PnP algorithm.

[0069] In the present embodiment, by obtaining the image of the calibration board captured by the target camera and determining the coordinates of the camera feature points in the target camera coordinate system, accurate data basis is provided for subsequent calibration. Then, by using the coordinates of the camera feature points in the target camera coordinate system and the known coordinates of the camera feature points in the robot body coordinate system, the extrinsic parameter data of the target camera relative to the robot body is obtained by establishing a conversion equation or a 3D-2D PnP algorithm. In summary, the present embodiment provides a target table calibration method that is efficient, accurate and low in cost, so that the robot can accurately combine the visual data captured by the camera with the motion control data of the robot, thereby improving the operation precision and environmental perception ability of the robot.

[0070] The foregoing embodiments mainly describe the calibration method of the laser radar and the target camera whose relative position relationship with the robot body is fixed. Next, the calibration method of the second target sensor whose relative position relationship with the robot body is variable is described in detail.

[0071] In some embodiments, the second target sensor includes a head camera, and the method further includes determining, based on the coordinates of the camera feature points in the robot body coordinate system, the extrinsic parameter data of the head camera relative to the head end joint.

[0072] Since the relative position relationship between the head camera and the robot body is variable, it does not have fixed external parameter data like a fixedly installed sensor (such as a laser radar or a target camera in the chassis). However, in the robot design, the relative position relationship between the head camera and the head end joint is fixed, and the head end joint is endowed with multiple degrees of freedom to perform actions similar to the human head, such as nodding up and down, turning left and right, and tilting, etc. Therefore, the embodiment is to determine the external parameter data of the head camera relative to the head end joint, so as to associate the observation data of the head camera with the motion and posture of the head end joint, thereby realizing more accurate spatial positioning and environmental perception.

[0073] Figure 4 Fig. 1 shows a flowchart of a process for determining the external parameter data of the head camera relative to the head end joint based on the coordinate data of the camera feature points in the robot body coordinate system according to an embodiment of the present application. As shown in Fig. 1, the process comprises the following steps. Figure 4

[0074] In step S410, the coordinates of the camera feature points in the head camera coordinate system are determined.

[0075] Specifically, an image of the calibration board captured by the head camera is obtained; the pixel coordinates of the camera feature points in the image are determined; and the pixel coordinates are converted into coordinates in the head camera coordinate system based on the internal parameter data of the head camera.

[0076] It should also be noted that when the head camera captures the calibration board, it needs to ensure that the calibration board is not blocked in the field of view of the head camera. During the shooting process, the head camera should be kept stable to avoid affecting the accurate positioning of the camera feature points.

[0077] In step S420, the pose of the head camera relative to the robot body at the current position is determined based on the coordinates of the camera feature points in the robot body coordinate system and the coordinates of the camera feature points in the head camera coordinate system.

[0078] For example, a conversion equation of the coordinates of the camera feature points in the robot body coordinate system and the head camera coordinate system is established, and by solving the conversion equation, a rotation matrix and a translation vector are obtained, i.e. the external parameter data of the head camera relative to the robot body at the current position is obtained.

[0079] In addition, as can be known from the foregoing analysis, the pose of the head camera relative to the robot body is not fixed, but dynamically changes with the motion of the head end joint. Therefore, at different time points or different task stages, the head camera can point in different directions or be at different positions. Therefore, what is obtained through step S420 is the pose of the head camera relative to the robot body at a certain position.

[0080] ​Step S430, determine the pose of the head end joint relative to the robot body.

[0081] Alternatively, the pose of the head end joint relative to the robot body can be determined through robot kinematics. Specifically, first, the angles of the joints of the robot are measured in real time, exemplarily provided by encoders or other sensors on the joints. Then, according to the forward kinematics model of the robot, the angles of the joints are used to calculate the position and orientation of the head end joint in the robot body coordinate system.

[0082] The forward kinematics model is a mathematical equation based on the geometric parameters of the joints and links of the robot, which defines how to calculate the position and pose of the end effector from the joint angles. By accumulating the local transformations (including rotation and translation) of each joint, the pose of the head end joint to the robot body can be obtained.

[0083] Step S440, based on the pose of the head camera relative to the robot body at the current position and the pose of the head end joint relative to the robot body, determine the extrinsic parameter data of the head camera relative to the head end joint.

[0084] The extrinsic parameter data of the head camera relative to the head end joint is calculated through the pose of the head camera relative to the robot body at the current position and the pose of the head end joint relative to the robot body, which is actually the inverse process of coordinate transformation. Specifically, the inverse of the pose transformation matrix of the head camera relative to the robot body is multiplied by the pose transformation matrix of the head end joint relative to the robot body, to obtain the transformation matrix of the head camera relative to the head end joint, i.e.

[0085] In this embodiment, first, the coordinates of the camera feature points in the head camera coordinate system are determined, and then combined with the coordinates of the target feature points in the robot body coordinate system, the pose of the head camera relative to the robot body at the current position is calculated. Subsequently, by determining the pose of the head end joint relative to the robot body, the extrinsic parameter data of the head camera relative to the head end joint is further calculated, so as to obtain the accurate position and orientation of the head camera relative to the head end joint. Through this scheme, it is ensured that the robot can accurately predict the field of view and image data of the head camera when performing head actions, thereby improving the action accuracy and reliability of the head end joint as the execution end. Moreover, the robot can adjust the position and pose of the head end joint in real time according to the visual information provided by the head camera, to adapt to environmental changes or task requirements.

[0086] Figure 5Fig. 1 shows a flowchart of a sensor calibration method according to an embodiment of the present application. Figure 5 In the embodiment, the second target sensor further comprises a hand camera, and the method further comprises the following steps.

[0087] In step S510, target images of the calibration board captured by the hand camera at multiple poses of the robot arm are obtained.

[0088] In an example, the calibration board is placed at different positions or distances within the robot workspace, ensuring that the hand camera can capture images of the calibration board at multiple preset poses (different angles and positions) of the robot arm, and then determine the target images.

[0089] In step S520, coordinates of the camera feature points in the hand camera coordinate system are identified based on the target images.

[0090] Specifically, first, the pixel coordinates of the camera feature points in the target images are determined; and then, based on the intrinsic data of the hand camera, the pixel coordinates are converted into coordinates in the hand camera coordinate system.

[0091] In step S530, coordinates of the target position in the hand end joint coordinate system are obtained when the robot arm moves to the target position of the calibration board.

[0092] Using the kinematic model of the robot, the pose of the hand end joint actuator in the robot body coordinate system is calculated according to the joint angles of the robot arm. Then, the coordinates of the target position of the calibration board are measured or calculated in the robot body coordinate system. Finally, the coordinates of the target position are converted from the robot body coordinate system to the hand end joint coordinate system.

[0093] In step S540, the extrinsic data of the hand camera relative to the hand end joint is determined based on the coordinates of the camera feature points in the hand camera coordinate system and the coordinates of the target position in the hand end joint coordinate system.

[0094] Based on the positional relationship between the camera feature points and the target position in the calibration board coordinate system (in some embodiments, the target position is the position of the camera feature points), and the coordinates of the target position in the hand end joint coordinate system, the coordinates of the camera feature points in the hand end joint coordinate system can be calculated; and based on the coordinates of the camera feature points in the hand camera coordinate system and the coordinates of the camera feature points in the hand end joint coordinate system, the extrinsic data of the hand camera relative to the hand end joint is determined.

[0095] For example, a coordinate transformation equation for camera feature points in the hand camera coordinate system and the hand end joint coordinate system is established. By solving this transformation equation, the rotation matrix and translation vector are obtained, which are the extrinsic parameters of the hand camera relative to the hand end joint.

[0096] In this embodiment, by acquiring target images of the calibration plate taken by the hand camera in multiple different poses, it can be ensured that the calibration process does not rely solely on a single viewpoint or a specific arm position, making the final determined extrinsic parameter data of the hand camera relative to the end joint of the hand more comprehensive and reliable. Furthermore, it can capture the viewpoint changes of the hand camera at different directions and distances, which helps correct the internal distortion of the hand camera and reduce calibration inaccuracies caused by random errors.

[0097] Figure 6 The diagram shown is a flowchart illustrating the process of determining the pose of the calibration plate coordinate system relative to the robot body according to an embodiment of this application. Figure 5 Extending from the illustrated embodiment Figure 6 The illustrated embodiment will be described in detail below. Figure 6 The illustrated embodiments and Figure 5 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0098] like Figure 6 As shown, in this embodiment, determining the pose of the calibration plate coordinate system relative to the robot body includes the following steps.

[0099] Step S610: Determine the pose of the calibration plate coordinate system relative to the hand camera.

[0100] For example, a handheld camera is aimed at a calibration board and an image is captured. Then, computer vision techniques, such as feature extraction algorithms, are used to identify and precisely locate target feature points on the calibration board. After the pixel coordinates of these target feature points in the image are determined, the pixel coordinates are converted into coordinates in the handheld camera coordinate system using the intrinsic parameters of the handheld camera (including focal length, principal point, distortion parameters, etc.). Subsequently, by comparing the coordinates of the target feature points in the handheld camera coordinate system with the coordinates of the target feature points in the calibration board coordinate system, the rotation and translation parameters of the calibration board coordinate system relative to the handheld camera, i.e., the pose of the calibration board coordinate system relative to the handheld camera, can be calculated.

[0101] For example, this process can also use algorithms such as iterative nearest point or other registration techniques to optimize the pose estimation of the calibration board coordinate system relative to the hand camera, ensuring that the alignment between the calibration board coordinate system and the hand camera coordinate system is as accurate as possible.

[0102] Step S620: Determine the pose of the hand end joint to the robot body.

[0103] As mentioned above, the pose of the hand end joint to the robot body can be determined by robot kinematics, which will not be described herein.

[0104] In step S630, the pose of the calibration board coordinate system relative to the robot body is determined based on the pose of the calibration board coordinate system relative to the hand camera, the pose of the hand end joint to the robot body, and the extrinsic data of the hand camera relative to the hand end joint.

[0105] Specifically, based on the pose of the calibration board coordinate system relative to the hand camera and the extrinsic data of the hand camera relative to the hand end joint, the extrinsic data of the calibration board coordinate system relative to the hand end joint can be determined; based on the extrinsic data of the calibration board coordinate system relative to the hand end joint and the pose of the hand end joint to the robot body, the pose of the calibration board coordinate system relative to the robot body can be determined.

[0106] That is, the pose of the calibration board coordinate system relative to the hand camera is multiplied by the extrinsic data of the hand camera relative to the hand end joint to obtain the extrinsic data of the calibration board coordinate system relative to the hand end joint. Then, the extrinsic data of the calibration board coordinate system relative to the hand end joint is multiplied by the pose of the hand end joint to the robot body to obtain the pose of the calibration board coordinate system relative to the robot body.

[0107] In the embodiment, the pose of the calibration board coordinate system relative to the robot body is determined by combining the pose of the calibration board coordinate system relative to the hand camera, the pose of the hand end joint to the robot body, and the extrinsic data of the hand camera relative to the hand end joint through mathematical coordinate transformation. This conversion describes the final pose of the calibration board coordinate system relative to the robot body, thereby completing the determination of the pose of the calibration board coordinate system relative to the robot body, so that the robot can accurately understand and utilize information from the sensors deployed by itself.

[0108] In some embodiments of the present application, the laser radar feature points are determined based on the hollow circles in the calibration board, and the camera feature points are determined based on the target color circles in the calibration board.

[0109] Figure 7 As shown in the schematic diagram of the calibration board provided by an embodiment of the present application. Figure 7 As shown, the calibration board is a planar calibration board, the target color circles are black circles, the hollow circles are located at the four corners of the calibration board, and the horizontal and vertical spacings between the black circles are fixed.

[0110] Specifically, this calibration board combines the advantages of lidar and camera in different environments, improving the accuracy and robustness of the calibration process. For the camera, the center of the target color circle is used as the camera feature point, which can better handle nonlinear distortion, reduce sensitivity to light, and stably detect in low-contrast or high-reflectivity environments compared to traditional chessboard corner extraction, thereby improving the accuracy of camera feature point extraction. For the lidar, the geometric characteristics of the hollow circle are fitted to extract the lidar feature points, which are not affected by the reflection intensity or surface material. Even if the circumference of the hollow circle is blocked, the center of the hollow circle (i.e., the lidar feature point) position can be accurately calculated, achieving sub-point cloud level accuracy and strong anti-interference ability. Combining these advantages, the scheme of the present application can provide stable and reliable calibration results in a variable working environment, providing a solid foundation for precise operation and environmental perception of robots.

[0111] As can be seen from the above embodiments, the sensor calibration method provided by the present embodiment only needs to use one calibration board and requires the robot to be stationary in front of the calibration board to complete the calibration of all sensors. This method is simple and easy to operate, can solve the problems of complex operation and high cost in the background art, and at the same time ensures the accuracy of calibration.

[0112] The sensors that can be calibrated by the present application are divided into two categories: static sensors and dynamic sensors. Simply put, static sensors include target cameras and lidars, etc., which are fixedly installed at specific positions of the robot, so the positional relationship relative to the robot body is constant. Dynamic sensors, such as left-hand cameras, right-hand cameras, and head cameras, are installed on movable joints of the robot, and their relative positions to the connected joints remain unchanged, but their positional relationship relative to the robot body changes with the movement of the connected joints. Through the sensor calibration method in the present application, the operation process can be simplified, the cost can be reduced, and the efficiency and convenience of robot sensor calibration can be improved while ensuring the accuracy of calibration.

[0113] The sensor calibration method embodiments of the present application are described in detail above in conjunction with Figure 7 The sensor calibration device embodiments of the present application are described in detail below in conjunction with Figure 8 It should be understood that the description of the sensor calibration method embodiments corresponds to the description of the sensor calibration device embodiments, and therefore, the parts not described in detail can be referred to the previous method embodiments.

[0114] Figure 8 Fig. 8 shows a structure schematic diagram of a sensor calibration device provided by an embodiment of the present application. As shown in Fig. 8, the sensor calibration device 80 provided by the present embodiment comprises: Figure 9

[0115] ​The first determining module 810 is configured to determine coordinates of a target feature point in the calibration board in a calibration board coordinate system.

[0116] The second determining module 820 is configured to determine a pose of the calibration board coordinate system relative to the robot body.

[0117] The third determining module 830 is configured to determine coordinates of the target feature point in a robot body coordinate system based on the coordinates of the target feature point in the calibration board coordinate system and the pose of the calibration board coordinate system relative to the robot body.

[0118] The fourth determining module 840 is configured to determine extrinsic parameter data of a first target sensor of the robot relative to the robot body based on the coordinates of the target feature point in the robot body coordinate system, the first target sensor being a sensor with a fixed relative positional relationship with the robot body.

[0119] In an embodiment of the present application, the first target sensor includes a laser radar, and the target feature point includes a laser radar feature point. The fourth determining module 840 is further configured to obtain scanning data of the laser radar on the calibration board, determine coordinates of the laser radar feature point in a laser radar coordinate system based on the scanning data, and determine the extrinsic parameter data of the laser radar relative to the robot body based on the coordinates of the laser radar feature point in the robot body coordinate system and the coordinates of the laser radar feature point in the laser radar coordinate system.

[0120] In an embodiment of the present application, the first target sensor includes a target camera, and the target feature point includes a camera feature point. The fourth determining module 840 is further configured to determine coordinates of the camera feature point in a target camera coordinate system, and determine the extrinsic parameter data of the target camera relative to the robot body based on the coordinates of the camera feature point in the robot body coordinate system and the coordinates of the camera feature point in the target camera coordinate system.

[0121] In an embodiment of the present application, the sensors of the robot further include a second target sensor with a variable relative positional relationship with the robot body, the second target sensor including a head camera, and the target feature point including a camera feature point. The fourth determining module 840 is further configured to determine the extrinsic parameter data of the head camera relative to a head end joint based on the coordinates of the camera feature point in the robot body coordinate system.

[0122] In an embodiment of the present application, the fourth determination module 840 is further configured to: determine the coordinates of the camera feature point in the head camera coordinate system; determine the pose of the head camera relative to the robot body in the current position based on the coordinates of the camera feature point in the robot body coordinate system and the coordinates of the camera feature point in the head camera coordinate system; determine the pose of the head end joint relative to the robot body; and determine the extrinsic parameter data of the head camera relative to the head end joint based on the pose of the head camera relative to the robot body in the current position and the pose of the head end joint relative to the robot body.

[0123] In an embodiment of the present application, the sensors of the robot deployment further include a second target sensor with a variable relative position relationship with the robot body, the second target sensor includes a hand camera, and the target feature point includes a camera feature point. The fourth determination module 840 is further configured to: obtain target images of the calibration board captured by the hand camera in multiple poses of the robot arm; identify the coordinates of the camera feature point in the hand camera coordinate system based on the target images; obtain the coordinates of the target position in the hand end joint coordinate system when the robot arm moves to the target position of the calibration board; and determine the extrinsic parameter data of the hand camera relative to the hand end joint based on the coordinates of the camera feature point in the hand camera coordinate system and the coordinates of the target position in the hand end joint coordinate system. Preferably, the hand camera includes a left hand camera and / or a right hand camera.

[0124] In an embodiment of the present application, the second determination module 820 is further configured to: determine the pose of the calibration board coordinate system relative to the hand camera; determine the pose of the hand end joint to the robot body; and determine the pose of the calibration board coordinate system relative to the robot body based on the pose of the calibration board coordinate system relative to the hand camera, the pose of the hand end joint to the robot body, and the extrinsic parameter data of the hand camera relative to the hand end joint.

[0125] In an embodiment of the present application, the target feature point includes a laser radar feature point and / or a camera feature point, the laser radar feature point is determined based on a hollow circle in the calibration board, and the camera feature point is determined based on a target color circle in the calibration board.

[0126] Hereinafter, an electronic device according to an embodiment of the present application will be described with reference to the accompanying drawings. Figure 9 FIG. 1 illustrates an electronic device according to an embodiment of the present application. Figure 9 FIG. 2 illustrates an electronic device according to an embodiment of the present application.

[0127] As shown in FIG. 3, the electronic device 90 includes one or more processors 901 and a memory 902. Figure 9

[0128] ​The processor 901 can be a central processing unit (CPU) or other form of processing unit that has data processing and / or instruction executing capabilities, and can control other components in the electronic device 90 to perform desired functions.

[0129] The memory 902 can include one or more computer program products that can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory, for example, can include random access memory (RAM), cache memory, and / or the like. The non-volatile memory, for example, can include read-only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer-readable storage media, which the processor 901 can execute to implement the sensor calibration method of various embodiments of the present application described above and / or other desired functions. Various contents such as the coordinates of the target feature points in the calibration board coordinate system, the pose of the calibration board coordinate system relative to the robot body, the coordinates of the target feature points in the robot body coordinate system, the extrinsic parameter data of the first target sensor relative to the robot body, and the like can also be stored in the computer-readable storage media.

[0130] In one example, the electronic device 90 can further include an input device 903 and an output device 904, which are interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0131] The input device 903 can include, for example, a keyboard, a mouse, and / or the like.

[0132] The output device 904 can output various information to the outside, including the coordinates of the target feature points in the calibration board coordinate system, the pose of the calibration board coordinate system relative to the robot body, the coordinates of the target feature points in the robot body coordinate system, the extrinsic parameter data of the first target sensor relative to the robot body, and the like. The output device 904 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and / or the like.

[0133] Of course, in order to simplify, Figure 9 Only some of the components in the electronic device 90 related to the present application are shown in FIG. 9, and components such as buses, input / output interfaces, and the like are omitted. In addition, the electronic device 90 can include any other appropriate components according to specific application cases.

[0134] In addition to the methods and devices described above, embodiments of the present application can also be a computer program product that includes computer program instructions that, when run by a processor, cause the processor to perform the steps of the sensor calibration methods described above according to various embodiments of the present application.

[0135] The computer program instructions can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server.

[0136] In addition, embodiments of the present application can also be a computer readable storage medium having stored thereon computer program instructions that, when run by a processor, cause the processor to perform the steps of the sensor calibration methods described above according to various embodiments of the present application.

[0137] The computer readable storage medium can be any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0138] The above describes the basic principles of the present application in conjunction with specific embodiments, but it should be noted that the advantages, benefits, effects, etc. mentioned in the present application are only examples and are not limiting, and these advantages, benefits, effects, etc. cannot be considered as the must-have of each embodiment of the present application. In addition, the above specific details are only for the purpose of example and understanding, and are not limiting, and the above details do not limit the present application to the must-use of the above specific details.

[0139] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0140] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0141] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0142] The above description has been given for illustrative and descriptive purposes. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A sensor calibration method characterized by, The method is applied to a robot deployed with sensors, and comprises: determining coordinates of a target feature point in a calibration board in a calibration board coordinate system; determining a pose of the calibration board coordinate system relative to a robot body; based on the coordinates of the target feature point in the calibration board coordinate system and the pose of the calibration board coordinate system relative to the robot body, determining coordinates of the target feature point in a robot body coordinate system; based on the coordinates of the target feature point in the robot body coordinate system, determining extrinsic parameter data of a first target sensor of the robot relative to the robot body, the first target sensor being a sensor with a fixed relative positional relationship with the robot body; the sensors deployed on the robot further include a second target sensor with a variable relative positional relationship with the robot body; the method further comprises: determining extrinsic parameter data of the second target sensor relative to an end joint at the installation position of the second target sensor.

2. The sensor calibration method of claim 1, wherein, The first target sensor includes a laser radar, and the target feature point includes a laser radar feature point; the determination of the extrinsic parameter data of the first target sensor of the robot relative to the robot body based on the coordinates of the target feature point in the robot body coordinate system comprises: obtaining scanning data of the laser radar on the calibration board; based on the scanning data, determining coordinates of the laser radar feature point in a laser radar coordinate system; based on the coordinates of the laser radar feature point in the robot body coordinate system and the coordinates of the laser radar feature point in the laser radar coordinate system, determining extrinsic parameter data of the laser radar relative to the robot body.

3. The sensor calibration method of claim 1, wherein, The first target sensor includes a target camera, and the target feature point includes a camera feature point; the determination of the extrinsic parameter data of the first target sensor of the robot relative to the robot body based on the coordinates of the target feature point in the robot body coordinate system comprises: determining coordinates of the camera feature point in a target camera coordinate system; based on the coordinates of the camera feature point in the robot body coordinate and the coordinates of the camera feature point in the target camera coordinate system, determining extrinsic parameter data of the target camera relative to the robot body.

4. The sensor calibration method according to any one of claims 1 to 3, characterized in that, The second target sensor includes a head camera, and the target feature point includes a camera feature point; the method further comprises: based on the coordinates of the camera feature point in the robot body coordinate system, determining extrinsic parameter data of the head camera relative to a head end joint.

5. The sensor calibration method of claim 4, wherein, The determination of the extrinsic parameter data of the head camera relative to the head end joint based on the coordinates of the camera feature point in the robot body coordinate system comprises: determining coordinates of the camera feature point in a head camera coordinate system; based on the coordinates of the camera feature point in the robot body coordinate and the coordinates of the camera feature point in the head camera coordinate system, determining a pose of the head camera relative to the robot body at a current position; determining a pose of the head end joint relative to the robot body; Determine extrinsic data of the head camera relative to the head end joint based on the pose of the head camera relative to the robot body at the current position and the pose of the head end joint relative to the robot body.

6. The sensor calibration method according to any one of claims 1 to 3, characterized in that, The second target sensor includes a hand camera, and the target feature point includes a camera feature point; the method further includes: Obtain a target image of the calibration board captured by the hand camera at a plurality of poses of a robot arm; Identify coordinates of the camera feature point in a hand camera coordinate system based on the target image; Obtain coordinates of a target position in a hand end joint coordinate system when the robot arm moves to the target position of the calibration board; Determine extrinsic data of the hand camera relative to the hand end joint based on the coordinates of the camera feature point in the hand camera coordinate system and the coordinates of the target position in the hand end joint coordinate system.

7. The sensor calibration method of claim 6, wherein, The hand camera includes a left hand camera and / or a right hand camera.

8. The sensor calibration method of claim 6, wherein, The determination of the pose of the calibration board coordinate system relative to the robot body includes: Determine the pose of the calibration board coordinate system relative to the hand camera; Determine the pose of the hand end joint to the robot body; Determine the pose of the calibration board coordinate system relative to the robot body based on the pose of the calibration board coordinate system relative to the hand camera and the pose of the hand end joint to the robot body and the extrinsic data of the hand camera relative to the hand end joint.

9. The sensor calibration method according to any one of claims 1 to 3, characterized in that, The target feature point includes a laser radar feature point and / or a camera feature point, the laser radar feature point is determined based on a hollow circle in the calibration board, and the camera feature point is determined based on a target color circle in the calibration board.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program is used to execute the sensor calibration method in any one of claims 1 to 9.

11. An electronic device, comprising: Comprise: A processor; A memory for storing instructions executable by the processor; The processor is used to execute the sensor calibration method in any one of claims 1 to 9.

12. A computer program product, characterised in that, The computer program product comprises instructions for executing the sensor calibration method in any one of claims 1 to 9 when executed on an electronic device. The computer program product comprises instructions for executing the sensor calibration method in any one of claims 1 to 9 when executed on an electronic device.

Images