Sensor calibration method, storage medium, electronic equipment and program product

Abstract

The invention provides a sensor calibration method, a storage medium, electronic equipment and a program product, and relates to the technical field of sensor calibration. The sensor calibration method comprises the following steps: determining coordinates of a target feature point in a calibration plate in a calibration plate coordinate system; determining the pose of the calibration plate coordinate system relative to the robot body; based on the coordinate of the target feature point in the calibration plate coordinate system and the pose of the calibration plate coordinate system relative to the robot body, determining the coordinate of the target feature point in the robot body coordinate system; based on the coordinates of the target feature points in the robot body coordinate system, external parameter data of a first target sensor of the robot relative to the robot body is determined, and the first target sensor is a sensor with a fixed relative position relation with the robot body. According to the method, the dependence on a professional calibration site and high-precision equipment is avoided, the calibration process is simplified, the calibration cost is reduced, and the calibration precision is ensured.

Description

Technical Field

[0001] The present application relates to the technical field of sensor calibration, and in particular to a sensor calibration method, storage medium, electronic device and program product. Background Art

[0002] In the field of robotics, sensor calibration aims to ensure that the robot can maintain its accuracy after long-term use or after being affected by external factors, such as hardware replacement, environmental changes, equipment aging, etc.

[0003] However, existing sensor calibration technologies often require professional calibration sites and high-precision calibration equipment, and the calibration process is complicated, such as requiring the robot to perform figure-eight motion, resulting in high calibration costs. Therefore, a sensor calibration method that is both simple and low-cost while ensuring calibration accuracy is currently needed. Summary of the invention

[0004] In view of this, embodiments of the present application provide a sensor calibration method, a storage medium, an electronic device, and a program product.

[0005] In a first aspect, an embodiment of the present application provides a sensor calibration method, which is applied to a robot equipped with a sensor, the method comprising: determining the coordinates of a target feature point in a calibration plate in a calibration plate coordinate system;

[0006] Determine the position and posture of the calibration plate coordinate system relative to the robot body; determine the coordinates of the target feature point in the robot body coordinate system based on the coordinates of the target feature point in the calibration plate coordinate system and the position and posture of the calibration plate coordinate system relative to the robot body; determine the external parameter data of the first target sensor of the robot relative to the robot body based on the coordinates of the target feature point in the robot body coordinate system, the first target sensor being a sensor with a fixed relative position relationship with the robot body.

[0007] In combination with the first aspect, in certain implementations of the first aspect, the first target sensor includes a laser radar, and the target feature point includes a laser radar feature point; based on the coordinates of the target feature point in the robot body coordinate system, determining the external parameter data of the robot's first target sensor relative to the robot body, including: acquiring scanning data of the laser radar calibration plate; based on the scanning data, determining the coordinates of the laser radar feature point in the laser radar coordinate system; based on the coordinates of the laser radar feature point in the robot body coordinate system and the coordinates of the laser radar feature point in the laser radar coordinate system, determining the external parameter data of the laser radar relative to the robot body.

[0008] In combination with the first aspect, in certain implementations of the first aspect, the first target sensor includes a target camera, and the target feature point includes a camera feature point; based on the coordinates of the target feature point in the robot body coordinate system, determining the external parameter data of the first target sensor of the robot relative to the robot body, including: determining the coordinates of the camera feature point in the target camera coordinate system; based on the coordinates of the camera feature point in the robot body coordinates and the coordinates of the camera feature point in the target camera coordinate system, determining the external parameter data of the target camera relative to the robot body.

[0009] In combination with the first aspect, in certain implementations of the first aspect, the sensors deployed by the robot also include a second target sensor whose relative position relationship with the robot body is variable, the second target sensor includes a head camera, and the target feature points include camera feature points; the method also includes: determining the external parameter data of the head camera relative to the head terminal joint based on the coordinates of the camera feature points in the robot body coordinate system.

[0010] In combination with the first aspect, in some implementations of the first aspect, determining the external parameter data of the head camera relative to the head end joint based on the coordinate data of the camera feature point in the robot body coordinate system includes:

[0011] Determine the coordinates of the camera feature points in the head camera coordinate system; determine the posture of the head camera relative to the robot body at the current position based on the coordinates of the camera feature points in the robot body coordinates and the coordinates of the camera feature points in the head camera coordinate system; determine the posture of the head terminal joint relative to the robot body; determine the external parameter data of the head camera relative to the head terminal joint based on the posture of the head camera relative to the robot body at the current position and the posture of the head terminal joint relative to the robot body.

[0012] In combination with the first aspect, in certain implementations of the first aspect, the sensors deployed by the robot also include a second target sensor whose relative position relationship with the robot body is variable, the second target sensor includes a hand camera, and the target feature points include camera feature points; the method also includes: obtaining a target image of a calibration plate taken by the hand camera under multiple postures of the robot arm; based on the target image, identifying the coordinates of the camera feature points in the hand camera coordinate system; obtaining the coordinates of the target position in the hand end joint coordinate system when the robot arm moves to the target position of the calibration plate; based on the coordinates of the camera feature points in the hand camera coordinate system and the coordinates of the target position in the hand end joint coordinate system, determining the external parameter data of the hand camera relative to the hand end joint; preferably, the hand camera includes a left-hand camera and / or a right-hand camera.

[0013] In combination with the first aspect, in certain implementations of the first aspect, determining the posture of the calibration plate coordinate system relative to the robot body includes: determining the posture of the calibration plate coordinate system relative to the hand camera; determining the posture from the hand end joint to the robot body; determining the posture of the calibration plate coordinate system relative to the robot body based on the posture of the calibration plate coordinate system relative to the hand camera, the posture from the hand end joint to the robot body, and the external parameter data of the hand camera relative to the hand end joint.

[0014] In combination with the first aspect, in certain implementations of the first aspect, the target feature points include lidar feature points and / or camera feature points, the lidar feature points are determined based on the hollow circles in the calibration plate, and the camera feature points are determined based on the target color circles in the calibration plate.

[0015] In the second aspect, an embodiment of the present application provides a sensor calibration device, which is applied to a robot deployed with a sensor, and the device includes: a first determination module, used to determine the coordinates of a target feature point in a calibration plate in a calibration plate coordinate system; a second determination module, used to determine the position and posture of the calibration plate coordinate system relative to the robot body; a third determination module, used to determine the coordinates of the target feature point in the robot body coordinate system based on the coordinates of the target feature point in the calibration plate coordinate system and the position and posture of the calibration plate coordinate system relative to the robot body; a fourth determination module, used to determine the external parameter data of the first target sensor of the robot relative to the robot body based on the coordinates of the target feature point in the robot body coordinate system, the first target sensor being a sensor having a fixed relative position relationship with the robot body.

[0016] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program for executing the laser point cloud processing method described in the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; and the processor is used to execute the laser point cloud processing method described in the first aspect.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes instructions, which, when executed on an electronic device, enable the electronic device to implement the laser point cloud processing method described in the first aspect.

[0019] In the present application, by determining the coordinates of the target feature points in the calibration plate coordinate system, reference data is provided for the subsequent calibration process. Moreover, this step can be obtained by recording the position of the target feature points in the calibration plate when making the calibration plate, which simplifies the preparation work for sensor calibration. Next, by determining the position and posture of the calibration plate coordinate system relative to the robot body, the sensor calibration process is associated with the spatial position of the robot body. Then, through mathematical calculations, the coordinates of the target feature points are converted from the calibration plate coordinate system to the robot body coordinate system. This step does not require additional complex operations, reducing the complexity and cost of the calibration process. Finally, based on the coordinates of the target feature points in the robot body coordinate system, the external parameter data of the first target sensor relative to the robot body is determined, and the calibration of the first target sensor is completed.

[0020] In summary, the present application provides a sensor calibration method that is simple and low-cost. This method can be implemented through mathematical calculations and coordinate transformation, avoiding dependence on professional calibration sites and high-precision equipment. At the same time, it simplifies the calibration process and reduces the calibration cost without losing calibration accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0022] Figure 1 The figure is a flow chart of a sensor calibration method provided in one embodiment of the present application.

[0023] Figure 2 The figure is a schematic diagram of a process for determining external parameter data of a first target sensor of a robot relative to a robot body provided by an embodiment of the present application.

[0024] Figure 3 The figure is a schematic diagram of a process for determining external parameter data of a first target sensor of a robot relative to a robot body provided by another embodiment of the present application.

[0025] Figure 4 The figure is a flow chart of determining the external parameter data of the head camera relative to the head end joint based on the coordinate data of the camera feature points in the robot body coordinate system provided by an embodiment of the present application.

[0026] Figure 5 Shown is a flow chart of a sensor calibration method provided in another embodiment of the present application.

[0027] Figure 6 The figure is a schematic diagram of a process for determining the position and posture of the calibration plate coordinate system relative to the robot body provided in one embodiment of the present application.

[0028] Figure 7 Shown is a schematic diagram of a calibration plate provided in an embodiment of the present application.

[0029] Figure 8 Shown is a schematic diagram of the structure of a sensor calibration device provided in one embodiment of the present application.

[0030] Fig. 9 Shown is a schematic structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] Figure 1 FIG. 1 is a flow chart of a sensor calibration method provided by an embodiment of the present application. For example, the method is applied to a robot equipped with a sensor. Figure 1 As shown, the method includes the following steps.

[0033] Step S110, determining the coordinates of the target feature points in the calibration plate in the calibration plate coordinate system.

[0034] The target feature point in the calibration plate refers to a specific point pre-set on the surface of the calibration plate for the calibration process. The calibration plate coordinate system refers to a coordinate system established based on the physical position and orientation of the calibration plate. Exemplarily, the origin of the calibration plate coordinate system is located at a specific point of the calibration plate, such as the center or a corner of the calibration plate, and the coordinate axes of the calibration plate coordinate system are parallel to the physical edges of the calibration plate. Based on the design parameters of the target feature points in the calibration plate and the construction rules of the calibration plate coordinate system, the coordinate values ​​of each target feature point on the calibration plate in the calibration plate coordinate system can be determined.

[0035] Step S120, determining the position and posture of the calibration plate coordinate system relative to the robot body.

[0036] The position and posture of the calibration plate coordinate system relative to the robot body includes the translation vector and the rotation matrix. The translation vector determines the position of the origin of the calibration plate coordinate system in the robot body coordinate system, and the rotation matrix describes the rotation relationship of the calibration plate coordinate system relative to the robot body coordinate system. Through this position and posture, the precise spatial relationship of the calibration plate relative to the robot body can be known.

[0037] Step S130, based on the coordinates of the target feature points in the calibration plate coordinate system and the position and posture of the calibration plate coordinate system relative to the robot body, determine the coordinates of the target feature points in the robot body coordinate system.

[0038] Specifically, according to the position of the calibration plate coordinate system relative to the robot body, the coordinates of the target feature points are converted from the calibration plate coordinate system to the robot body coordinate system through coordinate transformation. For example, the coordinates of the target feature points in the robot body coordinate system are in, represents the position of the calibration plate coordinate system relative to the robot body, P cb Indicates the coordinates of the target feature point in the calibration plate coordinate system.

[0039] Step S140, based on the coordinates of the target feature point in the robot body coordinate system, determine the external parameter data of the first target sensor of the robot relative to the robot body.

[0040] The first target sensor is a sensor with a fixed relative position relationship with the robot body. That is to say, no matter how the robot body moves or changes its posture, the position and direction of the first target sensor relative to the robot body remain unchanged. The external parameter data of the first target sensor relative to the robot body refers to the position and direction of the first target sensor in the robot body coordinate system, including the translation vector and rotation matrix of the first target sensor relative to the robot body coordinate system.

[0041] Specifically, the coordinates of the target feature point in the first target sensor coordinate system are obtained, and the external parameter data of the first target sensor relative to the robot body is determined based on the coordinates of the target feature point in the first target sensor coordinate system and the coordinates of the target feature point in the robot body coordinate system.

[0042] Optionally, by comparing the coordinates of the same target feature point in the first target sensor coordinate system and the robot body coordinate system, a transformation equation can be established, and then the rotation matrix and the translation vector can be obtained by solving the transformation equation.

[0043] For example, assuming that the coordinates of the target feature point in the first target sensor coordinate system are (x b ,y b ,z b), the coordinates of the target feature point in the robot body coordinate system are (x r ,y r ,z r ), then the conversion equation between the two is: Where R represents the rotation matrix, Represents the translation vector.

[0044] Optionally, the optimal rotation matrix R and translation vector can be found by mathematical optimization methods, such as the least squares method. That is, the external parameter data of the first target sensor relative to the robot body is obtained.

[0045] In this embodiment, by determining the coordinates of the target feature points in the calibration plate coordinate system, reference data is provided for the subsequent calibration process. Moreover, this step can be obtained by recording the position of the target feature points in the calibration plate when making the calibration plate, which simplifies the preparation work for sensor calibration. Next, by determining the position and posture of the calibration plate coordinate system relative to the robot body, the sensor calibration process is associated with the spatial position of the robot body. Then, through mathematical calculations, the coordinates of the target feature points are converted from the calibration plate coordinate system to the robot body coordinate system. This step does not require additional complex operations, reducing the complexity and cost of the calibration process. Finally, based on the coordinates of the target feature points in the robot body coordinate system, the external parameter data of the first target sensor relative to the robot body is determined, and the calibration of the first target sensor is completed.

[0046] In summary, the present application provides a sensor calibration method that is simple and low-cost. This method can be implemented through mathematical calculations and coordinate transformation, avoiding dependence on professional calibration sites and high-precision equipment. At the same time, it simplifies the calibration process and reduces the calibration cost without losing calibration accuracy.

[0047] Figure 2 The figure shows a flow chart of determining the external parameter data of the first target sensor of the robot relative to the robot body provided by an embodiment of the present application. Figure 1 Based on the embodiment shown, Figure 2 The embodiment shown is described below in detail. Figure 2 The embodiment shown and Figure 1 The differences and similarities of the illustrated embodiments are not described in detail.

[0048] First, in this embodiment, the first target sensor includes a laser radar, and illustratively, the laser radar is deployed in the chassis of the robot. In addition, the target feature point includes a laser radar feature point, and the laser radar feature point refers to a point with specific attributes detected by the laser radar when scanning the calibration plate, for example, the laser radar feature point is a point with abnormal reflection intensity in the calibration plate.

[0049] like Figure 2 As shown, based on the coordinates of the target feature point in the robot body coordinate system, determining the external parameter data of the first target sensor of the robot relative to the robot body includes the following steps.

[0050] Step S210, obtaining scanning data of the laser radar on the calibration plate.

[0051] Before the laser radar collects scanning data of the calibration plate, it is necessary to ensure that the calibration plate is within the scanning range of the laser radar, and that the robot and the calibration plate remain relatively still during the scanning process, and that the distance and angle between the calibration plate and the robot are appropriate.

[0052] In addition, in order to improve the accuracy of calibration, the laser radar needs to scan the calibration plate multiple times from different positions and angles to obtain more abundant information of the laser radar feature points. That is, the scanning data in this embodiment is multiple frames of data obtained by the laser radar scanning the calibration plate from different positions and angles.

[0053] Step S220, based on the scanning data, determine the coordinates of the laser radar feature point in the laser radar coordinate system.

[0054] Specifically, the scan data contains the distance and angle information of each LiDAR feature point relative to the LiDAR transmitter. Using this information, the position of each LiDAR feature point in the LiDAR coordinate system can be calculated.

[0055] Step S230, based on the coordinates of the laser radar feature points in the robot body coordinate system and the coordinates of the laser radar feature points in the laser radar coordinate system, determine the external parameter data of the laser radar relative to the robot body.

[0056] In one example, after obtaining the coordinates of the laser radar feature points in the robot body coordinate system and the laser radar coordinate system, a transformation equation between the two is established. By solving the transformation equation, the rotation matrix and translation vector are obtained, that is, the external parameter data of the laser radar relative to the robot body is obtained.

[0057] In another example, based on the coordinates of each lidar feature point in the robot body coordinate system and the lidar coordinate system, a 3D-3D ICP algorithm is used to determine the external parameter data of the lidar relative to the robot body.

[0058] In this embodiment, by acquiring the scanning data of the laser radar on the calibration plate, the point cloud information of the calibration plate in the laser radar coordinate system is collected. Then, by parsing these scanning data, the coordinates of the laser radar feature points in the laser radar coordinate system are determined, providing an accurate reference for the subsequent coordinate conversion. Finally, the coordinates of the laser radar feature points in the robot body coordinate system and the laser radar coordinate system are used to determine the external parameter data of the laser radar relative to the robot body through mathematical transformation or ICP algorithm.

[0059] In summary, this embodiment provides an efficient, accurate, and low-cost LiDAR calibration method, which enables the robot to maintain high precision and high performance of the LiDAR in various working environments, whether after long-term use, hardware replacement, environmental changes, or equipment aging.

[0060] Figure 3 The figure shows a flow chart of determining the external parameter data of the first target sensor of the robot relative to the robot body provided by another embodiment of the present application. Figure 1 Based on the embodiment shown, Figure 3 The embodiment shown is described below in detail. Figure 3 The embodiment shown and Figure 1 The differences and similarities of the illustrated embodiments are not described in detail.

[0061] First, in this embodiment, the first target sensor includes a target camera, and illustratively, the target camera is installed in the chassis of the robot. The target feature point includes a camera feature point, which refers to a distinct and distinguishable feature point in the image captured by the target camera. For example, the camera feature point is a point with a unique texture.

[0062] like Figure 3 As shown, based on the coordinates of the target feature point in the robot body coordinate system, determining the external parameter data of the first target sensor of the robot relative to the robot body includes the following steps.

[0063] Step S310, determining the coordinates of the camera feature points in the target camera coordinate system.

[0064] Specifically, an image of a calibration plate captured by a target camera is obtained; pixel coordinates of camera feature points in the calibration plate in the image are determined; and based on the intrinsic parameter data of the target camera, the pixel coordinates are converted into coordinates in the target camera coordinate system.

[0065] It should be noted that when the target camera shoots the calibration plate, it is necessary to ensure that the calibration plate is in the field of view of the target camera and is not blocked. During the shooting process, the target camera should remain stable to avoid affecting the accurate positioning of the camera feature points.

[0066] Step S320, based on the coordinates of the camera feature points in the robot body coordinate system and the coordinates of the camera feature points in the target camera coordinate system, determine the external parameter data of the target camera relative to the robot body.

[0067] In one example, after obtaining the coordinates of the camera feature points in the robot body coordinate system and the target camera coordinate system, a transformation equation between the two is established. By solving the transformation equation, the rotation matrix and translation vector are obtained, that is, the external parameter data of the target camera relative to the robot body is obtained.

[0068] In another example, based on the coordinates of the camera feature points in the robot body coordinate system and the coordinates of the camera feature points in the target camera coordinate system, the extrinsic parameter data of the target camera relative to the robot body is calculated using a 3D-2D PnP algorithm.

[0069] In this embodiment, by acquiring the image of the calibration plate taken by the target camera and determining the coordinates of the camera feature points in the target camera coordinate system, an accurate data basis is provided for subsequent calibration. Then, by using the coordinates of the camera feature points in the target camera coordinate system and the known coordinates of the camera feature points in the robot body coordinate system, the external parameter data of the target camera relative to the robot body is obtained by establishing a conversion equation or a 3D-2D PnP algorithm. In summary, this embodiment provides an efficient, accurate and low-cost target table calibration method so that the robot can accurately combine the visual data captured by the camera with the robot's motion control data, thereby improving the robot's operating accuracy and environmental perception capabilities.

[0070] The above embodiments mainly describe the calibration method of the laser radar and the target camera whose relative position relationship with the robot body is fixed. The following focuses on the calibration method of the second target sensor whose relative position relationship with the robot body is variable.

[0071] In some embodiments, the second target sensor includes a head camera, and the method further includes: determining external parameter data of the head camera relative to the head end joint based on the coordinates of the camera feature points in the robot body coordinate system.

[0072] Since the relative positional relationship between the head camera and the robot body is variable, it does not have fixed external parameter data like fixed sensors (such as laser radar or target cameras in the chassis). However, in robot design, the relative positional relationship between the head camera and the end joint of the head is fixed, and the end joint of the head is endowed with multiple degrees of freedom, which can perform actions similar to the human head, such as nodding up and down, turning left and right, and tilting. Therefore, this embodiment is to determine the external parameter data of the head camera relative to the end joint of the head, so as to link the observation data of the head camera with the movement and posture of the end joint of the head, so as to achieve more accurate spatial positioning and environmental perception.

[0073] Figure 4 The figure shows a flow chart of determining the external parameter data of the head camera relative to the head end joint based on the coordinate data of the camera feature points in the robot body coordinate system provided by an embodiment of the present application. Figure 4 As shown, the following steps are included.

[0074] Step S410, determining the coordinates of the camera feature points in the head camera coordinate system.

[0075] Specifically, an image of a calibration plate taken by a head camera is obtained; pixel coordinates of camera feature points in the calibration plate in the image are determined; and based on the intrinsic parameter data of the head camera, the pixel coordinates are converted into coordinates in the head camera coordinate system.

[0076] It is also important to note that when the head camera is shooting the calibration plate, it is necessary to ensure that the calibration plate is in the field of view of the head camera and is not blocked. During the shooting process, the head camera should remain stable to avoid affecting the accurate positioning of the camera feature points.

[0077] Step S420, based on the coordinates of the camera feature points in the robot body coordinate system and the coordinates of the camera feature points in the head camera coordinate system, determine the posture of the head camera relative to the robot body at the current position.

[0078] Exemplarily, a transformation equation for the coordinates of the camera feature points in the robot body coordinate system and the head camera coordinate system is established, and by solving the transformation equation, the rotation matrix and translation vector are obtained, that is, the external parameter data of the head camera relative to the robot body at the current position is obtained.

[0079] In addition, it can be seen from the previous analysis that the position of the head camera relative to the robot body is not fixed, but changes dynamically with the movement of the head end joint. Therefore, at different time points or different task stages, the head camera may point to different directions or be in different positions. Therefore, through step S420, the position of the head camera relative to the robot body at a certain position is obtained.

[0080] Step S430, determining the position and posture of the head end joint relative to the robot body.

[0081] Optionally, the position and orientation of the head end joint relative to the robot body can be determined by robot kinematics. Specifically, first, the angles of the robot joints are measured in real time, and these measurement data are provided by encoders or other sensors on the joints. Then, according to the forward kinematics model of the robot, the angles of these joints are used to calculate the position and orientation of the head end joint in the robot body coordinate system.

[0082] The forward kinematics model is a mathematical equation based on the geometric parameters of the robot's joints and links, which defines how to calculate the position and posture of the end effector from the joint angles. By accumulating the local transformations (including rotation and translation) of each joint, the position and posture of the head end joint to the robot body can be obtained.

[0083] Step S440, based on the posture of the head camera relative to the robot body at the current position, and the posture of the head end joint relative to the robot body, determine the external parameter data of the head camera relative to the head end joint.

[0084] The external parameter data of the head camera relative to the end joint of the head is calculated by the posture of the head camera relative to the robot body at the current position, and the posture of the end joint of the head relative to the robot body. This is actually the inverse process of coordinate transformation. Specifically, the posture transformation matrix of the head camera relative to the robot body is The inverse of the pose transformation matrix of the head end joint relative to the robot body Multiply them together to get the transformation matrix of the head camera relative to the end joint of the head Right now

[0085] In this embodiment, the coordinates of the camera feature points in the head camera coordinate system are first determined, and then the coordinates of these target feature points in the robot body coordinate system are combined to calculate the position and posture of the head camera relative to the robot body at the current position. Subsequently, by determining the position and posture of the head terminal joint relative to the robot body, the external parameter data of the head camera relative to the head terminal joint is further calculated, thereby obtaining the precise position and direction of the head camera relative to the head terminal joint. Through this scheme, it is ensured that the robot can accurately predict the field of view and image data of the head camera when performing head movements, thereby improving the movement accuracy and reliability of the head terminal joint, the execution end. In addition, the robot can adjust the position and posture of the head terminal joint in real time according to the visual information provided by the head camera to adapt to environmental changes or task requirements.

[0086] Figure 5FIG. 2 is a flow chart of a sensor calibration method provided by another embodiment of the present application. Figure 5 As shown, in this embodiment, the second target sensor also includes a hand camera, the hand camera includes a left hand camera and / or a right hand camera, and the method also includes the following steps.

[0087] Step S510, obtaining target images of the calibration plate captured by the hand camera at multiple positions of the robot arm.

[0088] Exemplarily, the calibration plate is placed at different positions or distances in the robot's workspace to ensure that the hand camera can capture the image of the calibration plate at multiple preset postures (different angles and positions) of the robot arm, thereby determining the target image.

[0089] Step S520: Based on the target image, identify the coordinates of the camera feature points in the hand camera coordinate system.

[0090] Specifically, the pixel coordinates of the camera feature points in the calibration plate in the target image are first determined; based on the intrinsic parameter data of the hand camera, the pixel coordinates are converted into coordinates in the hand camera coordinate system.

[0091] Step S530, obtaining the coordinates of the target position in the hand end joint coordinate system when the robot arm moves to the target position of the calibration plate.

[0092] Using the robot's kinematic model, the position of the hand end joint actuator in the robot body coordinate system is calculated based on the joint angle of the robot arm. Then, in the robot body coordinate system, the coordinates of the target position of the calibration plate are measured or calculated. Finally, the coordinates of the target position are converted from the robot body coordinate system to the hand end joint coordinate system.

[0093] Step S540, based on the coordinates of the camera feature points in the hand camera coordinate system and the coordinates of the target position in the hand end joint coordinate system, determine the external parameter data of the hand camera relative to the hand end joint.

[0094] Based on the positional relationship between the camera feature point and the target position in the calibration plate coordinate system (in some embodiments, the target position is the position where the camera feature point is located) and the coordinates of the target position in the hand end joint coordinate system, the coordinates of the camera feature point in the hand end joint coordinate system can be calculated; based on the coordinates of the camera feature point in the hand camera coordinate system and the coordinates of the camera feature point in the hand end joint coordinate system, the external parameter data of the hand camera relative to the hand end joint is determined.

[0095] Exemplarily, a coordinate transformation equation of the camera feature points in the hand camera coordinate system and the hand end joint coordinate system is established, and by solving the transformation equation, a rotation matrix and a translation vector are obtained, that is, external parameter data of the hand camera relative to the hand end joint is obtained.

[0096] In this embodiment, by acquiring the target image of the calibration plate taken by the hand camera in multiple different positions, it can be ensured that the calibration process does not rely solely on a single viewing angle or a specific arm position, so that the external parameter data of the hand camera relative to the hand end joint that is finally determined is more comprehensive and reliable. In addition, the viewing angle changes of the hand camera in different directions and distances can be captured, which helps to correct the internal distortion of the hand camera and reduce the calibration inaccuracy caused by accidental errors.

[0097] Figure 6 FIG. 1 is a schematic diagram of a process for determining the position of the calibration plate coordinate system relative to the robot body provided by an embodiment of the present application. Figure 5 Based on the embodiment shown, Figure 6 The embodiment shown is described below in detail. Figure 6 The embodiment shown and Figure 5 The differences and similarities of the illustrated embodiments are not described in detail.

[0098] like Figure 6 As shown, in this embodiment, determining the position and posture of the calibration plate coordinate system relative to the robot body includes the following steps.

[0099] Step S610, determining the position and posture of the calibration plate coordinate system relative to the hand camera.

[0100] Exemplarily, the hand camera is aimed at the calibration plate and an image is captured. Then, computer vision techniques, such as feature extraction algorithms, are used to identify and accurately locate the target feature points on the calibration plate. After the pixel coordinates of these target feature points in the image are determined, the pixel coordinates are converted into coordinates in the hand camera coordinate system by using the intrinsic parameters of the hand camera (including focal length, principal point, distortion parameters, etc.). Subsequently, by comparing the coordinates of the target feature points in the hand camera coordinate system with the coordinates of the target feature points in the calibration plate coordinate system, the rotation and translation parameters of the calibration plate coordinate system relative to the hand camera, that is, the position and posture of the calibration plate coordinate system relative to the hand camera, can be calculated.

[0101] Exemplarily, this process may also use an iterative closest point algorithm or other registration techniques to optimize the pose estimation of the calibration plate coordinate system relative to the hand camera to ensure that the alignment between the calibration plate coordinate system and the hand camera coordinate system is as accurate as possible.

[0102] Step S620, determining the position and posture of the hand end joint to the robot body.

[0103] As mentioned above, the robot kinematics can be used to determine the position and posture of the hand end joint to the robot body, which will not be repeated here.

[0104] Step S630, based on the posture of the calibration plate coordinate system relative to the hand camera, the posture of the hand end joint to the robot body, and the external parameter data of the hand camera relative to the hand end joint, determine the posture of the calibration plate coordinate system relative to the robot body.

[0105] Specifically, based on the pose of the calibration plate coordinate system relative to the hand camera and the external parameter data of the hand camera relative to the hand end joints, the external parameter data of the calibration plate coordinate system relative to the hand end joints can be determined; based on the external parameter data of the calibration plate coordinate system relative to the hand end joints and the pose of the hand end joints to the robot body, the pose of the calibration plate coordinate system relative to the robot body can be determined.

[0106] That is, by multiplying the position of the calibration plate coordinate system relative to the hand camera by the external parameter data of the hand camera relative to the hand end joint, the external parameter data of the calibration plate coordinate system relative to the hand end joint can be obtained. Then, by multiplying the external parameter data of the calibration plate coordinate system relative to the hand end joint by the position from the hand end joint to the robot body, the position of the calibration plate coordinate system relative to the robot body can be obtained.

[0107] In this embodiment, through mathematical coordinate transformation, the calibration plate coordinate system relative to the hand camera posture, the hand end joint to the robot body posture, the hand camera relative to the hand end joint external parameter data and other information are combined to achieve the conversion from the calibration plate coordinate system to the robot body. This conversion describes the final posture of the calibration plate coordinate system relative to the robot body, thereby completing the determination of the calibration plate coordinate system relative to the robot body posture, so that the robot can accurately understand and use the information from the sensors deployed by itself.

[0108] In some embodiments of the present application, the laser radar feature points are determined based on the hollow circles in the calibration plate, and the camera feature points are determined based on the target color circles in the calibration plate.

[0109] Figure 7 FIG. 1 is a schematic diagram of a calibration plate provided in an embodiment of the present application. Figure 7 As shown, the calibration plate is a plane calibration plate, the target color circle is a black circle, the hollow circles are located at the four corners of the calibration plate, and the horizontal and vertical spacings between the black circles are fixed.

[0110] Specifically, this calibration board combines the advantages of LiDAR and cameras in different environments, and improves the accuracy and robustness of the calibration process. For the camera, using the center of the target color circle as the camera feature point can better handle nonlinear distortion and reduce sensitivity to light compared to traditional checkerboard corner point extraction. It can also stably detect in low-contrast or high-reflection environments, thereby improving the accuracy of camera feature point extraction. For LiDAR, LiDAR feature points are extracted by fitting the geometric characteristics of the hollow circle, which is not affected by reflection intensity or surface material. Even when the circumference of the hollow circle is partially blocked, the position of the center of the hollow circle (i.e., the LiDAR feature point) can be accurately calculated, achieving sub-point cloud level accuracy, and has strong anti-interference capabilities. Combining these advantages, the present application solution can provide stable and reliable calibration results in a changing working environment, providing a solid foundation for the robot's precise operation and environmental perception.

[0111] Based on the above embodiments, it can be seen that the sensor calibration method provided in this embodiment only needs to use one calibration plate and requires the robot to be placed in front of the calibration plate to complete the calibration of all sensors. This method is simple and easy to implement, and can solve the problems of complex operation and high cost in the background technology, while ensuring the accuracy of calibration.

[0112] The sensors that can be calibrated in this application are divided into two categories: static sensors and dynamic sensors. Simply put, static sensors include target cameras and laser radars, etc., which are fixedly installed at specific positions of the robot, so their positional relationship relative to the robot body is constant. Dynamic sensors, such as left-hand cameras, right-hand cameras, and head cameras, are installed on the movable joints of the robot. Their relative positions to the connected joints remain unchanged, but their positional relationship relative to the robot body will change with the movement of the connected joints. Through the sensor calibration method in this application, the operation process can be simplified, costs can be reduced, and the efficiency and convenience of robot sensor calibration can be improved while ensuring the calibration accuracy.

[0113] Combined with 1 to Figure 7 , describes in detail the sensor calibration method embodiment of the present application, and the following is combined with Figure 8 , describes in detail the sensor calibration device embodiment of the present application. It should be understood that the description of the sensor calibration method embodiment corresponds to the description of the sensor calibration device embodiment, so the parts not described in detail can refer to the previous method embodiment.

[0114] Figure 8 FIG. 1 is a schematic diagram of the structure of a sensor calibration device provided in an embodiment of the present application. Fig. 9 As shown, the sensor calibration device 80 provided in the embodiment of the present application includes:

[0115] A first determination module 810 is used to determine the coordinates of the target feature points in the calibration plate in the calibration plate coordinate system;

[0116] The second determination module 820 is used to determine the position and posture of the calibration plate coordinate system relative to the robot body;

[0117] A third determination module 830 is used to determine the coordinates of the target feature point in the robot body coordinate system based on the coordinates of the target feature point in the calibration plate coordinate system and the position and posture of the calibration plate coordinate system relative to the robot body;

[0118] The fourth determination module 840 is used to determine the external parameter data of the first target sensor of the robot relative to the robot body based on the coordinates of the target feature point in the robot body coordinate system, and the first target sensor is a sensor with a fixed relative position relationship with the robot body.

[0119] In one embodiment of the present application, the first target sensor includes a laser radar, and the target feature point includes a laser radar feature point; the fourth determination module 840 is also used to obtain scanning data of the laser radar calibration plate; based on the scanning data, determine the coordinates of the laser radar feature point in the laser radar coordinate system; based on the coordinates of the laser radar feature point in the robot body coordinate system, and the coordinates of the laser radar feature point in the laser radar coordinate system, determine the external parameter data of the laser radar relative to the robot body.

[0120] In one embodiment of the present application, the first target sensor includes a target camera, and the target feature points include camera feature points; the fourth determination module 840 is also used to determine the coordinates of the camera feature points in the target camera coordinate system; based on the coordinates of the camera feature points in the robot body coordinates and the coordinates of the camera feature points in the target camera coordinate system, determine the external parameter data of the target camera relative to the robot body.

[0121] In an embodiment of the present application, the sensors deployed by the robot further include a second target sensor whose relative position relationship with the robot body is variable, the second target sensor includes a head camera, and the target feature points include camera feature points. The fourth determination module 840 is also used to determine the external parameter data of the head camera relative to the head end joint based on the coordinates of the camera feature points in the robot body coordinate system.

[0122] In one embodiment of the present application, the fourth determination module 840 is also used to determine the coordinates of the camera feature points in the head camera coordinate system; determine the posture of the head camera relative to the robot body at the current position based on the coordinates of the camera feature points in the robot body coordinates and the coordinates of the camera feature points in the head camera coordinate system; determine the posture of the head terminal joint relative to the robot body; determine the external parameter data of the head camera relative to the head terminal joint based on the posture of the head camera relative to the robot body at the current position and the posture of the head terminal joint relative to the robot body.

[0123] In one embodiment of the present application, the sensors deployed by the robot also include a second target sensor whose relative position relationship with the robot body is variable, the second target sensor includes a hand camera, and the target feature points include camera feature points; the fourth determination module 840 is also used to obtain the target image of the calibration plate taken by the hand camera under multiple postures of the robot arm; based on the target image, identify the coordinates of the camera feature points in the hand camera coordinate system; obtain the coordinates of the target position in the hand end joint coordinate system when the robot arm moves to the target position of the calibration plate; based on the coordinates of the camera feature points in the hand camera coordinate system and the coordinates of the target position in the hand end joint coordinate system, determine the external parameter data of the hand camera relative to the hand end joint; preferably, the hand camera includes a left-hand camera and / or a right-hand camera.

[0124] In one embodiment of the present application, the second determination module 820 is also used to determine the posture of the calibration plate coordinate system relative to the hand camera; determine the posture of the hand end joint to the robot body; based on the posture of the calibration plate coordinate system relative to the hand camera, the posture of the hand end joint to the robot body, and the external parameter data of the hand camera relative to the hand end joint, determine the posture of the calibration plate coordinate system relative to the robot body.

[0125] In one embodiment of the present application, the target feature points include lidar feature points and / or camera feature points. The lidar feature points are determined based on the hollow circles in the calibration plate, and the camera feature points are determined based on the target color circles in the calibration plate.

[0126] Below, reference Fig. 9 To describe an electronic device according to an embodiment of the present application. Fig. 9 Shown is a schematic structural diagram of an electronic device provided by an exemplary embodiment of the present application.

[0127] like Fig. 9 As shown, the electronic device 90 includes one or more processors 901 and a memory 902 .

[0128] The processor 901 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 90 to perform desired functions.

[0129] The memory 902 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 901 may run the program instructions to implement the sensor calibration method of each embodiment of the present application described above and / or other desired functions. Various contents such as the coordinates of the target feature points in the calibration plate coordinate system, the position of the calibration plate coordinate system relative to the robot body, the coordinates of the target feature points in the robot body coordinate system, and the external parameter data of the first target sensor relative to the robot body may also be stored in the computer-readable storage medium.

[0130] In one example, the electronic device 90 may further include: an input device 903 and an output device 904 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0131] The input device 903 may include, for example, a keyboard, a mouse, etc.

[0132] The output device 904 can output various information to the outside, including the coordinates of the target feature point in the calibration plate coordinate system, the position of the calibration plate coordinate system relative to the robot body, the coordinates of the target feature point in the robot body coordinate system, the external parameter data of the first target sensor relative to the robot body, etc. The output device 904 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0133] Of course, to simplify, Fig. 9 Only some of the components related to the present application in the electronic device 90 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 90 may also include any other appropriate components.

[0134] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the sensor calibration method according to various embodiments of the present application described above in this specification.

[0135] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0136] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the sensor calibration method according to various embodiments of the present application described above in this specification.

[0137] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0138] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0139] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0140] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0141] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0142] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A sensor calibration method, characterized in that: Applied to a robot equipped with a sensor, the method comprises: Determine the coordinates of the target feature points in the calibration plate in the calibration plate coordinate system; Determine the position and posture of the calibration plate coordinate system relative to the robot body; Determine the coordinates of the target feature point in the robot body coordinate system based on the coordinates of the target feature point in the calibration plate coordinate system and the position and posture of the calibration plate coordinate system relative to the robot body; Based on the coordinates of the target feature point in the robot body coordinate system, the external parameter data of the first target sensor of the robot relative to the robot body is determined, and the first target sensor is a sensor with a fixed relative position relationship with the robot body.

2. The sensor calibration method according to claim 1, characterized in that: The first target sensor includes a laser radar, and the target feature point includes a laser radar feature point; and determining the external parameter data of the first target sensor of the robot relative to the robot body based on the coordinates of the target feature point in the robot body coordinate system includes: Acquire scanning data of the laser radar on the calibration plate; Based on the scan data, determining the coordinates of the laser radar feature point in the laser radar coordinate system; Based on the coordinates of the laser radar feature point in the robot body coordinate system and the coordinates of the laser radar feature point in the laser radar coordinate system, the external parameter data of the laser radar relative to the robot body is determined.

3. The sensor calibration method according to claim 1, characterized in that: The first target sensor includes a target camera, and the target feature point includes a camera feature point; and determining the external parameter data of the first target sensor of the robot relative to the robot body based on the coordinates of the target feature point in the robot body coordinate system includes: Determine the coordinates of the camera feature points in the target camera coordinate system; Based on the coordinates of the camera feature points in the robot body coordinate system and the coordinates of the camera feature points in the target camera coordinate system, the external parameter data of the target camera relative to the robot body is determined.

4. The sensor calibration method according to any one of claims 1 to 3, characterized in that: The sensor deployed by the robot further includes a second target sensor whose relative position relationship with the robot body is variable, the second target sensor includes a head camera, and the target feature point includes a camera feature point; the method further includes: Based on the coordinates of the camera feature points in the robot body coordinate system, the external parameter data of the head camera relative to the head end joint is determined.

5. The sensor calibration method according to claim 4, characterized in that: The determining, based on the coordinate data of the camera feature point in the robot body coordinate system, the external parameter data of the head camera relative to the head end joint comprises: Determine the coordinates of the camera feature points in the head camera coordinate system; Determine the pose of the head camera relative to the robot body at the current position based on the coordinates of the camera feature points in the robot body coordinate system and the coordinates of the camera feature points in the head camera coordinate system; Determining the position and posture of the head end joint relative to the robot body; Based on the posture of the head camera relative to the robot body at the current position, and the posture of the head end joint relative to the robot body, the external parameter data of the head camera relative to the head end joint is determined.

6. The sensor calibration method according to any one of claims 1 to 3, characterized in that: The sensors deployed by the robot further include a second target sensor whose relative position relationship with the robot body is variable, the second target sensor includes a hand camera, and the target feature points include camera feature points; the method further includes: Acquire target images of the calibration plate captured by the hand camera at multiple positions of the robot arm; Based on the target image, identifying the coordinates of the camera feature points in the hand camera coordinate system; Obtaining the coordinates of the target position in the hand end joint coordinate system when the robot arm moves to the target position of the calibration plate; Determine the external parameter data of the hand camera relative to the hand end joint based on the coordinates of the camera feature point in the hand camera coordinate system and the coordinates of the target position in the hand end joint coordinate system; Preferably, the hand camera includes a left-hand camera and / or a right-hand camera.

7. The sensor calibration method according to claim 6, characterized in that: Determining the position and posture of the calibration plate coordinate system relative to the robot body includes: Determining the position and posture of the calibration plate coordinate system relative to the hand camera; Determine the position and posture of the hand end joint to the robot body; Based on the posture of the calibration plate coordinate system relative to the hand camera, the posture of the hand end joint to the robot body, and the external parameter data of the hand camera relative to the hand end joint, the posture of the calibration plate coordinate system relative to the robot body is determined.

8. The sensor calibration method according to any one of claims 1 to 3, characterized in that: The target feature points include laser radar feature points and / or camera feature points. The laser radar feature points are determined based on the hollow circles in the calibration plate, and the camera feature points are determined based on the target color circles in the calibration plate.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the sensor calibration method according to any one of claims 1 to 8.

10. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is used to execute the sensor calibration method described in any one of claims 1 to 8.

11. A computer program product, characterized in that The computer program product comprises instructions, which, when executed on an electronic device, enable the electronic device to implement the sensor calibration method according to any one of claims 1 to 8.

Images