Robot multi-camera external parameter automatic calibration method

Abstract

The invention discloses a robot multi-camera external parameter automatic calibration method, which comprises the following steps: setting a plurality of reference obstacles in a calibration site, and constructing a global reference obstacle grid map; the method comprises the following steps: acquiring shot pictures acquired by a camera when a robot is positioned in different directions, and performing pretreatment to obtain local reference obstacle grid maps in different directions; inputting the shot picture into a deep learning instance segmentation model to perform pixel level classification between the obstacle and the non-obstacle to obtain a binary picture, and combining the binary picture with the camera internal reference corresponding to the shot picture to obtain the projection of the obstacle pixel on the ground; searching within a certain range to generate a plurality of candidate external parameter drift matrixes, and performing error minimization selection on the candidate external parameter drift matrixes by using a local reference obstacle grid map to obtain real external parameter drift; and adding the real extrinsic parameter drift and the known nominal extrinsic parameter to obtain the calibrated extrinsic parameter of the camera. According to the invention, the external parameters of the camera can be automatically calibrated.

Description

Technical Field

[0001] This invention relates to the field of camera calibration technology, and in particular to an automatic calibration method for the extrinsic parameters of multiple cameras in a robot. Background Technology

[0002] Robots frequently rely on cameras to acquire scene information for environmental perception. The calibration accuracy of the camera's extrinsic parameters (i.e., the camera's position and orientation relative to the robot's coordinate system) directly determines the accuracy of the projection of the perceived data into the robot's coordinate system. During long-term use, robots are affected by vibrations, thermal expansion and contraction, and mechanical assembly errors, causing the camera's extrinsic parameters to drift over time. Inaccurate extrinsic parameters lead to deviations in the position of perceived information (such as object positions) within the robot's coordinate system, thus affecting the accuracy of navigation, obstacle avoidance, and robot operation. For example, the robot may collide with obstacles due to inaccurate obstacle position calculations. Therefore, automatic extrinsic parameter calibration is a necessary task during the long-term use of robots to ensure their environmental perception capabilities.

[0003] The known methods for calibrating the extrinsic parameters of robot cameras include the following:

[0004] The classic method based on calibration boards involves placing a calibration board (such as a checkerboard or dot array) in the robot's workspace, then photographing the board with the robot's camera to extract feature points. The pose of the calibration board is then matched with the camera observations using the pose information of the robot's end effector. Finally, the rigid body transformation relationship between the camera and the robot's base is solved. While this method is mature and highly accurate, it is cumbersome to operate and requires significant operator experience and high precision in the calibration board manufacturing process, thus presenting a high barrier to entry and substantial cost.

[0005] Adaptive calibration method: This method calibrates extrinsic parameters in real time during robot operation, for example, by fusing camera and LiDAR observations and adjusting extrinsic parameters online. Although this method greatly reduces human intervention, the algorithm is complex, the development cost is high, and there are significant challenges to program stability. Furthermore, it cannot be implemented on robots based on pure vision perception (without LiDAR).

[0006] Self-calibration based on natural environment features: This method utilizes natural features in the scene (such as road signs, lane lines, etc.) to adaptively adjust the camera extrinsic parameters during robot operation. Although this method does not require a dedicated calibration board and can operate in dynamic environments, its accuracy is heavily dependent on scene features, making it susceptible to environmental interference and exhibiting poor robustness.

[0007] Multi-sensor joint calibration: Common methods include joint calibration using a camera and an inertial mass unit (IMU). For example, motion information (angular velocity / acceleration) measured by the IMU is compared with camera observations to solve for extrinsic parameters. This method can fuse multiple sensors and is robust, but it significantly increases system complexity, such as time synchronization and data alignment between multiple sensors. Furthermore, it is computationally expensive, making it difficult to implement for robots with low computing power. Its accuracy also depends on the accuracy of other sensors; for example, high noise from the IMU directly affects calibration accuracy.

[0008] In summary, existing methods for calibrating / calibrating robot camera extrinsic parameters suffer from drawbacks such as operational complexity, high barriers to entry, and susceptibility to environmental influences on accuracy. These drawbacks lead to high calibration costs (including labor, computing power, and sensor costs) during actual robot production or maintenance. Therefore, a camera extrinsic parameter calibration method that is easy to operate, requires minimal operator skill (not dependent on operator experience), has low computing power requirements, and is highly robust is crucial for reducing robot maintenance costs and ensuring the robot's environmental perception capabilities. Summary of the Invention

[0009] To address the aforementioned technical problems, this invention proposes an automatic calibration method for the extrinsic parameters of multiple cameras on robots. This method enables robots equipped with multiple cameras to automatically calibrate the extrinsic parameters of the cameras during factory manufacturing or after-sales maintenance, thereby improving the robot's environmental perception accuracy and reducing calibration labor costs and procedures.

[0010] To achieve the above objectives, the technical solution of the present invention is as follows:

[0011] An automatic calibration method for extrinsic parameters of multiple cameras in a robot includes the following steps:

[0012] Several reference obstacles are set up in the calibration field to construct a global reference obstacle grid map, wherein the edge lines of the reference obstacles coincide with the grid lines in the global reference obstacle grid map;

[0013] The robot acquires images captured by cameras at different positions and performs preprocessing to obtain local reference obstacle grid maps at different positions; the robot's coordinate center coincides with any grid center in the global reference obstacle grid map, and the robot's horizontal or vertical axis is parallel to the horizontal or vertical axis of the global reference obstacle grid map.

[0014] The captured images are input into a deep learning instance segmentation model to classify obstacles and non-obstacles at the pixel level, resulting in a binary image. The binary image is then combined with the camera intrinsic parameters corresponding to the captured images to obtain the projection of obstacle pixels onto the ground.

[0015] Make the rotation matrix drift in Within, and setting the angle step size to m, and / or the drift of the translation matrix in The system searches with a displacement step size of n to generate several candidate extrinsic drift matrices. Based on each candidate extrinsic drift matrix and the projection of obstacle pixels onto the ground, several corresponding BEV grid maps are obtained. The system compares the local reference obstacle grid map with the several BEV grid maps to minimize the error and obtain the true extrinsic drift. The true extrinsic drift is added to the known nominal extrinsic parameters to obtain the calibrated camera extrinsic parameters.

[0016] Preferably, the plurality of reference obstacles are regular hexahedrons with different lengths and widths but uniform heights.

[0017] Preferably, the preprocessing includes image normalization, cropping, and scaling.

[0018] Preferably, It is ±2cm.

[0019] Preferably, m is any value between 0.01 degrees and 0.1 degrees, and n is any value between 0.01 cm and 0.05 cm.

[0020] Preferably, the loss function L used for error calculation is as follows:

[0021]

[0022] In the formula, N represents the total number of pixels in the BEV raster map. The pixel values ​​are obtained based on robot vision calculations. The values ​​of the known base map pixels.

[0023] Based on the above technical solution, the beneficial effects of the present invention are:

[0024] 1) This invention utilizes accurate occupancy grid maps and camera perception results for automatic alignment and calibration. Through standard calibration environment, deep learning segmentation, and BEV projection, the operation is simple and requires minimal manual intervention, reducing calibration labor costs.

[0025] 2) This invention utilizes a local reference obstacle grid map as a benchmark to minimize the error of candidate extrinsic parameters, significantly reducing the error caused by camera installation deviation. Its accuracy is unaffected by environmental changes (such as changes in lighting), ensuring that the calibrated extrinsic parameters guarantee a high degree of consistency between obstacle projection and the real environment, improving the robot's environmental perception and path planning accuracy, and exhibiting strong robustness.

[0026] 3) The calibrated camera extrinsic parameters of this invention can significantly improve the accuracy and reliability of robot environmental perception, thereby providing reliable environmental perception information for robot path planning and motion control. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of a method for automatic calibration of extrinsic parameters of a robot with multiple cameras in one embodiment;

[0028] Figure 2 This is a schematic diagram of the robot dog in one embodiment;

[0029] Figure 3 This is a schematic diagram of the placement of the reference obstacle in one embodiment;

[0030] Figure 4 This is an example of the placement of a global baseline obstacle grid map and robot data acquisition;

[0031] Figure 5 This is a schematic diagram of the camera extrinsic parameter calibration process in one embodiment. Detailed Implementation

[0032] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0033] like Figure 1 As shown, this embodiment provides an automatic calibration method for the extrinsic parameters of multiple cameras in a robot. Taking a robot dog as an example, as follows... Figure 2 As shown, four cameras are installed, including a front-facing camera, two side-facing cameras, and a rear-facing camera. The purpose is to perceive the robot's surrounding environment (the number of cameras depends on the robot's functional requirements; no additional camera sensors are needed. The purpose of this solution is to calibrate the extrinsic parameters of each installed camera). The specific steps include:

[0034] Camera extrinsic parameters refer to the relationship between the camera's position and orientation based on the robot's coordinate system, as shown in the following mathematical expression P. r =TP c Expanded as:

[0035]

[0036] Among them, P r The coordinates are relative to the robot's coordinate system. Indicates; P c The coordinates are relative to the camera coordinate system. The representation is: T represents the rotation and translation matrices, i.e., the extrinsic parameters of the camera; it consists of two parts, a 3x3 rotation matrix. and a 3x1 translation matrix .

[0037] In practical use, since the robot's camera has a nominal installation position and orientation, there is a corresponding nominal extrinsic parameter matrix T. nominal , which are known terms. Due to deviations in mechanical installation and the influence of subsequent robot vibrations, the actual camera extrinsic parameters have a small deviation in the nominal extrinsic parameter matrix, namely...

[0038]

[0039] Therefore, it is only necessary to solve for the deviation T of the extrinsic parameter matrix. offset Then you can know the true extrinsic parameter matrix.

[0040] Alternatively, they can be expressed as follows:

[0041]

[0042] in, For the true rotation matrix, Let be the nominal rotation matrix. This represents the offset between the nominal rotation matrix and the actual rotation matrix.

[0043] Step 100: Set up several reference obstacles in the calibration site and construct a global reference obstacle grid map, wherein the edge lines of the reference obstacles coincide with the grid lines in the global reference obstacle grid map.

[0044] In this embodiment, firstly, a calibration site is selected, ensuring sufficient lighting and a flat surface. The specific size of the site can be adjusted according to the actual size of the robot. This solution uses a 5m x 5m room as an example, and the calibration site is divided into a 50x50 grid map, that is, each grid is 10cm in size. Figure 3 The ground is shown in the image.

[0045] To facilitate implementation and calculation, choose obstacles of different sizes. Use cardboard boxes with varying lengths and widths but uniform heights. The height of the boxes should be relatively low (e.g., 10 cm or 20 cm). Different lengths and widths increase the diversity of the obstacle map, while a uniform low height avoids obstructing the robot's view. Furthermore, to simplify calculations, it is recommended that the length and width of the cardboard boxes be multiples of the grid dimensions (e.g., 30 cm or 50 cm in this example). When placing the boxes, ensure that the edges of the boxes align perfectly with the grid lines. Figure 3 The placement of the baseline obstacles is shown.

[0046] By placing reference obstacles (such as cardboard boxes) in a fixed calibration scenario and acquiring images at different positions and orientations, the reliance on calibration boards or external equipment is avoided. At the same time, the requirements for the reading site and obstacles are not high, making it easy to implement in practice.

[0047] Define the origin and coordinate system of the map. By using the selected field grid and obstacle size and position, a baseline obstacle map can be obtained directly. In order to improve the accuracy of camera extrinsic calibration, it is recommended to further increase the resolution of the grid map. For example, if the field is divided into 50x50 grids (grid size is 10cmx10cm), its resolution can be increased to a 500x500 grid map (grid size is 1cmx1cm). Figure 4 This is an example of a global baseline obstacle grid map.

[0048] Step 200: Obtain images captured by cameras at different positions of the robot and perform preprocessing to obtain local reference obstacle grid maps at different positions; the coordinate center of the robot coincides with any grid center in the global reference obstacle grid map, and the horizontal or vertical axis of the robot is parallel to the horizontal or vertical axis of the global reference obstacle grid map.

[0049] In this embodiment, as Figure 4 As shown, the robot is placed at different locations within the calibrated area. To facilitate the calculation of the reference obstacle grid map relative to the robot's coordinate system, it is recommended that the robot's coordinate center coincide with a specific grid cell in a 500x500 grid map, and that the robot remain parallel to the horizontal or vertical axis of the global map. For each location, since the global reference obstacle grid map and the robot's precise position are known, each camera can obtain a corresponding reference obstacle grid map, i.e., a local reference obstacle grid map. Assuming there are M placement locations and N cameras, with each camera having a reference grid map size of RxR relative to the robot, the reference map data can be obtained as a four-dimensional matrix MAP. real Its size is .

[0050] Simultaneously, for each placement position, each camera needs to capture an image from its current perspective (each camera only needs to capture one clear photo at each position). Assuming the camera's resolution is height H and width W, the image data obtained through the camera is a five-dimensional matrix I. real Its size is Where C represents the number of channels in the color image, typically 3. Data acquisition from the camera can be obtained through a computing chip installed on the robot and stored via a storage medium, such as... Figure 2 As shown.

[0051] Step 300: Input the captured image into a deep learning instance segmentation model to perform pixel-level classification between obstacles and non-obstacles to obtain a binary image. The binary image is combined with the camera intrinsic parameters corresponding to the captured image to obtain the projection of obstacle pixels on the ground.

[0052] In this embodiment, a deep learning instance segmentation model is used to classify the captured images at the pixel level. Then, based on the classification category, it is determined whether the image represents an obstacle. Current deep learning instance segmentation models are very mature, and the obstacles and calibration scene designed in this scheme are very simple, with no obvious interfering factors, making them particularly suitable for segmentation. Therefore, the pixel classification obtained through the deep learning instance segmentation model can be considered accurate. Figure 5 As shown, the specific operation process is as follows:

[0053] Images captured by the robot's camera undergo preprocessing. The specific preprocessing method depends on the requirements of the selected deep learning instance segmentation model. Common image preprocessing operations include image normalization, cropping, and scaling. The preprocessed data is then fed into a deep learning inference model for inference, yielding inferred data. The inferred data undergoes post-processing, the specific method of which also depends on the requirements of the selected deep learning instance segmentation model. Post-processing produces the image segmentation result, which is a pixel-level classification. This pixel-level classification result is further subdivided, with pixels belonging to the obstacle category marked as 1 and those not belonging as 0, thus obtaining a binary image with the same resolution as the original image.

[0054] The binary image is combined with the corresponding camera intrinsic parameters to obtain its projection onto the ground. The calculation method is as follows:

[0055] For the obstacle pixel (u, v), the ray in the camera coordinate system is:

[0056]

[0057] in, It is along the scale of the ray. This is the intrinsic parameter matrix of the camera.

[0058] Assuming ground height Then the scale along the ray can be determined. The formula is as follows:

[0059]

[0060] Then Substitute and you can get and Where h is the known height of the camera. This represents the component of the ray along the Z-axis of the camera coordinate system (vertical direction).

[0061] The acquired images are segmented to separate obstacles from non-obstacles, generating binary obstacle masks to ensure accuracy when projected onto the BEV map. Furthermore, the very simple scene from which the images were acquired reduces the reliance on deep learning capabilities.

[0062] Step 400, causing the rotation matrix to drift in Within, and setting the angle step size to m, and / or the drift of the translation matrix in The system searches with a displacement step size of n to generate several candidate extrinsic drift matrices. Based on each candidate extrinsic drift matrix and the projection of obstacle pixels onto the ground, several corresponding BEV grid maps are obtained. The system compares the local reference obstacle grid map with the several BEV grid maps to minimize the error and obtain the true extrinsic drift. The true extrinsic drift is added to the known nominal extrinsic parameters to obtain the calibrated camera extrinsic parameters.

[0063] In this embodiment, as Figure 5 As shown, based on the projection of the obstacle pixels onto the ground, combined with the nominal extrinsic parameters of the corresponding camera and the local reference obstacle grid map (obtained through the global reference obstacle grid map and the position and orientation of the corresponding robot), the calibrated camera extrinsic parameters are obtained by substituting them into the extrinsic parameter solver. The detailed design of the solver is described below.

[0064] Real camera external parameters The nominal extrinsic parameters can be considered as the sum of the extrinsic parameter drifts caused by various reasons. Since the nominal extrinsic parameters are known, we only need to solve for the extrinsic parameter drift to obtain the true extrinsic parameters of the robot's camera. There is also a priori assumption that the difference between the true and nominal extrinsic parameters should be relatively small. Therefore, we can assume that the drift of the rotation matrix... for Inside (e.g.) The drift of the translation matrix is ​​caused by installation errors. for Caused by internal factors (e.g., ±2cm).

[0065] Based on the assumed maximum installation deviation and The corresponding step size can be set, which generates the installation error value, and thus the corresponding candidate extrinsic parameter drift matrix. Note that a smaller step size theoretically results in higher accuracy, but also increases the computational load. However, an excessively large step size will severely affect the calibration effect. Therefore, it is generally recommended to choose a small but not too small value, where m can be any value between 0.01 degrees and 0.1 degrees, such as 0.05 degrees; and the displacement step size n can be any value between 0.01 cm and 0.05 cm, such as 0.01 cm. This generates a large number of candidate extrinsic parameter drift matrices only within a certain range, avoiding full-space search and significantly improving the computational efficiency of extrinsic parameter optimization.

[0066] After obtaining several candidate extrinsic parameter drift matrices T offset Then, the candidate true extrinsic parameter matrix can be obtained by combining the nominal extrinsic parameter matrix. For each candidate true extrinsic parameter matrix, the projection of obstacle pixels on the ground can be transformed into robot coordinates using the following formula.

[0067]

[0068] Then, this new projection is discretized to obtain the corresponding BEV grid map. The BEV grid has the robot center as the origin, a resolution of r meters / pixel, and a size of... .

[0069]

[0070]

[0071] in, Represents the horizontal index of the BEV map. This represents the vertical index of the BEV map, where r is the resolution of the BEV map (in meters per pixel).

[0072] After obtaining the BEV raster map corresponding to each candidate extrinsic parameter drift matrix, the error is calculated by comparing it with the local baseline obstacle raster map. The loss function is defined as:

[0073]

[0074] Where N is the total number of pixels in the BEV map. The pixel values ​​are obtained based on robot vision calculations. The values ​​of the known base map pixels.

[0075] The minimum error obtained through the loss function corresponds to and This can be considered as the true deviation obtained from the solution, thereby calculating the true extrinsic parameter drift T. offset .

[0076] nominal external parameter T nominalThe actual extrinsic parameter drift T obtained from calibration offset The calibrated image extrinsing parameters are superimposed to obtain the calibrated image extrinsic parameters, achieving fully automated, high-precision, and environmentally adaptive extrinsic parameter calibration. The entire process involves minimal and simple manual intervention and requires no experience from the operator.

[0077] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0078] The above are merely preferred embodiments of the present application and are not intended to limit the embodiments of the present application. For those skilled in the art, the embodiments of the present application can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of the present application should be included within the protection scope of the embodiments of the present application.

Claims

1. A method for automatic calibration of extrinsic parameters of multiple cameras in a robot, characterized in that, Includes the following steps: Several reference obstacles are set up in the calibration field to construct a global reference obstacle grid map, wherein the edge lines of the reference obstacles coincide with the grid lines in the global reference obstacle grid map; The robot acquires images captured by cameras at different positions and performs preprocessing to obtain local reference obstacle grid maps at different positions; the robot's coordinate center coincides with any grid center in the global reference obstacle grid map, and the robot's horizontal or vertical axis is parallel to the horizontal or vertical axis of the global reference obstacle grid map. The captured images are input into a deep learning instance segmentation model to classify obstacles and non-obstacles at the pixel level, resulting in a binary image. The binary image is then combined with the camera intrinsic parameters corresponding to the captured images to obtain the projection of obstacle pixels onto the ground. Make the rotation matrix drift in Within, and setting the angle step size to m, and / or the drift of the translation matrix in The system searches with a displacement step size of n to generate several candidate extrinsic drift matrices. Based on each candidate extrinsic drift matrix and the projection of obstacle pixels onto the ground, several corresponding BEV grid maps are obtained. The system compares the local reference obstacle grid map with the several BEV grid maps to minimize the error and obtain the true extrinsic drift. The true extrinsic drift is added to the known nominal extrinsic parameters to obtain the calibrated camera extrinsic parameters.

2. The automatic calibration method for extrinsic parameters of a robot with multiple cameras according to claim 1, characterized in that, The aforementioned reference obstacles are regular hexahedrons with different lengths and widths but uniform heights.

3. The automatic calibration method for extrinsic parameters of a robot with multiple cameras according to claim 1, characterized in that, The preprocessing includes image normalization, cropping, and scaling.

4. The automatic calibration method for extrinsic parameters of a robot with multiple cameras according to claim 1, characterized in that, It is ±2cm.

5. The automatic calibration method for extrinsic parameters of a robot with multiple cameras according to claim 1, characterized in that, m can be any value between 0.01 degrees and 0.1 degrees, and n can be any value between 0.01 cm and 0.05 cm.

6. The automatic calibration method for extrinsic parameters of a robot with multiple cameras according to claim 1, characterized in that, The loss function L used for error calculation is as follows: ； In the formula, N represents the total number of pixels in the BEV raster map. The pixel values ​​are obtained based on robot vision calculations. The values ​​of the known base map pixels.

Images