Three-dimensional system ground calibration method, medium, equipment and product

Abstract

The invention relates to the technical field of visual computing, and particularly provides a three-dimensional system ground calibration method, a medium, equipment and a product, and the method can comprise the steps: obtaining an initial calibration parameter from a main camera to a to-be-calibrated ground coordinate system; wherein the initial calibration parameters comprise an initial rotation matrix and an initial translation vector; the main camera is randomly selected from a calibrated multi-camera stereoscopic vision system, and the main camera corresponds to a main camera coordinate system; determining calibration coordinate positions of at least two pairs of calibration points in the main camera coordinate system; wherein the at least two pairs of calibration points are pixel points on images acquired at the same target position at the same moment by at least two cameras with different visual angles in the multi-camera stereoscopic vision system; and optimizing the initial calibration parameter based on the calibration coordinate position to obtain a ground calibration parameter. Some embodiments of the application can improve the precision of ground calibration.

Description

Technical Field

[0001] This application relates to the field of visual computing technology, and more specifically, to a method, medium, device, and product for ground calibration of a stereo system. Background Technology

[0002] Multi-camera stereo vision systems reconstruct 3D positional relationships within a scene by acquiring images from different perspectives and performing triangulation calculations. To better represent the positional relationship between the camera and the subject relative to a fixed reference point, or to more conveniently describe the pose changes of the subject in the spatial environment, an additional world coordinate system is usually specified and linked to the stereo vision system. In motion capture, this additional external world coordinate system typically refers to the ground coordinate system, and the process of linking the ground coordinate system to the stereo vision system is called ground calibration. Currently, ground calibration typically involves placing an L-shaped calibration object or calibration plate on the ground as the ground calibration detection object, either automatically or manually, selecting specific points on the calibration object in the image, and then performing mathematical calculations based on camera imaging theory. However, due to the limited size of the calibration object and the possibility of uneven ground surfaces, the calibration plane completed along the inclined calibration object will also exhibit a certain degree of inclination, failing to perfectly match the real ground, thus leading to deviations in the acquired motion capture data.

[0003] Therefore, how to provide a more accurate ground calibration method has become an urgent technical problem to be solved. Summary of the Invention

[0004] The purpose of some embodiments of this application is to provide a method, medium, device and product for ground calibration of a three-dimensional system. The technical solutions of the embodiments of this application can improve the accuracy of ground calibration and reduce calibration error.

[0005] In a first aspect, some embodiments of this application provide a method for ground calibration of a stereo system, comprising: obtaining initial calibration parameters from a main camera to a ground coordinate system to be calibrated; wherein the initial calibration parameters include an initial rotation matrix and an initial translation vector; the main camera is randomly selected from a pre-calibrated multi-camera stereo vision system, and the main camera corresponds to a main camera coordinate system; determining the calibration coordinate positions of at least two pairs of calibration points in the main camera coordinate system; wherein the at least two pairs of calibration points are pixels on images acquired simultaneously by at least two cameras with different viewpoints in the multi-camera stereo vision system at the same target location; and optimizing the initial calibration parameters based on the calibration coordinate positions to obtain ground calibration parameters.

[0006] Some embodiments of this application first obtain the initial calibration parameters from the main camera to the ground coordinate system to be calibrated, and then optimize the initial calibration parameters using the calibration coordinate positions corresponding to at least two pairs of calibration points collected by cameras from different viewpoints to obtain the ground calibration parameters. The scheme of optimizing the initial calibration parameters by calibration coordinate positions in the embodiments of this application can improve the accuracy of ground calibration and make the ground calibration coordinate system fit the real ground. At the same time, this scheme does not require manual intervention and improves the efficiency of ground calibration.

[0007] In some embodiments, before obtaining the initial calibration parameters from the main camera to the ground coordinate system to be calibrated, the method further includes: determining the internal calibration parameters of each camera in the multi-camera stereo vision system by calibrating each camera, and determining the external calibration parameters between cameras in the multi-camera stereo vision system; wherein the internal calibration parameters include an intrinsic parameter matrix and distortion coefficients; and the external calibration parameters include a camera rotation matrix and a camera translation matrix between any camera in the multi-camera stereo vision system and other cameras besides the aforementioned camera.

[0008] Some embodiments of this application establish a stereo vision system by calibrating the intrinsic parameters of the cameras and the extrinsic parameters between the cameras in a multi-camera stereo vision system. In particular, it realizes the calculation of the same calibration point from 2D coordinates to 3D coordinates in the main camera coordinate system under different viewpoints, providing technical support and data input for subsequent ground calibration.

[0009] In some embodiments, optimizing the initial calibration parameters based on the calibration coordinate position to obtain ground calibration parameters includes: transforming the calibration coordinate position in the main camera coordinate system to the ground coordinate system to be calibrated according to the initial calibration parameters to obtain 3D reference coordinate values ​​in the ground coordinate system to be calibrated; and optimizing the initial calibration parameters based on the 3D reference coordinate values ​​and the actual position of the target position in the ground coordinate system to be calibrated to obtain the ground calibration parameters.

[0010] Some embodiments of this application convert the calibration coordinate position into a 3D reference coordinate value through initial calibration parameters. By combining the initial calibration parameters with the actual position of the target position, the initial calibration parameters can be optimized to obtain ground calibration parameters. This achieves accurate acquisition of ground calibration parameters, truly conforms to the real ground, and reduces the influence of other external factors.

[0011] In some embodiments, optimizing the initial calibration parameters based on the 3D reference coordinate values ​​and the actual position of the target location in the ground coordinate system to be calibrated to obtain the ground calibration parameters includes: calculating the ground distance residual between the height value in the Z-axis direction of the 3D reference coordinate values ​​and the ground distance residual in the actual position; accumulating the ground distance residual values ​​of all calibration points, and optimizing the initial rotation matrix with the ground distance residual value as the optimization target set as the optimization value to obtain the optimized rotation matrix in the ground calibration parameters.

[0012] Some embodiments of this application obtain the height value on the 3D reference coordinates, calculate the ground distance residual value between the actual ground distance of the calibration point and the ground distance, and use it as a set value for target optimization to obtain an optimized rotation matrix. This can make the ground calibration result truly fit the ground, improve the accuracy of ground calibration, and reduce calibration error.

[0013] In some embodiments, the ground calibration parameters include ground yaw angle parameters. The optimization of the initial calibration parameters based on the 3D reference coordinate values ​​and the actual position of the target location in the ground coordinate system to be calibrated, to obtain the ground calibration parameters, includes: calculating the horizontal residual value between the 3D reference coordinate values ​​and the actual position in the horizontal direction; wherein the horizontal direction is the X-axis direction or the Y-axis direction; and optimizing the ground yaw angle parameters based on the cumulative result of all horizontal residual values, using the horizontal residual value as a set optimization target, to obtain optimized ground yaw angle parameters.

[0014] Some embodiments of this application optimize the horizontal residual value in the horizontal direction to obtain optimized ground yaw angle parameters, thereby achieving accurate calibration in the horizontal direction.

[0015] In some embodiments, the ground calibration parameters include all parameters of the ground coordinate system to be calibrated, including pitch angle, roll angle, and heading angle. The optimization of the initial calibration parameters based on the 3D reference coordinate values ​​and the actual position of the target location in the ground coordinate system to be calibrated, to obtain the ground calibration parameters, includes: calculating the distance-to-ground residual between the altitude value in the Z-axis direction of the 3D reference coordinate values ​​and the distance-to-ground height in the actual position; calculating the horizontal residual between the horizontal direction of the 3D reference coordinate values ​​and the horizontal direction of the actual position; wherein the horizontal direction is the X-axis direction or the Y-axis direction; and optimizing the ground coordinate system to be calibrated based on the sum of all distance-to-ground residuals and the sum of all horizontal residuals, using the summed result as the optimization target, to obtain the optimized ground coordinate system.

[0016] Some embodiments of this application optimize the vertical and horizontal distance residuals and the horizontal residuals to obtain an optimized ground coordinate system, thereby achieving accurate calibration of the entire ground coordinate system.

[0017] In some embodiments, determining the calibration coordinate positions of at least two pairs of calibration points in the main camera coordinate system includes: calculating the calibration coordinate positions of the at least two pairs of calibration points in the main camera coordinate system using triangulation calculation and optimization algorithms.

[0018] Some embodiments of this application achieve accurate conversion from 2D image calibration points to 3D calibration coordinate positions through triangulation calculations and optimization algorithms.

[0019] Secondly, some embodiments of this application provide an apparatus for ground calibration of a stereo system, comprising: an initial calibration module for acquiring initial calibration parameters from a main camera to a ground coordinate system to be calibrated; wherein the initial calibration parameters include an initial rotation matrix and an initial translation vector; the main camera is randomly selected from a pre-calibrated multi-camera stereo vision system, and the main camera corresponds to a main camera coordinate system; a coordinate transformation module for determining the calibration coordinate positions of at least two pairs of calibration points in the main camera coordinate system; wherein the at least two pairs of calibration points are pixels on images acquired simultaneously by at least two cameras with different viewpoints in the multi-camera stereo vision system at the same target location; and an optimization module for optimizing the initial calibration parameters based on the calibration coordinate positions to obtain ground calibration parameters.

[0020] Thirdly, some embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method described in any embodiment of the first aspect.

[0021] Fourthly, some embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the method as described in any embodiment of the first aspect.

[0022] Fifthly, some embodiments of this application provide a computer program product, the computer program product including a computer program, wherein the computer program, when executed by a processor, can implement the method described in any embodiment of the first aspect. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of some embodiments of this application, the accompanying drawings used in some embodiments of this application will be briefly described below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 Schematic diagram of ground calibration of a three-dimensional system provided for some embodiments of this application; Figure 2 One of the flowcharts for a method of ground calibration of a three-dimensional system provided in some embodiments of this application; Figure 3 One example diagram showing a tilted ground grid as part of some embodiments of this application; Figure 4 Example diagram two showing a tilted ground grid for some embodiments of this application; Figure 5 One of the example diagrams showing the optimized ground coordinate leveled ground grid provided for some embodiments of this application; Figure 6 Example diagram two of the optimized ground coordinate leveled ground grid provided for some embodiments of this application; Figure 7 A second flowchart illustrating a method for ground calibration of a three-dimensional system provided for some embodiments of this application; Figure 8 Block diagrams of apparatus for three-dimensional system ground calibration provided for some embodiments of this application; Figure 9 A schematic diagram of an electronic device provided for some embodiments of this application. Detailed Implementation

[0025] The technical solutions of some embodiments of this application will now be described with reference to the accompanying drawings.

[0026] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0027] In related technologies, the relationship between the three-dimensional geometric position of a point on the surface of an object in space and its corresponding point in an image requires establishing a geometric model of camera imaging. These geometric model parameters are the camera parameters, and the process of solving for these parameters is called camera calibration. Camera calibration in multi-camera stereo vision systems is further divided into intrinsic parameter calibration and extrinsic parameter calibration. Intrinsic parameters refer to the internal parameters describing the imaging characteristics of a single camera, including focal length, principal point position, and lens distortion. These parameters determine how the three-dimensional world is projected onto a two-dimensional image. Extrinsic parameters describe the positional relationship between the camera and another camera or with a specific reference frame. They consist of a rotation matrix R and a translation vector T, also known as pose. The significance of extrinsic parameters is to correlate images from different viewpoints for tasks such as 3D reconstruction and motion estimation. Since the camera and the object being photographed can be placed at any position in the environment, it is necessary to represent the positions of the camera and the object. This requires establishing a coordinate system in the environment, called the world coordinate system (or ground coordinate system). Corresponding to this is the camera coordinate system. The origin of the camera coordinate system is located at the optical center of the lens. The x and y axes are parallel to the two sides of the camera plane, respectively, and the z-axis is the lens optical axis, perpendicular to the image plane. The camera coordinate system and the world coordinate system can be transformed to each other using specific rotation matrices and translation vectors. In the field of motion capture, the world coordinate system is usually specifically the ground coordinate system. The process of linking the ground coordinate system with the established stereo vision is called ground calibration. The task of ground calibration is to determine the rotation matrix and translation vector from the ground coordinate system to the master camera coordinate system.

[0028] Common ground calibration tools include L-shaped calibration frames and planar calibration boards with checkerboard patterns. The origin and two axes of the desired world coordinate system are located on these calibration objects. By detecting the pixel coordinates of specified calibration points on the calibration objects in the camera image, and combining this with the known true positions of these calibration points in the world coordinate system, the rotation matrix and translation vector from the desired world coordinate system to the respective camera coordinate system can be obtained, referred to as the ground calibration result. Typically, after ground calibration is completed, a grid representing the ground can be drawn on the camera image, and whether it is parallel and closely aligned with the real ground is used to determine whether the ground calibration was successful. However, due to the influence of image resolution and image sharpness, there is a certain error in detecting the calibration points of the calibration objects from the image. The placement of the calibration objects may also be slightly tilted due to uneven ground, causing the calibration plane completed along the tilted calibration objects to also have a certain tilt. Since the size of the calibration objects is limited and cannot be laid flat across the entire ground, small errors can be amplified at the far end due to the seesaw effect, causing the calibrated plane to tilt and fail to closely align with the real ground. Due to the influence of ground calibration, the motion-captured human body data may show a person moving close to the ground, but the reconstructed digital person's feet may either be stuck in the calibrated ground or suspended in the air on the calibrated plane. Therefore, the results of ground calibration need further adjustment and optimization, especially in large-scale scenarios.

[0029] In view of this, some embodiments of this application provide a method for ground calibration of a stereo system. This method first obtains initial calibration parameters between the main camera and the ground coordinate system to be calibrated; then, by selecting multiple pairs of calibration points corresponding to the same point in images from different camera perspectives, the 3D position of the calibration points in the main camera coordinate system is first obtained through triangulation calculation; then, these 3D positions are transformed to the world coordinate system using the initial calibration parameters to obtain the calibration coordinate positions; and the ground calibration results are adjusted by iteratively optimizing the projection height in the world coordinate system, ensuring consistency between the actual ground and the calibrated ground grid. The embodiments of this application can achieve efficient, automatic, and accurate ground calibration.

[0030] The following is in conjunction with the appendix Figure 1 The following are exemplary schematic diagrams illustrating ground calibration of a three-dimensional system provided by some embodiments of this application.

[0031] like Figure 1 In the schematic diagram of the stereoscopic system ground calibration shown, different numbers of cameras 100 are set up around the perimeter of the site, pointing towards the capture area 200. An L-shaped ground calibration object is placed in the center of the capture area. The different numbers of cameras constitute a multi-camera stereoscopic vision system. The number of cameras is at least two, but this embodiment does not specify a particular number. Specifically, the multi-camera stereoscopic vision system is used to calibrate the ground of the capture area 200.

[0032] The following is in conjunction with the appendix Figure 2 The implementation process of ground calibration of a stereo system provided in some embodiments of this application is illustrated by way of example. It should be noted that the implementation process of ground calibration of the stereo system can be executed by the processor inside the multi-camera stereo vision system, or by a terminal device (e.g., a mobile terminal or a non-portable computer terminal) associated with the multi-camera stereo vision system. The embodiments of this application do not make specific limitations here.

[0033] Please see the appendix Figure 2 , Figure 2 A flowchart of a method for ground calibration of a stereo system is provided for some embodiments of this application. The method for ground calibration of a stereo system may include: S210, obtain the initial calibration parameters from the main camera to the ground coordinate system to be calibrated; wherein, the initial calibration parameters include the initial rotation matrix and the initial translation vector; the main camera is randomly selected from the calibrated multi-camera stereo vision system, and the main camera corresponds to a main camera coordinate system.

[0034] For example, in a specific example of this application, the main camera can be any one of the cameras in a multi-camera stereo vision system. After selecting the main camera, the initial rotation matrix Rg and the initial translation vector Tg from the main camera to the ground coordinate system to be calibrated are first calculated. Here, the ground coordinate system to be calibrated refers to a world coordinate system whose origin lies on the plane representing the ground, and the plane formed by the x and y axes of the world coordinate system is parallel to the plane representing the ground. Rg and Tg from the main camera to the ground coordinate system to be calibrated refer to the world coordinate system coinciding with the main camera coordinate system after undergoing a rotation transformation of Rg followed by a translation transformation of Tg. The relationship can be expressed as: Po = Rg Pw+Tg, where Po is the coordinate point in the master camera coordinate system and Pw is the coordinate point in the world coordinate system. This formula expresses how the coordinates of a point in the world coordinate system, after undergoing a rotation transformation (Rg) and a translation transformation (Tg), coincide with the coordinates of the same point in the master camera coordinate system. Rg and Tg can be calculated using multiple 2D detection points on calibration objects combined with the PnP method; OpenCV provides the `solvePnP` function.

[0035] In some embodiments of this application, before performing S210, the method for ground calibration of the stereo system further includes: determining the internal calibration parameters of each camera in the multi-camera stereo vision system by calibrating each camera, and determining the external calibration parameters between cameras in the multi-camera stereo vision system; wherein the internal calibration parameters include an intrinsic parameter matrix and distortion coefficients; and the external calibration parameters include a camera rotation matrix and a camera translation matrix between any camera in the multi-camera stereo vision system and other cameras besides the aforementioned camera.

[0036] For example, in the specific example of this application, the multi-camera stereo vision system first needs to be calibrated. The calibration method for the multi-camera stereo vision system can be the Zhang Zhengyou calibration method. The intrinsic and extrinsic parameters of each camera are calculated by sampling the calibration plate at different positions in the captured space or capturing the movement of the calibration rod. The camera intrinsic parameters include the intrinsic parameter matrix K composed of the camera's focal length and principal point position, the distortion coefficients D of the camera image, and the camera rotation matrix R and camera translation matrix T from any camera to the other cameras.

[0037] For example, in a binocular system (as a specific example of a multi-camera stereo vision system), the left camera is generally regarded as the master camera. After determining the master camera, the rotation matrix Ri and translation matrix Ti from the master camera to the other cameras can be obtained sequentially based on the extrinsic calibration results (i.e., external calibration parameters). The relationship between the two camera coordinate systems pointing to the same coordinate point can be expressed as the following equation: Pi = Ri Po+Ti, where Pi is a point in the i-th camera coordinate system and Po is the coordinate of that point in the main camera coordinate system.

[0038] The above calculations can complete the initial calibration of the ground coordinate system to be calibrated. However, when the initial calibration result deviates from the actual ground, it is necessary to adjust the rotation direction of the ground coordinate system to make it parallel to the actual ground. Therefore, the following operations are required to obtain the optimized ground calibration parameters.

[0039] For example, such as Figure 3 and Figure 4 The image shown is an example of a tilted ground grid display of the binocular system's ground calibration results. Figure 3 This is the view from the left camera (main camera). Figure 4 This is the view from the right camera. In this example system, an ArUco code pattern calibration board is used to automatically complete the initial ground calibration. It is evident that because the calibration board is not parallel to the ground when placed, but rather slightly tilted, the visualized ground grid is not parallel to the actual ground. This is evident from… Figure 3 The stacking of cardboard boxes on the ground on the right side of the main camera view is particularly noticeable. Therefore, the ground calibration results need to be adjusted before application.

[0040] S220, determine the calibration coordinate positions of at least two pairs of calibration points in the coordinate system of the main camera; wherein, the at least two pairs of calibration points are pixels on the images of the same target position acquired by at least two cameras with different viewpoints in the multi-camera stereo vision system at the same time.

[0041] For example, in a specific example of this application, N pairs of corresponding points (as a specific example of calibration points) are randomly selected from images from at least two cameras with different viewpoints as data to correct the ground calibration results (i.e., the initial calibration parameters). The corresponding points from different viewpoint cameras refer to the pixel coordinates in the images from different cameras at the same time, all pointing to the same target location in real 3D space, and the height of this target location above the ground is known. Furthermore, the number of N pairs of corresponding points should be greater than or equal to 2, and each pair should contain pixel coordinates from images from at least two cameras. The 3D coordinate positions of the selected N pairs of corresponding points in the main camera coordinate system are calculated (as a specific example of calibration coordinate positions).

[0042] For example, in Figure 3 In the example of the binocular system, five pairs of corresponding points (i.e., N=5) were manually selected from the left and right camera views, sequentially from the three legs of the chair to the corners of the two cardboard boxes. These five pairs of corresponding points all contact the actual ground, and therefore their height above the ground is considered h=0mm. Besides manual selection, calibration objects can also be placed at different locations on the ground to complete the automatic detection of corresponding points. Without loss of generality, a multi-camera stereo vision system consisting of four cameras can be used as an example to illustrate how to randomly select N pairs of corresponding points from images from at least two cameras with different viewpoints. Specifically, there are 6 combinations for selecting two cameras out of the four, 4 combinations for selecting three cameras, and 1 combination for selecting four cameras, for a total of 11 camera combinations. Ki points are selected from each camera combination; these Ki points are visible in all camera images of that combination. The ground calibration and leveling operation can be completed in the following steps as long as the sum N of the points selected from these 11 combinations is greater than or equal to 2.

[0043] In some embodiments of this application, S220 may include: calculating the calibration coordinate positions of the at least two pairs of calibration points in the main camera coordinate system using triangulation calculation and optimization algorithms.

[0044] For example, in the specific example of this application, the corresponding points selected from two viewpoint cameras can be directly calculated in 3D coordinates using the triangulation calculation formula. The triangulation calculation requires the use of intrinsic and extrinsic parameters in the camera calibration. The corresponding points selected from more than two viewpoints can be calculated by combining pairwise triangulation calculation with optimization methods. The simplest optimization method is to obtain the pairwise triangulation results and then take the average value to calculate the 3D coordinates of the corresponding points in the main camera coordinate system.

[0045] S230, Based on the calibration coordinate position, the initial calibration parameters are optimized to obtain the ground calibration parameters.

[0046] For example, in a specific example of this application, the ground calibration parameters of the main camera relative to the ground calibration plane corresponding to the ground coordinate system to be calibrated are optimized by using multiple 3D coordinate positions under the main camera (as a specific example of calibration coordinate positions).

[0047] In some embodiments of this application, S230 may include: transforming the calibration coordinate position under the main camera coordinates to the ground coordinate system to be calibrated according to the initial calibration parameters, to obtain the 3D reference coordinate value under the ground coordinate system to be calibrated; optimizing the initial calibration parameters based on the 3D reference coordinate value and the actual position of the target position in the ground coordinate system to be calibrated, to obtain the ground calibration parameters.

[0048] For example, in a specific example of this application, based on Rg and Tg in the initial calibration parameters mentioned above, the 3D coordinate position p1 is transformed to the ground coordinate system to be calibrated, resulting in the 3D reference coordinate value p2. It can be understood that p2 can be a set or general term containing multiple ground coordinate points. Subsequently, based on p2 and the actual position of the target location in the ground coordinate system to be calibrated, the initial calibration result is optimized to obtain the ground calibration parameters.

[0049] In some embodiments of this application, S230 may include: calculating the ground distance residual between the height value in the Z-axis direction of the 3D reference coordinates and the ground distance residual in the actual position; accumulating the ground distance residual values ​​of all calibration points, and optimizing the initial rotation matrix with the ground distance residual value as the optimization target set in the set value, to obtain the optimized rotation matrix in the ground calibration parameters. This optimization process is an iterative process, that is, using the optimized rotation matrix as the initial rotation matrix, continuing to transform the 3D calibration position in the main camera coordinates to the world coordinate system represented by the initial calibration parameters, calculating the ground distance residual value, and continuing to optimize the rotation matrix in the ground calibration parameters with the ground distance residual value as the optimization target.

[0050] For example, in a specific example of this application, the data (i.e., the height value) in the Z-axis direction of the p2 coordinate system to be calibrated is compared with the actual ground height h of the target location to obtain the ground height residual value. All ground height residual values ​​for all points and ground heights in p2 are accumulated, and a ground height residual value of 0 (as a specific example of a set value) is used as the optimization objective to iteratively optimize the rotation matrix Rg of the main camera relative to the ground, resulting in Rg' (as a specific example of the optimized rotation matrix). Specifically, in practical applications, the above operation can be repeated multiple times until the rotation matrix Rg gradually approaches the ideal state. Once the accumulated ground height residual value is less than or close to the set value of 0, the ground calibration optimization task is completed, and the ideal Rg' is obtained.

[0051] Due to errors or improper calibration, the z-coordinate (i.e., height value) of the 3D coordinate position point p2 is inconsistent with its actual height h above the ground. Therefore, the residual r can be obtained by subtraction (as a specific example of the distance residual value). The distance residual values ​​after transforming all points p1 into points p2 are accumulated by the absolute value or square value of the residual r. The residual of 0 is used as the optimization objective to optimize the rotation matrix Rg of the main camera relative to the ground. The optimization method can first convert the current Rg into a rotation vector or Euler angles with 3 degrees of freedom, and then use a nonlinear optimization algorithm with numerical calculation, such as the LM algorithm (Levenberg–Marquardt) and the DogLeg algorithm. The optimized rotation vector or Euler angles are then converted into the optimized rotation matrix Rg'.

[0052] Understandably, the key to resolving the misalignment between the ground calibration results (i.e., ground calibration parameters) and the actual ground lies in optimizing the rotation matrix Rg between the main camera coordinate system and the ground coordinate system to be calibrated. Rg is a 3x3 matrix with 9 variables, but only 3 degrees of freedom. The rotation matrix can be converted into a rotation vector with 3 degrees of freedom, or 3 Euler angles. Leveling the ground only affects the pitch and roll angles of the 3 Euler angles; the yaw angle does not affect the parallelism between the calibrated plane and the actual ground. The yaw angle only affects the rotation of the ground coordinate system within the ground plane, so its true degree of freedom is only 2. Therefore, only two 3D coordinate points of the main camera coordinate system at known actual heights above the ground are needed; however, multiple 3D points will help achieve overall balance and optimal error control. Specifically, the number of coordinate points can be flexibly adjusted according to the actual application scenario.

[0053] In some embodiments of this application, the ground calibration parameters include ground yaw angle parameters, which can be obtained by the rotation matrix Rg. S230 may include: calculating the horizontal residual value between the 3D reference coordinate value and the actual position in the horizontal direction; wherein the horizontal direction is the X-axis direction or the Y-axis direction; based on the cumulative result of all horizontal residual values, and with the horizontal residual value as the set value as the optimization target, optimizing the ground yaw angle parameters to obtain optimized ground yaw angle parameters.

[0054] For example, in a specific example of this application, the horizontal residual value is calculated by comparing the value of the X-axis or Y-axis of the 3D coordinate position point with the x-axis or y-axis coordinate of the actual position. The residual value is then optimized with the residual value being zero (as a specific example of the set value) to obtain the optimized ground yaw angle parameter.

[0055] In some embodiments of this application, the ground calibration parameters include all parameters of the ground coordinate system to be calibrated, including pitch angle, roll angle, and heading angle. S230 may include: calculating the ground distance residual between the altitude value in the Z-axis direction of the 3D reference coordinate values ​​and the ground distance in the actual position; calculating the horizontal residual between the horizontal direction of the 3D reference coordinate values ​​and the horizontal direction of the actual position; wherein the horizontal direction is the X-axis direction or the Y-axis direction; based on the cumulative result of all ground distance residuals and the cumulative result of all horizontal residuals, using the cumulative result as the set value as the optimization target, optimizing the ground coordinate system to be calibrated to obtain the optimized grid yaw angle.

[0056] For example, in the specific example of this application, if we want to not only level the calibrated ground plane to match the real ground, but also correct the ground yaw angle, pitch angle, and roll angle in the ground coordinate system to be calibrated, then we need to input at least three 3D coordinate position points under the main camera. In addition to knowing their actual height h above the ground, we also need to obtain the x-axis or y-axis coordinates of that actual position in the calibration coordinate system, that is, the horizontal coordinate values. For example, in the calculation of the residual r, we need to consider not only the residual value above the ground in the Z-axis direction, but also integrate the horizontal residual values ​​in the X-axis or Y-axis directions. The integration method is to add the residuals in the corresponding directions, and then use zero (as a specific example of a set value) as the optimization target, and perform iterative optimization in the above manner to obtain the optimized ground coordinate system.

[0057] For example, after the above optimization is completed on the ground coordinate system to be calibrated, we can obtain... Figure 3 and Figure 4 Corresponding to Figure 5 and Figure 6 An example image showing the optimized and leveled ground grid.

[0058] The following is in conjunction with the appendix Figure 7 The present application provides an exemplary description of the specific process of ground calibration for a three-dimensional system, based on some embodiments thereof.

[0059] Please see the appendix Figure 7 , Figure 7 A flowchart of a method for ground calibration of a three-dimensional system is provided for some embodiments of this application.

[0060] The above process is illustrated below by example.

[0061] S710: Obtain the initial calibration parameters from the master camera in the calibrated multi-camera stereo vision system to the ground coordinate system to be calibrated.

[0062] S720 selects at least two pairs of corresponding points from at least two cameras with different viewpoints in a multi-camera stereo vision system.

[0063] S730, calculate at least two pairs of corresponding points in the 3D coordinate position p1 of the main camera coordinate system.

[0064] S740, transform the 3D coordinate position p1 to the ground coordinate system to be calibrated according to the initial calibration parameters to obtain the 3D reference coordinate value p2.

[0065] S750, calculate the residual between the height value in the 3D reference coordinate value p2 and the actual height h above the ground.

[0066] S760, accumulate all residuals after p1 is transformed into p2, and optimize the rotation matrix of the main camera relative to the ground coordinate system to be calibrated with the residual being 0 as the optimization objective.

[0067] S770: Determine whether the optimization objective of zero residual has been achieved. If so, proceed to S780. Otherwise, use the optimized rotation matrix obtained in S760 as the initial calibration parameter in S710 and return to S740 to continue optimization.

[0068] S780 outputs the optimized rotation matrix.

[0069] It should be noted that the specific implementation process of S710~S780 can be referred to the method embodiments provided above. To avoid repetition, detailed descriptions are omitted here.

[0070] As can be seen from the above embodiments of this application, this application can automatically solve the problem of the coordinate plane of ground calibration not being parallel to the real ground by using a few known real image points with ground height in addition to normal camera calibration. These few pixels used for calibration optimization can be selected manually or automatically detected by placing a calibration object, which has the advantages of high efficiency and flexibility; thereby improving the accuracy of ground calibration and providing a basis for the accurate acquisition of motion capture data.

[0071] Please refer to Figure 8 , Figure 8 The diagram illustrates a block diagram of a ground calibration apparatus for a three-dimensional system provided in some embodiments of this application. It should be understood that this ground calibration apparatus corresponds to the method embodiments described above and is capable of performing the various steps involved in the method embodiments. The specific functions of this ground calibration apparatus can be found in the description above; detailed descriptions are omitted here to avoid repetition.

[0072] Figure 8The device for ground calibration of a stereo system includes at least one software function module that can be stored in a memory or embedded in the device in the form of software or firmware. The device includes: an initial calibration module 810, used to obtain initial calibration parameters from the main camera to the ground coordinate system to be calibrated; wherein the initial calibration parameters include an initial rotation matrix and an initial translation vector; the main camera is randomly selected from a calibrated multi-camera stereo vision system, and the main camera corresponds to a main camera coordinate system; a coordinate transformation module 820, used to determine the calibration coordinate positions of at least two pairs of calibration points in the main camera coordinate system; wherein the at least two pairs of calibration points are pixels on images acquired by at least two cameras with different viewpoints in the multi-camera stereo vision system at the same time from the same target position; and an optimization module 830, used to optimize the initial calibration parameters based on the calibration coordinate positions to obtain ground calibration parameters.

[0073] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0074] Some embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can perform the operation of any of the methods corresponding to the methods provided in the above embodiments.

[0075] Some embodiments of this application also provide a computer program product, which includes a computer program, wherein when the computer program is executed by a processor, it can implement the operation of any of the methods corresponding to the above embodiments provided in the above embodiments.

[0076] like Figure 9 As shown, some embodiments of this application provide an electronic device 900, which includes a memory 910, a processor 920, and a computer program stored in the memory 910 and executable on the processor 920. When the processor 920 reads the program from the memory 910 via a bus 930 and executes the program, it can implement the methods of any of the above embodiments.

[0077] Processor 920 can process digital signals and can include various computing architectures. For example, it can be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 920 can be a microprocessor.

[0078] The memory 910 can be used to store instructions executed by the processor 920 or data related to the execution of instructions. These instructions and / or data may include code for implementing some or all of the functions of one or more modules described in the embodiments of this application. The processor 920 of this disclosure embodiment can be used to execute the instructions in the memory 910 to implement the methods shown above. The memory 910 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memories well known to those skilled in the art.

[0079] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0080] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0081] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for ground calibration of a three-dimensional system, characterized in that, include: Obtain the initial calibration parameters from the main camera to the ground coordinate system to be calibrated; wherein, the initial calibration parameters include an initial rotation matrix and an initial translation vector; the main camera is randomly selected from the calibrated multi-camera stereo vision system, and the main camera corresponds to a main camera coordinate system; Determine the calibration coordinate positions of at least two pairs of calibration points in the main camera coordinate system; wherein, the at least two pairs of calibration points are pixels on images acquired by at least two cameras with different viewpoints in the multi-camera stereo vision system at the same time from the same target position; The initial calibration parameters are optimized based on the calibration coordinates to obtain the ground calibration parameters.

2. The method as described in claim 1, characterized in that, Before obtaining the initial calibration parameters from the main camera to the ground coordinate system to be calibrated, the method further includes: By calibrating each camera in the multi-camera stereo vision system, the internal calibration parameters of each camera are determined, as well as the external calibration parameters between cameras in the multi-camera stereo vision system are determined; wherein, the internal calibration parameters include intrinsic parameter matrices and distortion coefficients; the external calibration parameters include camera rotation matrices and camera translation matrices between any camera in the multi-camera stereo vision system and other cameras except for any camera.

3. The method as described in claim 1 or 2, characterized in that, The optimization of the initial calibration parameters based on the calibration coordinates to obtain ground calibration parameters includes: The calibration coordinates in the main camera coordinate system are transformed to the ground coordinate system to be calibrated according to the initial calibration parameters to obtain the 3D reference coordinates in the ground coordinate system to be calibrated. Based on the 3D reference coordinate values ​​and the actual position of the target location in the ground coordinate system to be calibrated, the initial calibration parameters are optimized to obtain the ground calibration parameters.

4. The method as described in claim 3, characterized in that, The initial calibration parameters are optimized based on the 3D reference coordinate values ​​and the actual position of the target location in the ground coordinate system to be calibrated, to obtain the ground calibration parameters, including: Calculate the ground distance residual between the height value in the Z-axis direction of the 3D reference coordinates and the ground distance in the actual position; The distance-to-ground residual values ​​of all calibration points are accumulated, and the initial rotation matrix is ​​optimized using the distance-to-ground residual value as the optimization target to obtain the optimized rotation matrix in the ground calibration parameters.

5. The method as described in claim 3, characterized in that, The ground calibration parameters include ground yaw angle parameters. The initial calibration parameters are optimized based on the 3D reference coordinate values ​​and the actual position of the target location in the ground coordinate system to be calibrated, to obtain the ground calibration parameters, including: Calculate the horizontal residual value between the 3D reference coordinate values ​​in the horizontal direction and the actual position in the horizontal direction; wherein, the horizontal direction is the X-axis direction or the Y-axis direction; Based on the sum of all horizontal residual values, the ground yaw angle parameters are optimized with the horizontal residual values ​​as the set optimization target, resulting in optimized ground yaw angle parameters.

6. The method as described in claim 3, characterized in that, The ground calibration parameters include all parameters of the ground coordinate system to be calibrated, including pitch angle, roll angle, and yaw angle. The initial calibration parameters are optimized based on the 3D reference coordinate values ​​and the actual position of the target location in the ground coordinate system to be calibrated, resulting in the ground calibration parameters, including: Calculate the ground distance residual between the height value in the Z-axis direction of the 3D reference coordinates and the ground distance in the actual position; Calculate the horizontal residual value between the 3D reference coordinate values ​​in the horizontal direction and the actual position in the horizontal direction; wherein, the horizontal direction is the X-axis direction or the Y-axis direction; Based on the sum of all ground distance residuals and the sum of all horizontal residuals, the ground coordinate system to be calibrated is optimized with the sum of the sums as the optimization target, and the optimized ground coordinates are obtained.

7. The method as described in claim 1 or 2, characterized in that, Determining the calibration coordinate positions of at least two pairs of calibration points in the main camera coordinate system includes: Using triangulation calculation and optimization algorithms, the calibration coordinate positions of the at least two pairs of calibration points in the main camera coordinate system are calculated.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, performs the method as described in any one of claims 1-7.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and running on the processor, wherein the computer program is executed by the processor to perform the method as claimed in any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program, wherein the computer program is executed by a processor to perform the method as described in any one of claims 1-7.

Images