Method and apparatus for calibrating extrinsic parameters of multi-camera systems

Abstract

This application provides a method and apparatus for extrinsic parameter calibration of a multi-camera system. A calibration board is set up for each of the two cameras. The observed pixel coordinates of each corner point on the calibration board at each time step are obtained based on the images captured by the cameras. A first residual of the camera at each time step is constructed based on the spatial relationship between the poses of the two cameras and the two calibration boards at each time step. A second residual of the camera at each time step is constructed based on the spatial and temporal relationships between the poses of the two cameras and the two calibration boards at each time step. The temporal relationship includes the pose change of the camera between any two time steps. Then, a total residual is constructed based on the first and second residuals. Optimization is performed using the total residual as the objective to obtain the extrinsic parameters of the two cameras. This method constructs the total residual of the camera based on the spatial and temporal relationships during camera movement, and optimizes the camera extrinsic parameters using the total residual as the objective, thereby making the extrinsic parameter calibration of the camera more accurate.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method and apparatus for extrinsic parameter calibration of a multi-camera system. Background Technology

[0002] Multi-camera systems can be applied to applications such as 3D reconstruction, motion capture, and multi-view video. A multi-camera system includes at least two cameras (or webcams), such as the common dual-camera system. Dual-camera systems have been widely used in mobile phone photography and spatial perception of extended reality (XR) devices because they have a wider field of view, better low-light image quality, and the ability to acquire one-dimensional depth information. XR is a general term for various technologies such as virtual reality (VR), augmented reality (AR), and mixed reality (MR).

[0003] Currently, there are many methods for extrinsic parameter calibration of multi-camera systems, such as the epipolar geometry method with 2D-2D constraints, the 3D-2D perspective-n-point (PnP) method, and the 3D-3D iterative closest point (ICP) method. These methods are all highly constrained, requiring multiple cameras to meet a common-view condition. Common-view refers to multiple cameras observing the same feature points, and calibration is performed using these common-view feature points.

[0004] However, in many application scenarios, multiple cameras are in a state of weak co-view. Weak co-view refers to a situation with few shared feature points or poor data quality, but there is a high demand for camera extrinsic parameters. If traditional calibration methods are used, the following problems exist: Because multiple cameras are in a weak co-view state, the data quality acquired by each camera is poor, making it difficult to extract high-precision point cloud data; because the amount of data used for extrinsic parameter calculation is small, the calculation results are difficult to converge to an exact solution, resulting in inaccurate extrinsic parameter calibration results. Summary of the Invention

[0005] This application provides a method and apparatus for calibrating the extrinsic parameters of a multi-camera system. The method constructs the total residual of the camera based on the spatial and temporal relationships during camera movement, and optimizes the camera extrinsic parameters with the total residual as the target, thereby making the extrinsic parameter calibration of the camera more accurate.

[0006] In a first aspect, embodiments of this application provide a method for calibrating the extrinsic parameters of a multi-camera system, including:

[0007] The system acquires images of the corresponding calibration board at multiple times using a camera, and the position of the camera changes at these multiple times. The multi-camera system includes two cameras, each corresponding to a calibration board.

[0008] The observed pixel coordinates of each corner point on the calibration board at each time step are obtained from the image.

[0009] Based on the spatial relationship between the poses of the two cameras and the two calibration boards at each time step, the first estimated value of the pixel coordinates of each corner point on the calibration board at each time step is determined. Based on the observed value and the first estimated value of the pixel coordinates of each corner point on the calibration board, the first residual of the camera at each time step is determined.

[0010] Based on the spatial and temporal relationships of the poses of the two cameras and the two calibration boards at each time point, a second estimated value of the pixel coordinates of each corner point on the calibration board at each time point is determined. Based on the observed values ​​and the second estimated values ​​of the pixel coordinates of each corner point on the calibration board, a second residual of the camera at each time point is determined. The temporal relationship includes the pose change of the camera between any two time points.

[0011] The total residual is obtained based on the first and second residuals of the two cameras at each time point;

[0012] The total residual is used as the objective to optimize and solve for the extrinsic parameters of the two cameras.

[0013] In some embodiments, determining the first estimated value of the pixel coordinates of each corner point on the calibration board at each time step based on the spatial relationship between the poses of the two cameras and the two calibration boards at each time step includes:

[0014] Based on the relative pose between the camera and the corresponding calibration board at each time point, the relative pose between the two calibration boards, the relative pose between the two cameras, and the three-dimensional coordinates of each corner point on the calibration board, the first estimated value of the pixel coordinates of each corner point on the calibration board at each time point is determined.

[0015] In some embodiments, the second estimate includes a second sub-estimate and / or a third sub-estimate, and the second residual includes the second sub-residual and / or the third sub-residual;

[0016] The step of determining the second estimated value of the pixel coordinates of each corner point on the calibration board at each time step based on the spatial and temporal relationships of the poses of the two cameras and the two calibration boards at each time step includes:

[0017] Based on the relative pose between the camera and the calibration board at each time point and the pose change of the camera between any two time points, determine the second sub-estimated value of the pixel coordinates of each corner point on the calibration board at each time point;

[0018] Based on the relative pose between another camera and the corresponding calibration board at each time point, the relative pose between the two calibration boards, the relative pose between the two cameras, and the pose change of the camera at any two time points, the third sub-estimate of the pixel coordinates of each corner point on the calibration board at each time point is determined.

[0019] The step of determining the second residual of the camera at each time step based on the observed values ​​and second estimated values ​​of the pixel coordinates of each corner point on the calibration board includes:

[0020] Based on the observed values ​​and second sub-estimates of the pixel coordinates of each corner point on the calibration board, the second sub-residual of the camera at each time step is determined;

[0021] The third sub-residual of the camera at each time step is determined based on the observed values ​​and the third sub-estimated values ​​of the pixel coordinates of each corner point on the calibration board.

[0022] In some embodiments, the first residual of camera 0 at time i is... Determined by the following formula:

[0023]

[0024] in, This represents the first estimated value of the pixel coordinates of the nth corner point of the calibration board corresponding to camera 0. This represents the three-dimensional coordinates of the nth corner point. This represents the observed pixel coordinates of the nth corner point. Represents the projection function of camera 0. This indicates the relative pose between the two calibration plates. This indicates the relative pose between the two cameras. This represents the relative pose of camera 1 and the calibration plate at time j.

[0025] In some embodiments, the first residual of camera 1 at time i Determined by the following formula:

[0026]

[0027] in, This represents the first estimated value of the pixel coordinates of the nth corner point of the calibration board corresponding to camera 1. This represents the three-dimensional coordinates of the nth corner point. This represents the observed pixel coordinates of the nth corner point. This represents the projection function of camera 1. This indicates the relative pose between the two calibration plates. This indicates the relative pose between the two cameras. This represents the relative pose of camera 0 and the calibration plate at time j.

[0028] In some embodiments, the second sub-residual of camera 0 at time i... Determined by the following formula:

[0029]

[0030] in, The second sub-estimated value represents the pixel coordinates of the nth corner point of the calibration board corresponding to camera 0. This represents the three-dimensional coordinates of the nth corner point. This represents the observed pixel coordinates of the nth corner point. Represents the projection function of camera 0. This represents the relative pose of camera 0 and the calibration board at time i. This represents the pose change of camera 0 from time i to time j.

[0031] In some embodiments, the second sub-residual of camera 1 at time i Determined by the following formula:

[0032]

[0033] in, The second sub-estimated value represents the pixel coordinates of the nth corner point of the calibration board corresponding to camera 1. This represents the three-dimensional coordinates of the nth corner point. This represents the observed pixel coordinates of the nth corner point. This represents the projection function of camera 1. This represents the relative pose of camera 1 and the calibration board at time i. This represents the pose change of camera 1 from time i to time j.

[0034] In some embodiments, the third sub-residue of camera 0 at time i Determined by the following formula:

[0035]

[0036] in, This represents the third sub-estimate of the pixel coordinates of the nth corner point of the calibration board corresponding to camera 0. This represents the three-dimensional coordinates of the nth corner point. This represents the observed pixel coordinates of the nth corner point. Represents the projection function of camera 0. This indicates the relative pose between the two calibration plates. This indicates the relative pose between the two cameras. This represents the pose change of camera 0 between time i and time j. This represents the relative pose of camera 1 and the calibration board at time i.

[0037] In some embodiments, the third sub-residue of camera 1 at time i Determined by the following formula:

[0038]

[0039] in, The third sub-estimated value represents the pixel coordinates of the nth corner point of the calibration board corresponding to camera 1. This represents the three-dimensional coordinates of the nth corner point. This represents the observed pixel coordinates of the nth corner point. This represents the projection function of camera 1. This indicates the relative pose between the two calibration plates. This indicates the relative pose between the two cameras. This represents the pose change of camera 1 from time i to time j. This represents the relative pose of camera 0 and the calibration board at time i.

[0040] In some embodiments, obtaining the total residual based on the first and second residuals of the two cameras at each time point includes:

[0041] The total residual is calculated using the following formula:

[0042]

[0043] in, This represents the first residual of camera 0 at time i. This represents the first residual of camera 1 at time i. This represents the second sub-residual of camera 0 at time i. This represents the second sub-residual of camera 1 at time i. This represents the third sub-residual of camera 0 at time i. Let represent the third sub-residual of camera 1 at time i.

[0044] In some embodiments, the relative pose of the camera and the calibration plate at each time point can be obtained by the perspective n-point PnP algorithm.

[0045] In some embodiments, the pose changes of the camera at any two moments are obtained by solving the epipolar geometry method.

[0046] In some embodiments, the optimization aimed at the total residual to obtain the extrinsic parameters of the two cameras includes:

[0047] The initial values ​​of the extrinsic parameters of the camera are determined based on the relative poses of the two cameras and the calibration board at any two moments, as well as the relative poses between the two cameras.

[0048] Based on the initial values ​​of the camera's extrinsic parameters, the total residual is optimized to obtain the camera's extrinsic parameters corresponding to the minimum total residual.

[0049] In some embodiments, determining the initial values ​​of the camera's extrinsic parameters based on the relative poses of the two cameras and the calibration board at any two moments, and the relative pose between the two cameras, includes:

[0050] Based on the relative poses of the two cameras and the calibration board at time i and time j, the following equations are constructed:

[0051]

[0052] Transform the equation into Form, among which, , , The initial values ​​of the camera extrinsic parameters are obtained by solving based on the transformed form;

[0053] in, This indicates the relative pose between the two calibration plates. This indicates the relative pose between the two cameras. This represents the relative pose of camera 1 and the calibration board at time i. Let represent the relative pose of camera 1 and the calibration board at time j. This represents the relative pose of camera 0 and the calibration board at time i. This represents the relative pose of camera 0 and the calibration plate at time j.

[0054] On the other hand, embodiments of this application provide an extrinsic parameter calibration device for a multi-camera system, comprising:

[0055] An image acquisition module is used to acquire images of the corresponding calibration board at multiple times using a camera. The position of the camera changes at the multiple times. The multi-camera system includes two cameras, each camera corresponding to a calibration board.

[0056] A pixel coordinate determination module is used to obtain the observed values ​​of the pixel coordinates of each corner point on the calibration board at each time based on the image.

[0057] The residual determination module is used to determine the first estimated value of the pixel coordinates of each corner point on the calibration board at each time step based on the spatial relationship of the poses of the two cameras and the two calibration boards at each time step, and to determine the first residual of the camera at each time step based on the observed value and the first estimated value of the pixel coordinates of each corner point on the calibration board.

[0058] The residual determination module is further configured to determine the second estimated value of the pixel coordinates of each corner point on the calibration board at each time step based on the spatial and temporal relationship of the poses of the two cameras and the two calibration boards at each time step, and to determine the second residual of the camera at each time step based on the observed value and the second estimated value of the pixel coordinates of each corner point on the calibration board, wherein the temporal relationship includes the pose change of the camera between any two time steps.

[0059] The residual determination module is also used to obtain the total residual based on the first residual and the second residual of the two cameras at each time point;

[0060] The optimization module is used to optimize with the total residual as the objective and solve for the extrinsic parameters of the two cameras.

[0061] On the other hand, embodiments of this application provide an electronic device, the electronic device including: a processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to perform the method as described in any of the above.

[0062] On the other hand, embodiments of this application provide a computer-readable storage medium for storing a computer program that causes a computer to perform the methods described in any of the foregoing descriptions.

[0063] On the other hand, embodiments of this application provide a computer program product, including a computer program, characterized in that, when the computer program is executed by a processor, it implements the method described in any of the above-mentioned embodiments.

[0064] The extrinsic parameter calibration method and apparatus for a multi-camera system provided in this application involves setting up a calibration board for each of the two cameras during extrinsic parameter calibration. The observed pixel coordinates of each corner point on the calibration board at each time step are obtained based on the images captured by the cameras. A first residual for each camera at each time step is constructed based on the spatial relationship between the poses of the two cameras and the two calibration boards at each time step. A second residual for each camera at each time step is constructed based on the spatial and temporal relationships between the poses of the two cameras and the two calibration boards at each time step. This temporal relationship includes the pose change of the camera between any two time steps. Then, a total residual is constructed based on the first and second residuals. Optimization is performed using the total residual as the objective to obtain the extrinsic parameters of the two cameras. This method constructs the total residual of the camera based on the spatial and temporal relationships during camera movement and optimizes the camera extrinsic parameters using the total residual as the objective, thereby making the extrinsic parameter calibration of the camera more accurate. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a schematic diagram illustrating the calibration process for a multi-camera system to which this application applies;

[0067] Figure 2 A flowchart of the extrinsic parameter calibration method for a multi-camera system provided in Embodiment 1 of this application;

[0068] Figure 3 This is a schematic diagram of the camera's attitude trajectory during movement.

[0069] Figure 4 This is a schematic diagram of the external parameter calibration device for a multi-camera system provided in Embodiment 2 of this application;

[0070] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of this application. Detailed Implementation

[0071] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0073] This application provides a method for extrinsic parameter calibration of a multi-camera system, applicable to electronic devices with multi-camera systems, such as XR devices and mobile phones. XR refers to combining the real and virtual worlds through computers to create a virtual environment that allows for human-computer interaction. XR is also a collective term for various technologies such as VR, AR, and MR. By integrating the visual interaction technologies of these three technologies, it brings a seamless "immersive experience" between the virtual and real worlds to the user.

[0074] VR: A technology for creating and experiencing virtual worlds. It computationally generates a virtual environment, which is a multi-source information (virtual reality mentioned in this article includes at least visual perception, and may also include auditory perception, tactile perception, motion perception, and even taste perception, olfactory perception, etc.) that realizes the fusion of virtual environment, interactive three-dimensional dynamic visual scenes and simulation of physical behavior, immersing users in the simulated virtual reality environment, and enabling applications in various virtual environments such as maps, games, videos, education, medical care, simulation, collaborative training, sales, assisted manufacturing, maintenance and repair.

[0075] VR devices refer to terminals that achieve virtual reality effects. They can typically be provided in the form of glasses, head-mounted displays (HMDs), or contact lenses to achieve visual perception and other forms of perception. Of course, the form of virtual reality devices is not limited to these, and they can be further miniaturized or enlarged as needed.

[0076] AR: An AR scene refers to a simulated scene in which at least one virtual object is superimposed on a physical scene or its representation. For example, an electronic system may have an opaque display and at least one imaging sensor for capturing images or videos of the physical scene, which are representations of the physical scene. The system combines the images or videos with virtual objects and displays this combination on the opaque display. Individuals use the system to indirectly view the physical scene via images or videos of the physical scene and observe the virtual objects superimposed on the physical scene. When the system uses one or more image sensors to capture images of the physical scene and uses those images to present the AR scene on an opaque display, the displayed images are referred to as video pass-through. Alternatively, the electronic system for displaying the AR scene may have a transparent or semi-transparent display through which an individual can directly view the physical scene. The system may display virtual objects on the transparent or semi-transparent display, allowing an individual to observe the virtual objects superimposed on the physical scene using the system. As another example, the system may include a projection system that projects virtual objects onto the physical scene. Virtual objects can be projected, for example, onto a physical surface or as holograms, allowing individuals to observe virtual objects superimposed on a physical setting using the system. Specifically, a technique involves calculating the camera's pose parameters in the real world (or 3D world, the real world) in real time during image acquisition, and adding virtual elements to the captured images based on these parameters. Virtual elements include, but are not limited to, images, videos, and 3D models. The goal of AR technology is to overlay the virtual world onto the real world on a screen for interactive experiences.

[0077] MR (Mixed Reality): By presenting virtual scene information within a real-world setting, an interactive feedback loop is established between the real world, the virtual world, and the user to enhance the realism of the user experience. For example, computer-generated sensory input (e.g., virtual objects) is integrated with sensory input or its representation from a physical setting within a simulated scene. In some MR scenes, the computer-generated sensory input can adapt to changes in sensory input from the physical setting. Additionally, some electronic systems used to present MR scenes can monitor orientation and / or position relative to the physical setting, enabling virtual objects to interact with real objects (i.e., physical elements from the physical setting or their representations). For example, the system can monitor motion so that virtual plants appear stationary relative to physical buildings.

[0078] In image measurement and machine vision applications, to determine the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image, a geometric model of the camera imaging must be established. These geometric model parameters are the camera parameters. The process of solving for these parameters is called camera calibration. In other words, the significance of camera calibration is to restore the points in the image to the real three-dimensional space, accurately reflecting the actual relative spatial relationships between the points.

[0079] The mapping of a point in real-world 3D space to the final pixel space involves the following processes:

[0080] 1. The positional or motion relationship between a point in three-dimensional space and the camera lens.

[0081] The process involves two coordinate systems: a world coordinate system specified in three-dimensional space, and a camera coordinate system formed by the camera lens and optical axis.

[0082] 2. Optical Imaging

[0083] This process involves the camera coordinate system and the image coordinate system. The image coordinate system is located on the imaging plane, with its origin at the intersection of the optical axis and the imaging plane. Because optical lenses are not ideal, radial and tangential distortions occur during the imaging process.

[0084] 3. Photoelectric conversion to pixels

[0085] This process involves transforming from the image coordinate system to the pixel coordinate system. This process includes rotation, scaling, and translation.

[0086] The parameters used in the transformation from the world coordinate system to the camera coordinate system in the above three processes are independent of the camera itself; these are called the camera's extrinsic parameters. Extrinsic parameters relate to the positional and motion relationships between points in 3D space and the camera. They are not fixed parameters and have different definitions depending on the application scenario.

[0087] Typically, camera extrinsic parameters can be represented by extrinsic parameter matrices, which include rotation and translation matrices. The rotation matrix describes the orientation of the world coordinate system axes relative to the camera coordinate axes, while the translation matrix describes the position of the origin in the camera coordinate system.

[0088] Camera intrinsic parameters are parameters related to the camera's own characteristics, such as focal length and pixel size. Different types of cameras have different intrinsic parameters. Camera types include, but are not limited to, pinhole cameras and fisheye cameras.

[0089] Taking a pinhole camera as an example, the internal parameters of a pinhole camera include: , , , , , , and .in, , Indicates the camera's focal length. , Indicates the position of the optical center. , Represents the radial distortion parameter. , This represents the tangential distortion parameter.

[0090] The extrinsic parameter calibration method of this application embodiment is applicable to multi-camera systems, which include multiple cameras. Each camera is equipped with a dedicated calibration board. For any two cameras, the extrinsic parameter calibration method of this application embodiment can be used.

[0091] Camera calibration can be done using a calibration plate, which is a flat plate with a fixed-spacing pattern array. Common calibration plates include solid circle array patterns and chessboard patterns (referred to as chessboard grids).

[0092] Figure 1 This is a schematic diagram of the calibration process for a multi-camera system to which this application applies, as shown below. Figure 1 As shown, the multi-camera system includes two cameras: cam0 and cam1. cam0 is equipped with a dedicated calibration board: calibration board 0, and cam1 is equipped with a dedicated calibration board: calibration board 1. Figure 1 In the system shown, calibration board 0 and calibration board 1 use a checkerboard pattern.

[0093] Figure 1 In the system shown, although the two cameras are in motion, their relative positions are fixed. The rigid body constraint between the two cameras is defined as follows: , This indicates the relative pose between two cameras.

[0094] The positions of the two calibration plates are fixed, and the rigid body constraint between the two cameras is defined as follows: , This indicates the relative pose between two calibration plates.

[0095] The camera is in motion, and the relative pose between the camera and the calibration plate changes continuously as the camera moves. Figure 1 In this context, i and j represent two different moments, and correspondingly, , , , , , The definition is as follows.

[0096] : The relative pose of cam0 and calibration plate 0 at time i.

[0097] : The relative pose of cam0 and calibration plate 0 at time j.

[0098] : The relative pose of cam1 and calibration board 1 at time i.

[0099] : The relative pose of cam1 and calibration plate1 at time j.

[0100] : The pose change of cam0 from time i to time j.

[0101] : The pose change of cam1 from time i to time j.

[0102] This application's embodiments introduce rigid body constraints jointly constructed by the camera's temporal pose change and the spatial relationship between the camera and the calibration plate. , Together, they calibrated the camera's extrinsic parameters, thus making the calibration results more accurate.

[0103] It should be noted that in the embodiments of this application, the viewpoints of the two cameras may have a shared viewing area, or they may not have a shared viewing area, or they may have a weak shared viewing area. Regardless of the shared viewing area of ​​the two cameras, the camera extrinsic parameters can be calibrated.

[0104] according to Figure 1 The spatial relationship between the poses of the two cameras and the two calibration boards shown can be obtained by the following two spatial links:

[0105] (1)

[0106] (2)

[0107] Formulas (1) and (2) mean that the relative pose of any camera with the calibration board at time j is represented by the relative pose between the two cameras, the relative pose between the two calibration boards, and the relative pose of another camera with the calibration board 0 at time j.

[0108] according to Figure 1 The spatial and temporal relationships of the poses of the two cameras and the two calibration boards shown can be used to obtain the following four spatiotemporal links. Taking the two moments i and j as examples, the temporal relationship can be used to describe the pose change of the camera between any two moments.

[0109] (3)

[0110] (4)

[0111] (5)

[0112] (6)

[0113] Formulas (3) and (4) mean that the relative pose of any camera with the calibration board at time j is represented by the relative pose of the camera with the calibration board at time i and the pose change of the camera from time i to time j.

[0114] Formulas (5) and (6) mean that the relative pose of any camera with the calibration board at time j is represented by the relative pose of another camera with the calibration board at time i, the relative pose between the two cameras, the relative pose between the two calibration boards, and the pose change of the camera from time i to time j.

[0115] It is understood that, in the embodiments of this application This represents the inverse operation, i.e., the inverse matrix of matrix X. According to the rules of matrix operations, formulas (1)-(6) can be transformed. For example, formula (2) can be transformed into: Similarly, other formulas can also be transformed, which will not be listed here.

[0116] exist Figure 1 Based on the system shown, Embodiment 1 of this application provides a method for calibrating the extrinsic parameters of a multi-camera system. Figure 2 This is a flowchart of the extrinsic parameter calibration method for a multi-camera system provided in Embodiment 1 of this application. The method in this embodiment is executed by a calibration device, which can be a camera or a device independent of the camera, such as... Figure 2 As shown, the method in this embodiment includes the following steps:

[0117] S201. The camera acquires images of the corresponding calibration board at multiple times. The position of the camera changes at multiple times. The multi-camera system includes two cameras, each corresponding to a calibration board.

[0118] In this embodiment, two cameras capture images of the corresponding calibration board at a preset frequency to obtain images of the calibration board at multiple moments. During the shooting process, the two cameras are in motion. For example, in an XR device using a dual-camera system, when the user wears a head-mounted XR device and moves, the cameras will also move accordingly.

[0119] S202. Obtain the observed pixel coordinates of each corner point on the calibration board at each time step based on the image.

[0120] The points used for calibration on the calibration board are called corner points. When the calibration board uses a checkerboard pattern, the corner point is the vertex where two squares intersect. There are usually multiple corner points on the calibration board, and the number of corner points on the calibration board is selected according to the actual application scenario.

[0121] Pixel coordinates, also known as UV coordinates, are used to locate any pixel in an image. To obtain the pixel coordinates of a point in three-dimensional space, three transformations are typically required: first, from the world coordinate system to the camera coordinate system; then, perspective projection from the camera coordinate system to the image coordinate system; and finally, a second transformation is performed on the image coordinate system to obtain the pixel coordinate system.

[0122] Taking a pinhole camera, which exhibits radial and tangential distortion, as an example, the pixel coordinates of each corner point can be calculated using the following set of equations.

[0123]

[0124] in, , Indicates the camera's focal length. , Indicates the position of the optical center. , Represents the radial distortion parameter. , Indicates the tangential distortion parameter, ( , , () represents the three-dimensional coordinates of the corner point. The three-dimensional coordinate system of the corner point refers to the coordinate values ​​of the corner point in the world coordinate system.

[0125] It is understood that different types of cameras have different intrinsic parameters, and correspondingly, the calculation process of the pixel coordinates of each corner point is different. Any existing method can be used to calculate the pixel coordinates of each corner point. This application does not limit the calculation method of the pixel coordinates of each corner point.

[0126] In this embodiment of the application, when calculating the observed values ​​of the pixel coordinates of each corner point, the intrinsic parameters of the camera can be considered as known. That is, this application does not focus on the calibration of the intrinsic parameters of the camera, and directly uses the intrinsic parameters of the camera when calibrating the extrinsic parameters.

[0127] S103. Based on the spatial relationship between the poses of the two cameras and the two calibration boards at each time step, determine the first estimated value of the pixel coordinates of each corner point on the calibration board at each time step. Based on the observed value and the first estimated value of the pixel coordinates of each corner point on the calibration board, determine the first residual of the camera at each time step.

[0128] First, it's necessary to define the observed and estimated pixel coordinates of the corner points. The observed values ​​can be understood as those inferred from the camera's pose, while the estimated values ​​are inferred from the relative poses of the camera and the calibration board. In mathematical statistics, the residual refers to the difference between the observed and estimated values.

[0129] The spatial relationship includes the relative pose between the camera and the calibration board, the relative pose between the two calibration boards, and the relative pose between the two cameras. Accordingly, based on the relative pose between the camera and the corresponding calibration board at each time step, the relative pose between the two calibration boards, the relative pose between the two cameras, and the three-dimensional coordinates of each corner point on the calibration board, the first estimated value of the pixel coordinates of each corner point on the calibration board at each time step is determined.

[0130] For example, the first residual of camera 0 at time i. It is determined by the following formula (7), where formula 7 is obtained from formula (1).

[0131] (7)

[0132] in, This represents the first estimated value of the pixel coordinates of the nth corner point of the calibration board corresponding to camera 0. This represents the three-dimensional coordinates of the nth corner point. This represents the observed pixel coordinates of the nth corner point. Let n represent the projection function of camera 0, and n represent the index of the corner point, with n ranging from 0 to k. This indicates the relative pose between the two calibration plates. This indicates the relative pose between two cameras. This represents the relative pose of camera 1 and the calibration plate at time j.

[0133] For example, the first residual of camera 1 at time i. It is determined by the following formula (8), where formula (8) is obtained from formula (1).

[0134] (8)

[0135] in, This represents the first estimated value of the pixel coordinates of the nth corner point of the calibration board corresponding to camera 1. This represents the three-dimensional coordinates of the nth corner point. This represents the observed pixel coordinates of the nth corner point. This represents the projection function of camera 1. This indicates the relative pose between the two calibration plates. This indicates the relative pose between the two cameras. This represents the relative pose of camera 0 and the calibration plate at time j.

[0136] S104. Based on the spatial and temporal relationships of the poses of the two cameras and the two calibration boards at each moment, determine the second estimated value of the pixel coordinates of each corner point on the calibration board at each moment. Based on the observed value and the second estimated value of the pixel coordinates of each corner point on the calibration board, determine the second residual of the camera at each moment. The temporal relationship includes the pose change of the camera between any two moments.

[0137] Optionally, the second estimate includes a second sub-estimate and / or a third sub-estimate, and correspondingly, the second residual includes a second sub-residual and / or a third sub-residual.

[0138] The spatial relationship includes the relative pose between the camera and the calibration board, the relative pose between the two calibration boards, and the relative pose between the two cameras. The temporal relationship includes the pose change of the camera between any two moments. Accordingly, for any camera, based on the relative pose between the camera and the calibration board at each moment and the pose change of the camera between any two moments, a second sub-estimate of the pixel coordinates of each corner point on the calibration board at each moment is determined. And / or, based on the relative pose between the other camera and its corresponding calibration board at each moment, the relative pose between the two calibration boards, the relative pose between the two cameras, and the pose change of the camera between any two moments, a third sub-estimate of the pixel coordinates of each corner point on the calibration board at each moment is determined.

[0139] After determining the second sub-estimate and / or the third sub-estimate, the second sub-residual of the camera at each time step is determined based on the observed pixel coordinates of each corner point on the calibration board and the second sub-estimate, and / or the third sub-residual of the camera at each time step is determined based on the observed pixel coordinates of each corner point on the calibration board and the third sub-estimate.

[0140] For example, the second sub-residue of camera 0 at time i. It is determined by the following formula (9), which is obtained from formula (3).

[0141]

[0142] in, The second sub-estimated value represents the pixel coordinates of the nth corner point of the calibration board corresponding to camera 0. This represents the three-dimensional coordinates of the nth corner point. This represents the observed pixel coordinates of the nth corner point. Represents the projection function of camera 0. This represents the relative pose of camera 0 and the calibration board at time i. This represents the pose change of camera 0 from time i to time j.

[0143] For example, the second sub-residue of camera 1 at time i. It is determined by the following formula (10), which is obtained from formula (4).

[0144] (10)

[0145] in, The second sub-estimated value represents the pixel coordinates of the nth corner point of the calibration board corresponding to camera 1. This represents the three-dimensional coordinates of the nth corner point. This represents the observed pixel coordinates of the nth corner point. This represents the projection function of camera 1. This represents the relative pose of camera 1 and the calibration board at time i. This represents the pose change of camera 1 from time i to time j.

[0146] For example, the third sub-residue of camera 0 at time i. It is determined by the following formula (11), which is obtained from formula (5).

[0147] (11)

[0148] in, This represents the third sub-estimate of the pixel coordinates of the nth corner point of the calibration board corresponding to camera 0. This represents the three-dimensional coordinates of the nth corner point. This represents the observed pixel coordinates of the nth corner point. Represents the projection function of camera 0. This indicates the relative pose between the two calibration plates. This indicates the relative pose between two cameras. This represents the pose change of camera 0 between time i and time j. This represents the relative pose of camera 1 and the calibration board at time i.

[0149] For example, the third sub-residue of camera 1 at time i. It is determined by the following formula (12), which is obtained from formula (6).

[0150] (12)

[0151] in, The third sub-estimated value represents the pixel coordinates of the nth corner point of the calibration board corresponding to camera 1. This represents the three-dimensional coordinates of the nth corner point. This represents the observed pixel coordinates of the nth corner point. This represents the projection function of camera 1. This indicates the relative pose between the two calibration plates. This indicates the relative pose between two cameras. This represents the pose change of camera 1 from time i to time j. This represents the relative pose of camera 0 and the calibration board at time i.

[0152] It should be noted that the residual calculation formulas shown in formulas (7)-(12) correspond to the optimized link shown in formulas (1)-(6). Since formulas (1)-(6) can be transformed according to the matrix operation rules, formulas (7)-(12) are also transformed accordingly.

[0153] S105. Obtain the total residual based on the first and second residuals of the two cameras at each time point.

[0154] For the entire movement of the camera, it can be simplified to: Figure 3 Given the attitude trajectory shown, the spatiotemporal link can be constructed by combining multiple time points. At multiple time points, assuming that the second residual includes the second sub-residual and the third sub-residual, for example, the total residual is calculated by the following formula (13):

[0155] (13)

[0156] in, This represents the first residual of camera 0 at time i. This represents the first residual of camera 1 at time i. This represents the second sub-residual of camera 0 at time i. This represents the second sub-residual of camera 1 at time i. This represents the third sub-residual of camera 0 at time i. Let represent the third sub-residual of camera 1 at time i.

[0157] In formula (13), t represents each moment in the camera movement process, c represents any two moments extracted from the camera attitude trajectory, and the values ​​of i and j are both from 0 to t. In c, i and j are not equal.

[0158] This method constructs the camera's total residual based on the spatial and temporal relationships during camera movement. Optimization of this total residual yields the camera's extrinsic parameters, resulting in more accurate extrinsic parameter calibration. Furthermore, the camera extrinsic parameter calibration no longer depends on the shared field of view of the two cameras; even if the two cameras do not share a field of view, or if they have a weak shared field of view, the extrinsic parameters can still be accurately calibrated.

[0159] S106. Optimize with the total residual as the objective and solve for the extrinsic parameters of the two cameras.

[0160] By constructing the total residual, a bundle adjustment (BA) optimization problem is constructed. BA optimization is to use 3D points to project and find the most suitable camera pose (rotation matrix and translation matrix). In other words, it is to iteratively adjust the beam to make the beam satisfy the constraint plane.

[0161] In this application, the extrinsic parameters that can be provided include: the relative pose of the camera and the calibration board at each moment, i.e. , , , And the pose changes of the camera at any two moments, i.e. , .

[0162] Optional, , , , It can be solved using the PnP algorithm. , It can be solved using the epipolar geometry method. Both the PnP algorithm and the epipolar geometry method are mature algorithms, and will not be elaborated on here.

[0163] exist In the calculation formula (i.e., formula 7), the known term is... .exist In the calculation formula (i.e., formula 8), the known term is... .exist In the calculation formula (i.e., formula 9), the known term is... and .exist In the calculation formula (i.e., formula 10), the known term is... and .exist In the calculation formula (i.e., formula 11), the known term is... and .exist In the calculation formula (i.e., formula 12), the known term is... and .

[0164] After calculating the above-mentioned known parameters, the total residual is optimized based on the known parameters to obtain the camera's extrinsic parameters corresponding to the minimum total residual. Optionally, the extrinsic parameters of the cameras can be determined based on the relative poses of the two cameras and the calibration board at any two moments, as well as the relative poses between the two cameras. The initial value is determined based on the camera's extrinsic parameters. The initial values ​​are used to optimize the total residuals, and the camera's extrinsic parameters corresponding to the minimum total residuals are obtained by solving. .

[0165] For example, based on the relative poses of the two cameras and the calibration board at time i and time j, the following equation is constructed, namely formula (14):

[0166] (14)

[0167] Convert formula (14) to Form, among which, , , The camera extrinsic parameters are obtained by solving based on the transformed form. The initial value.

[0168] in, This indicates the relative pose between the two calibration plates. This indicates the relative pose between two cameras. This represents the relative pose of camera 1 and the calibration board at time i. Let represent the relative pose of camera 1 and the calibration board at time j. This represents the relative pose of camera 0 and the calibration board at time i. This represents the relative pose of camera 0 and the calibration plate at time j.

[0169] Optionally, camera extrinsic parameters can also be adjusted. Arbitrary initial values ​​can be assigned; this application embodiment does not impose any limitation on this, regarding camera extrinsic parameters. Assigning a suitable initial value can improve optimization efficiency and shorten optimization time.

[0170] It is understandable that optimization cannot yield the camera extrinsic parameters with the minimum total residual. It can also obtain the calibration plate extrinsic parameters with the minimum total residual. .

[0171] Optionally, in this embodiment, the Levenberg-Marquardt (LM) algorithm can be used to solve for the extrinsic parameters of the two cameras. The key to the LM algorithm is to use the model function fm to make a linear approximation of the parameter vector p to be estimated in its neighborhood, ignoring derivatives of second order and above, thus transforming it into a linear least squares problem. It has advantages such as fast convergence speed. A simple description of the LM algorithm is an iterative process: "If the objective function value increases, adjust a coefficient and continue solving; if the objective function value decreases, adjust a coefficient and continue solving."

[0172] It is understandable that other existing optimization algorithms can also be used to optimize the total residual, such as the Gauss-Newton method and non-linear least squares problems.

[0173] In this embodiment, when calibrating the extrinsic parameters of a multi-camera system, a calibration board is set up for each of the two cameras. The observed pixel coordinates of each corner point on the calibration board at each time step are obtained based on the images captured by the cameras. Based on the spatial relationship between the poses of the two cameras and the two calibration boards at each time step, a first residual of the camera at each time step is constructed. Furthermore, based on the spatial and temporal relationships between the poses of the two cameras and the two calibration boards at each time step, a second residual of the camera at each time step is constructed. This temporal relationship includes the pose change of the camera between any two time steps. Then, a total residual is constructed based on the first and second residuals. Optimization is performed with the total residual as the objective to obtain the extrinsic parameters of the two cameras. This method constructs the total residual of the camera based on the spatial and temporal relationships during camera movement, thereby making the extrinsic parameter calibration of the camera more accurate.

[0174] To facilitate better implementation of the extrinsic parameter calibration method for a multi-camera system according to the embodiments of this application, the embodiments of this application also provide an adjustment device for extrinsic parameter calibration of a multi-camera system. Figure 4 This is a schematic diagram of the external parameter calibration device for a multi-camera system provided in Embodiment 2 of this application, as shown below. Figure 4 As shown, the extrinsic parameter calibration device 100 of the multi-camera system may include:

[0175] Image acquisition module 11 is used to acquire images of the corresponding calibration plate at multiple times via a camera. The position of the camera changes at the multiple times. The multi-camera system includes two cameras, each camera corresponding to a calibration plate.

[0176] The pixel coordinate determination module 12 is used to obtain the observed values ​​of the pixel coordinates of each corner point on the calibration board at each time based on the image.

[0177] The residual determination module 13 is used to determine the first estimated value of the pixel coordinates of each corner point on the calibration board at each time according to the spatial relationship of the poses of the two cameras and the two calibration boards at each time, and to determine the first residual of the camera at each time according to the observed value and the first estimated value of the pixel coordinates of each corner point on the calibration board.

[0178] The residual determination module 13 is further configured to determine the second estimated value of the pixel coordinates of each corner point on the calibration board at each time according to the spatial and temporal relationship of the poses of the two cameras and the two calibration boards at each time, and to determine the second residual of the camera at each time according to the observed value and the second estimated value of the pixel coordinates of each corner point on the calibration board, wherein the temporal relationship includes the pose change of the camera between any two time points;

[0179] The residual determination module 13 is also used to obtain the total residual based on the first residual and the second residual of the two cameras at each time point;

[0180] The optimization module 14 is used to optimize with the total residual as the objective and solve for the extrinsic parameters of the two cameras.

[0181] In some embodiments, the residual determination module 13 is specifically used for:

[0182] Based on the relative pose between the camera and the corresponding calibration board at each time point, the relative pose between the two calibration boards, the relative pose between the two cameras, and the three-dimensional coordinates of each corner point on the calibration board, the first estimated value of the pixel coordinates of each corner point on the calibration board at each time point is determined.

[0183] In some embodiments, the second estimate includes a second sub-estimate and / or a third sub-estimate, and the second residual includes the second sub-residual and / or the third sub-residual;

[0184] The residual determination module 13 is specifically used for:

[0185] Based on the relative pose between the camera and the calibration board at each time point and the pose change of the camera between any two time points, determine the second sub-estimated value of the pixel coordinates of each corner point on the calibration board at each time point;

[0186] Based on the relative pose between another camera and the corresponding calibration board at each time point, the relative pose between the two calibration boards, the relative pose between the two cameras, and the pose change of the camera at any two time points, the third sub-estimate of the pixel coordinates of each corner point on the calibration board at each time point is determined.

[0187] Based on the observed values ​​and second sub-estimates of the pixel coordinates of each corner point on the calibration board, the second sub-residual of the camera at each time step is determined;

[0188] The third sub-residual of the camera at each time step is determined based on the observed values ​​and the third sub-estimated values ​​of the pixel coordinates of each corner point on the calibration board.

[0189] In some embodiments, the first residual of camera 0 at time i is... Determined by the following formula:

[0190]

[0191] in, This represents the first estimated value of the pixel coordinates of the nth corner point of the calibration board corresponding to camera 0. This represents the three-dimensional coordinates of the nth corner point. This represents the observed pixel coordinates of the nth corner point. Represents the projection function of camera 0. This indicates the relative pose between the two calibration plates. This indicates the relative pose between the two cameras. This represents the relative pose of camera 1 and the calibration plate at time j.

[0192] In some embodiments, the first residual of camera 1 at time i Determined by the following formula:

[0193]

[0194] in, This represents the first estimated value of the pixel coordinates of the nth corner point of the calibration board corresponding to camera 1. This represents the three-dimensional coordinates of the nth corner point. This represents the observed pixel coordinates of the nth corner point. This represents the projection function of camera 1. This indicates the relative pose between the two calibration plates. This indicates the relative pose between the two cameras. This represents the relative pose of camera 0 and the calibration plate at time j.

[0195] In some embodiments, the second sub-residual of camera 0 at time i... Determined by the following formula:

[0196]

[0197] in, The second sub-estimated value represents the pixel coordinates of the nth corner point of the calibration board corresponding to camera 0. This represents the three-dimensional coordinates of the nth corner point. This represents the observed pixel coordinates of the nth corner point. Represents the projection function of camera 0. This represents the relative pose of camera 0 and the calibration board at time i. This represents the pose change of camera 0 from time i to time j.

[0198] In some embodiments, the second sub-residual of camera 1 at time i Determined by the following formula:

[0199]

[0200] in, The second sub-estimated value represents the pixel coordinates of the nth corner point of the calibration board corresponding to camera 1. This represents the three-dimensional coordinates of the nth corner point. This represents the observed pixel coordinates of the nth corner point. This represents the projection function of camera 1. This represents the relative pose of camera 1 and the calibration board at time i. This represents the pose change of camera 1 from time i to time j.

[0201] In some embodiments, the third sub-residue of camera 0 at time i Determined by the following formula:

[0202]

[0203] in, This represents the third sub-estimate of the pixel coordinates of the nth corner point of the calibration board corresponding to camera 0. This represents the three-dimensional coordinates of the nth corner point. This represents the observed pixel coordinates of the nth corner point. Represents the projection function of camera 0. This indicates the relative pose between the two calibration plates. This indicates the relative pose between the two cameras. This represents the pose change of camera 0 between time i and time j. This represents the relative pose of camera 1 and the calibration board at time i.

[0204] In some embodiments, the third sub-residue of camera 1 at time i Determined by the following formula:

[0205]

[0206] in, The third sub-estimated value represents the pixel coordinates of the nth corner point of the calibration board corresponding to camera 1. This represents the three-dimensional coordinates of the nth corner point. This represents the observed pixel coordinates of the nth corner point. This represents the projection function of camera 1. This indicates the relative pose between the two calibration plates. This indicates the relative pose between the two cameras. This represents the pose change of camera 1 from time i to time j. This represents the relative pose of camera 0 and the calibration board at time i.

[0207] In some embodiments, obtaining the total residual based on the first and second residuals of the two cameras at each time point includes:

[0208] The total residual is calculated using the following formula:

[0209]

[0210] in, This represents the first residual of camera 0 at time i. This represents the first residual of camera 1 at time i. This represents the second sub-residual of camera 0 at time i. This represents the second sub-residual of camera 1 at time i. This represents the third sub-residual of camera 0 at time i. Let represent the third sub-residual of camera 1 at time i.

[0211] In some embodiments, the relative pose of the camera and the calibration plate at each time point can be obtained by the perspective n-point PnP algorithm.

[0212] In some embodiments, the pose changes of the camera at any two moments are obtained by solving the epipolar geometry method.

[0213] In some embodiments, the optimization module 14 is specifically used for:

[0214] The initial values ​​of the extrinsic parameters of the camera are determined based on the relative poses of the two cameras and the calibration board at any two moments, as well as the relative poses between the two cameras.

[0215] Based on the initial values ​​of the camera's extrinsic parameters, the total residual is optimized to obtain the camera's extrinsic parameters corresponding to the minimum total residual.

[0216] In some embodiments, the optimization module 14 is specifically used for:

[0217] Based on the relative poses of the two cameras and the calibration board at time i and time j, the following equations are constructed:

[0218]

[0219] Transform the equation into Form, among which, , , The initial values ​​of the camera extrinsic parameters are obtained by solving based on the transformed form;

[0220] in, This indicates the relative pose between the two calibration plates. This indicates the relative pose between the two cameras. This represents the relative pose of camera 1 and the calibration board at time i. Let represent the relative pose of camera 1 and the calibration board at time j. This represents the relative pose of camera 0 and the calibration board at time i. This represents the relative pose of camera 0 and the calibration plate at time j.

[0221] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here.

[0222] The apparatus 100 of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly manifested as execution by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0223] This application also provides an electronic device. Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of this application, such as... Figure 5 As shown, the electronic device 200 may include:

[0224] The system includes a memory 21, a processor 22, and a dual-camera module 23. The memory 21 stores computer programs and transmits the program code to the processor 22. In other words, the processor 22 can call and run the computer program from the memory 21 to implement the methods in the embodiments of this application. The dual-camera module 23 is used to acquire images and send the images to the memory 21 and the processor 22 for processing.

[0225] For example, the processor 22 can be used to execute the above-described method embodiments according to instructions in the computer program.

[0226] In some embodiments of this application, the processor 22 may include, but is not limited to:

[0227] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0228] In some embodiments of this application, the memory 21 includes, but is not limited to:

[0229] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0230] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 21 and executed by the processor 22 to perform the method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0231] like Figure 5 As shown, the electronic device 200 may further include a transceiver 24, which can be connected to the processor 22 or the memory 21.

[0232] The processor 22 can control the transceiver 24 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 24 may include a transmitter and a receiver. The transceiver 24 may further include antennas, and the number of antennas may be one or more.

[0233] Understandable, although Figure 5 As not shown in the diagram, the electronic device 200 may also include a Wi-Fi module, a positioning module, a Bluetooth module, a display, a controller, etc., which will not be described in detail here.

[0234] It should be understood that the components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0235] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0236] This application also provides a computer program product comprising a computer program stored in a computer-readable storage medium. The processor of an electronic device reads the computer program from the computer-readable storage medium and executes the computer program, causing the electronic device to perform the corresponding processes in the map relocation method incorporating semantic information as described in the embodiments of this application. For the sake of brevity, these details will not be elaborated further here.

[0237] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0238] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0239] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for calibrating extrinsic parameters of a multi-camera system, characterized in that, include: The system acquires images of the corresponding calibration board at multiple times using a camera, and the position of the camera changes at these multiple times. The multi-camera system includes two cameras, each corresponding to a calibration board. The observed pixel coordinates of each corner point on the calibration board at each time step are obtained from the image. Based on the spatial relationship between the poses of the two cameras and the two calibration boards at each time step, the first estimated value of the pixel coordinates of each corner point on the calibration board at each time step is determined. Based on the observed value and the first estimated value of the pixel coordinates of each corner point on the calibration board, the first residual of the camera at each time step is determined. Based on the spatial and temporal relationships of the poses of the two cameras and the two calibration boards at each time point, a second estimated value of the pixel coordinates of each corner point on the calibration board at each time point is determined. Based on the observed values ​​and the second estimated values ​​of the pixel coordinates of each corner point on the calibration board, a second residual of the camera at each time point is determined. The temporal relationship includes the pose change of the camera between any two time points. The total residual is obtained based on the first and second residuals of the two cameras at each time point; The total residual is used as the objective to optimize and solve for the extrinsic parameters of the two cameras.

2. The method of claim 1, wherein, The step of determining the first estimated value of the pixel coordinates of each corner point on the calibration board at each time step based on the spatial relationship of the poses of the two cameras and the two calibration boards at each time step includes: Based on the relative pose between the camera and the corresponding calibration board at each time point, the relative pose between the two calibration boards, the relative pose between the two cameras, and the three-dimensional coordinates of each corner point on the calibration board, the first estimated value of the pixel coordinates of each corner point on the calibration board at each time point is determined.

3. The method of claim 2, wherein, The second estimate includes a second sub-estimate and / or a third sub-estimate, and the second residual includes a second sub-residual and / or a third sub-residual; The step of determining the second estimated value of the pixel coordinates of each corner point on the calibration board at each time step based on the spatial and temporal relationships of the poses of the two cameras and the two calibration boards at each time step includes: Based on the relative pose between the camera and the calibration board at each time point and the pose change of the camera between any two time points, determine the second sub-estimated value of the pixel coordinates of each corner point on the calibration board at each time point; Based on the relative pose between another camera and the corresponding calibration board at each time point, the relative pose between the two calibration boards, the relative pose between the two cameras, and the pose change of the camera at any two time points, the third sub-estimate of the pixel coordinates of each corner point on the calibration board at each time point is determined. The step of determining the second residual of the camera at each time step based on the observed values ​​and second estimated values ​​of the pixel coordinates of each corner point on the calibration board includes: Based on the observed values ​​and second sub-estimates of the pixel coordinates of each corner point on the calibration board, the second sub-residual of the camera at each time step is determined; The third sub-residual of the camera at each time step is determined based on the observed values ​​and the third sub-estimated values ​​of the pixel coordinates of each corner point on the calibration board.

4. The method according to claim 2, characterized in that, First residual error of camera 0 at the i-th time instant is determined by the equation: wherein, denotes a first estimate of the pixel coordinates of the nth corner point of the calibration board corresponding to camera 0, denotes the three-dimensional coordinates of the nth corner point, denotes the observed pixel coordinates of the nth corner point, denotes the projection function of camera 0, denotes the relative pose between the two calibration boards, denotes the relative pose between the two cameras, denotes the relative pose of camera 1 with respect to the calibration board at the j-th time instant.

5. The method according to claim 2, characterized in that, The first residual of camera 1 at time i Determined by the following formula: in, This represents the first estimated value of the pixel coordinates of the nth corner point of the calibration board corresponding to camera 1. This represents the three-dimensional coordinates of the nth corner point. This represents the observed pixel coordinates of the nth corner point. This represents the projection function of camera 1. This indicates the relative pose between the two calibration plates. This indicates the relative pose between the two cameras. This represents the relative pose of camera 0 and the calibration plate at time j.

6. The method according to claim 3, characterized in that, The second sub-residue of camera 0 at time i Determined by the following formula: in, The second sub-estimated value represents the pixel coordinates of the nth corner point of the calibration board corresponding to camera 0. This represents the three-dimensional coordinates of the nth corner point. This represents the observed pixel coordinates of the nth corner point. This represents the projection function of camera 0. This represents the relative pose of camera 0 and the calibration board at time i. This represents the pose change of camera 0 from time i to time j.

7. The method according to claim 3, characterized in that, The second sub-residue of camera 1 at time i Determined by the following formula: in, The second sub-estimated value represents the pixel coordinates of the nth corner point of the calibration board corresponding to camera 1. This represents the three-dimensional coordinates of the nth corner point. This represents the observed pixel coordinates of the nth corner point. This represents the projection function of camera 1. This represents the relative pose of camera 1 and the calibration board at time i. This represents the pose change of camera 1 from time i to time j.

8. The method according to claim 3, characterized in that, The third sub-residue of camera 0 at time i Determined by the following formula: in, This represents the third sub-estimate of the pixel coordinates of the nth corner point of the calibration board corresponding to camera 0. This represents the three-dimensional coordinates of the nth corner point. This represents the observed pixel coordinates of the nth corner point. This represents the projection function of camera 0. This indicates the relative pose between the two calibration plates. This indicates the relative pose between the two cameras. This represents the pose change of camera 0 between time i and time j. This represents the relative pose of camera 1 and the calibration board at time i.

9. The method according to claim 3, characterized in that, The third sub-residue of camera 1 at time i Determined by the following formula: in, The third sub-estimated value represents the pixel coordinates of the nth corner point of the calibration board corresponding to camera 1. This represents the three-dimensional coordinates of the nth corner point. This represents the observed pixel coordinates of the nth corner point. This represents the projection function of camera 1. This indicates the relative pose between the two calibration plates. This indicates the relative pose between the two cameras. This represents the pose change of camera 1 from time i to time j. This represents the relative pose of camera 0 and the calibration board at time i.

10. The method according to claim 3, characterized in that, The step of obtaining the total residual based on the first and second residuals of the two cameras at each time step includes: The total residual is calculated using the following formula: in, This represents the first residual of camera 0 at time i. This represents the first residual of camera 1 at time i. This represents the second sub-residual of camera 0 at time i. This represents the second sub-residual of camera 1 at time i. This represents the third sub-residual of camera 0 at time i. Let represent the third sub-residual of camera 1 at time i.

11. The method according to any one of claims 2-10, characterized in that, The relative poses of the camera and the calibration board at each time point can be obtained by using the perspective n-point PnP algorithm.

12. The method according to any one of claims 2-10, characterized in that, The pose changes of the camera at any two moments are obtained by solving the epipolar geometry method.

13. The method according to any one of claims 1-10, characterized in that, The optimization, which aims to obtain the extrinsic parameters of the two cameras by targeting the total residual, includes: The initial values ​​of the extrinsic parameters of the camera are determined based on the relative poses of the two cameras and the calibration board at any two moments, as well as the relative poses between the two cameras. Based on the initial values ​​of the camera's extrinsic parameters, the total residual is optimized to obtain the camera's extrinsic parameters corresponding to the minimum total residual.

14. The method according to claim 13, characterized in that, The step of determining the initial values ​​of the camera's extrinsic parameters based on the relative poses of the two cameras and the calibration board at any two moments, and the relative poses between the two cameras, includes: Based on the relative poses of the two cameras and the calibration board at time i and time j, the following equations are constructed: Transform the equation into Form, among which, , , The initial values ​​of the camera extrinsic parameters are obtained by solving based on the transformed form; in, This indicates the relative pose between the two calibration plates. This indicates the relative pose between the two cameras. This represents the relative pose of camera 1 and the calibration board at time i. Let represent the relative pose of camera 1 and the calibration board at time j. This represents the relative pose of camera 0 and the calibration board at time i. This represents the relative pose of camera 0 and the calibration plate at time j.

15. An extrinsic parameter calibration device for a multi-camera system, characterized in that, include: An image acquisition module is used to acquire images of the corresponding calibration board at multiple times using a camera. The position of the camera changes at the multiple times. The multi-camera system includes two cameras, each camera corresponding to a calibration board. A pixel coordinate determination module is used to obtain the observed values ​​of the pixel coordinates of each corner point on the calibration board at each time based on the image. The residual determination module is used to determine the first estimated value of the pixel coordinates of each corner point on the calibration board at each time step based on the spatial relationship of the poses of the two cameras and the two calibration boards at each time step, and to determine the first residual of the camera at each time step based on the observed value and the first estimated value of the pixel coordinates of each corner point on the calibration board. The residual determination module is further configured to determine the second estimated value of the pixel coordinates of each corner point on the calibration board at each time step based on the spatial and temporal relationship of the poses of the two cameras and the two calibration boards at each time step, and to determine the second residual of the camera at each time step based on the observed value and the second estimated value of the pixel coordinates of each corner point on the calibration board, wherein the temporal relationship includes the pose change of the camera between any two time steps. The residual determination module is also used to obtain the total residual based on the first residual and the second residual of the two cameras at each time point; The optimization module is used to optimize with the total residual as the objective and solve for the extrinsic parameters of the two cameras.

16. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store a computer program, the processor being used to invoke and run the computer program stored in the memory to perform the method of any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method as described in any one of claims 1 to 14.

18. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 14.

Images