Camera calibration method and device, electronic equipment and storage medium
By stitching and optimizing the photometric error of the camera's image to be stitched, the imaging offset and artifact problems caused by camera pose changes are solved, and the camera's recalibration efficiency and image fusion quality are improved.
Patent Information
- Application Number
- CN202410922560.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-10
- Publication Date
- 2024-11-12
AI Technical Summary
The prior art causes adjacent camera imaging offsets when camera pose changes, and misalignment and artifacts appear in the fusion area, which requires recalibration, but this process is inefficient and depends on special calibration sites and professionals.
By splicing the images to be stitched by adjacent cameras, the common-view area images are separated, the pixel coordinates of the reference target are determined, coordinate system transformation and photometric error optimization are performed, the camera external parameters are updated until the preset error conditions are met, and artifacts are reduced And improve image fusion effect.
Without the need for complete calibration scenarios and professionals, use the reference targets in the current environment to optimize the camera external parameters, reduce cumbersome operations, improve camera recalibration efficiency, eliminate artifacts, and optimize image fusion effect.
Smart Images

Figure CN118941639A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a camera calibration method, device, electronic equipment and storage medium. Background Art
[0002] Image stitching aims to stitch multiple images into a whole image to show a wider scene or provide richer information, such as panoramic image stitching, surround view image stitching, and drone image stitching. Taking surround view image stitching as an example, the images collected by the on-board surround view camera can be stitched to generate a surround view bird's-eye view map. The surround view bird's-eye view map allows the driver or the autonomous driving system to understand the situation around the vehicle more clearly, which helps to improve driving safety and comfort.
[0003] However, when the position and posture of a camera (such as a car-mounted surround view camera) changes, the camera's imaging will also shift, causing misalignment in the fusion part of the common view area between adjacent cameras. That is, pixels in the fusion area that were originally aligned or had very small errors have larger and non-negligible errors, resulting in artifacts. At this time, the camera usually needs to be recalibrated. However, this usually requires special calibration sites, facilities and professionals, resulting in poor recalibration efficiency. Summary of the invention
[0004] The problem solved by the present invention is how to improve the recalibration efficiency of a camera.
[0005] In order to solve the above problems, the present invention provides a camera calibration method, device, electronic device and storage medium.
[0006] In a first aspect, the present invention provides a camera calibration method, comprising:
[0007] Stitching the images to be stitched from adjacent cameras to generate a stitched image containing a common viewing area of the adjacent cameras, wherein the adjacent cameras include two or more cameras with overlapping shooting areas;
[0008] Separating a common view area image from the stitched image, and determining pixel coordinates of a reference target located in the common view area according to the common view area image;
[0009] The pixel coordinates are transformed into a coordinate system to determine the pixel values corresponding to the pixel coordinates in each of the images to be stitched, and the camera extrinsic parameters of the adjacent cameras are optimized according to the photometric errors between the pixel values to determine the camera extrinsic parameters that meet the preset error conditions.
[0010] Optionally, determining pixel coordinates of a reference target located in the common view area according to the common view area image includes:
[0011] The pixel coordinates of the frame area of the reference object are extracted from the common viewing area image.
[0012] Optionally, after determining the pixel coordinates of a reference target located in the common view area according to the common view area image, the method further includes:
[0013] Determine a pixel gradient of the common view area image, and filter pixels of the common view area image according to the pixel gradient;
[0014] And / or, determining the distance between each pixel in the common viewing area image and the frame area, and filtering the pixels of the common viewing area image according to the distance.
[0015] Optionally, performing coordinate system transformation on the pixel coordinates to determine pixel values corresponding to the pixel coordinates in each of the images to be stitched includes:
[0016] Back-projecting the pixel coordinates into the camera coordinate system of the adjacent camera to determine the camera space coordinates corresponding to the pixel coordinates;
[0017] Convert the camera space coordinates into world coordinates in three-dimensional space by inverse extrinsic transformation;
[0018] The world coordinates are converted into the camera coordinate system, and then projected from the camera coordinate system to the pixel coordinate system corresponding to each of the images to be stitched, to obtain the target coordinates of the pixel coordinates in the pixel coordinate system corresponding to each of the images to be stitched;
[0019] Determine the pixel value corresponding to the target coordinate in each of the images to be stitched.
[0020] Optionally, the optimizing the camera extrinsic parameters of the adjacent cameras according to the photometric errors between the pixel values includes:
[0021] Determine a corresponding L1 norm according to the photometric error;
[0022] Solving an overdetermined equation formed by pixel coordinates of each pixel in the common viewing area image to determine an external parameter disturbance;
[0023] updating a disturbance amount according to the extrinsic parameter disturbance, and updating the camera extrinsic parameter according to the L1 norm and the disturbance amount to obtain a new camera extrinsic parameter;
[0024] The photometric error is re-determined according to the new camera extrinsic parameter, and the above steps are iterated and looped until the photometric error corresponding to the determined camera extrinsic parameter is less than a preset threshold.
[0025] Optionally, solving the overdetermined equation formed by the pixel coordinates of each pixel in the common viewing area image includes:
[0026] A Jacobian matrix and a Hessian matrix are constructed to solve the overdetermined equation, wherein the Jacobian matrix is used to describe the local linear approximation of the overdetermined equation, and the Hessian matrix is used for the overdetermined equation to converge to an optimal solution.
[0027] Optionally, the camera calibration method further includes:
[0028] Performing scaling processing on the common viewing area image to determine a multi-layer image;
[0029] The multi-level image is processed layer by layer to optimize the camera extrinsic parameters layer by layer.
[0030] Optionally, the step of stitching the images to be stitched from adjacent cameras to generate a stitched image including a common viewing area of the adjacent cameras includes:
[0031] Back-projecting the images to be stitched into the corresponding camera coordinate system to obtain an intermediate image;
[0032] The intermediate image is converted into a corresponding pixel coordinate system according to a preset camera intrinsic parameter to obtain a target image, wherein the preset camera intrinsic parameter is an intrinsic parameter of a virtual camera with a preset lens orientation;
[0033] The target image is subjected to image stitching and common view area fusion to obtain the stitched image.
[0034] In a second aspect, the present invention provides a camera calibration device, comprising:
[0035] A generation module, used for stitching the images to be stitched from adjacent cameras to generate a stitched image containing a common viewing area of the adjacent cameras, wherein the adjacent cameras include two or more cameras with overlapping shooting areas;
[0036] A processing module, used to separate a common view area image from the stitched image, and determine the pixel coordinates of a reference target located in the common view area according to the common view area image;
[0037] The calibration module is used to perform coordinate system transformation on the pixel coordinates, determine the pixel values corresponding to the pixel coordinates in each of the images to be stitched, and optimize the camera extrinsic parameters of the adjacent cameras according to the photometric errors between the pixel values to determine the camera extrinsic parameters that meet the preset error conditions.
[0038] In a third aspect, the present invention provides an electronic device, including a memory and a processor;
[0039] The memory is used to store computer programs;
[0040] The processor is used to implement the camera calibration method as described above when executing the computer program.
[0041] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the camera calibration method as described above is implemented.
[0042] After stitching the images of adjacent cameras to generate a stitched image containing the common view area of the adjacent cameras, the common view area image is separated from the stitched image, and subsequent processing can be performed on only the common view area image to reduce the amount of calculation; then, the pixel coordinates of a reference target located in the common view area are determined according to the common view area image, for example, edge textures with obvious features in the reference target can be determined, which is conducive to calculating the photometric error at corresponding pixel positions between adjacent cameras without requiring all pixel positions in the common view area to participate in the calculation; finally, the pixel coordinates are transformed into a coordinate system to determine the pixel values at positions corresponding to the pixel coordinates in each image to be stitched, and the adjacent cameras (i.e., the images photographing the images to be stitched) are transformed according to the photometric errors between the pixel values. The camera extrinsic parameters of each camera of the image are optimized to determine the camera extrinsic parameters that meet the preset error conditions, which can reduce the pixel error of the common view area of adjacent cameras, eliminate the artifacts in the common view area, and thus optimize the image fusion effect; and the present invention can be applied in scenarios without complete calibration (for example, without the help of a professional calibration chessboard), that is, it can use the favorable conditions in the current environment to find reference targets with rich textures, such as lane lines, zebra crossings, and roadside lines, to complete the camera recalibration process. For example, in outdoor scenarios, ordinary and common road reference targets can be used as a medium to complete the camera extrinsic optimization, thereby reducing the requirements for cumbersome operations, sites, and professionals, and improving the camera recalibration efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 Schematic diagram of the camera calibration method according to an embodiment of the present invention;
[0044] Figure 2 The image fusion result when the camera imaging is offset according to the embodiment of the present invention;
[0045] Figure 3 is the image fusion result after the camera is recalibrated in the embodiment of the present invention;
[0046] Figure 4 is a system architecture diagram of a camera calibration device according to an embodiment of the present invention;
[0047] Figure 5 4 is a system architecture diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0048] In the field of image processing, existing technologies usually use feature point methods to perform image stitching, and the specific process includes:
[0049] (1) Identify and extract feature points and descriptors in two views;
[0050] (2) Match the feature points in the two views and solve the homography matrix;
[0051] (3) Fix one of the views as the reference view, such as the left view, and project the other view onto the plane of the left view to complete the stitching;
[0052] (4) Optimize the homography matrix by minimizing the reprojection error.
[0053] Among them, by identifying and extracting the corner points of the chessboard, calculating the coordinates of the corner points in the camera coordinate system, and measuring the coordinates of the actual chessboard corner points in the world coordinate system, the corresponding homography matrix is solved, and then the rotation matrix and translation vector, i.e., the camera extrinsic parameters, are decomposed; the above process is generally referred to as the chessboard calibration process. Taking the surround image stitching of the vehicle as an example, after the calibration is completed, the surround bird's-eye view around the car can be stitched and fused. However, after a long period of driving time, the camera under the rearview mirror may be offset or the fusion area may be misaligned due to distortion, which will make the imaging of adjacent cameras in the common view area blurred, i.e., artifacts.
[0054] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be interpreted as being limited to the embodiments described herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.
[0055] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.
[0056] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to"; the term "based on" means "based at least in part on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments"; the term "optionally" means "optional embodiments". The relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0057] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0058] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.
[0059] like Figure 1 As shown, a camera calibration method provided by an embodiment of the present invention includes:
[0060] S100: stitching the images to be stitched from adjacent cameras to generate a stitched image that includes a common viewing area of the adjacent cameras, wherein the adjacent cameras include two or more cameras with overlapping shooting areas.
[0061] Specifically, the images to be stitched of adjacent cameras are stitched to obtain a stitched image, which includes the common view area of adjacent cameras, wherein the adjacent cameras are two or more cameras with overlapping shooting areas, and the images to be stitched of adjacent cameras represent images shot by cameras with overlapping shooting areas, or in other words, there is an overlapping area between the images to be stitched; for example, in the camera model, the world coordinates of the reference target (also called scene target; for example, targets with obvious texture features on the road such as lane lines or zebra crossings) are transformed to the camera coordinate system through the initial camera extrinsic parameters (taking the on-board surround view camera as an example, the surround view camera has been calibrated with respect to the ground corner points through a checkerboard when the vehicle leaves the factory), and then transformed to the pixel coordinate system through the camera intrinsic parameters, that is, a projection is obtained; the pixel coordinates are back-projected to the camera coordinate system, transformed to the pixel coordinate system through the intrinsic parameters of the overhead virtual camera, and converted into a bird's-eye view of the reference target; after the adjacent cameras complete the bird's-eye view transformation, the images are stitched (including the fusion process of the common view area).
[0062] Among them, the common view area of adjacent cameras refers to the area that can be observed simultaneously in the field of view of two or more cameras. In a multi-camera system, in order to achieve effective data fusion and joint processing, the common view area of adjacent cameras usually needs to have sufficient overlap, and the difference in viewing angle between cameras should be as small as possible.
[0063] Among them, image stitching refers to merging two or more partially overlapping but not completely overlapping images into a complete image through matching, registration, superposition and other technologies, to ensure that the stitched image is visually coherent without obvious misalignment or discontinuity; the fusion of common view areas is part of the image stitching process, which refers to the merging or fusing of overlapping information in multiple views to eliminate duplicate information, improve image quality or reduce artifacts. It can solve problems such as inconsistent overlapping areas and uneven lighting that may arise in the image stitching process, thereby improving the quality and realism of the stitching results.
[0064] The fusion process of the common view area may include the following steps:
[0065] (1) Feature extraction and matching: For images in the common view area, feature extraction and matching are required before stitching. This includes extracting key features such as feature points and descriptors in the common view area and using a matching algorithm to find the corresponding feature points in the two images.
[0066] (2) Image registration: After feature matching is completed, adjacent images need to be registered, that is, they need to be aligned into a common reference coordinate system; this can be achieved by calculating the geometric transformation between images (such as translation, rotation, scaling, etc.). In the common view area, perspective transformation or affine transformation is usually used to register images.
[0067] (3) Pixel value fusion: After registration, the pixel values in the common view area need to be fused to eliminate discontinuities or overlaps at the joints. Common fusion methods include: mean fusion: taking the average of the pixel values in the overlapping parts; weighted fusion: assigning different weights to pixels of different images based on information such as pixel position and brightness, and then performing weighted averaging; hybrid fusion: using image fusion algorithms (such as Laplacian pyramid, multi-resolution fusion, etc.) to fuse images to achieve a smoother transition effect.
[0068] (4) Edge transition processing: The edge of the common view area needs special processing to ensure a natural edge transition after stitching. Common processing methods include using gradient, pixel blending, image fusion and other technologies to achieve a smooth edge transition effect.
[0069] The camera calibration method of this embodiment can be applied to a fisheye camera (which can increase the field of view of the camera) or a pinhole camera, etc.
[0070] S200 , separating a common view area image from the stitched image, and determining pixel coordinates of a reference target located in the common view area according to the common view area image.
[0071] Specifically, the common view area of adjacent cameras is cropped out, that is, the common view area image is separated from the stitched image. After the common view area image is grayed and binarized, the contour coordinates of the reference target located in the common view area (for example, the pixel coordinates of the border area of the reference target) can be extracted through the findContours method of OpenCV.
[0072] The common viewing area image refers to the image of the overlapping part that can be observed by two adjacent cameras in their fields of view, usually a superimposed image of the overlapping part of the fields of view of the two cameras, which is used to reflect the scene information captured by the two cameras in the overlapping area.
[0073] S300, performing coordinate system transformation on the pixel coordinates, determining pixel values corresponding to the pixel coordinates in each of the images to be stitched, and optimizing camera extrinsic parameters of the adjacent cameras according to photometric errors between the pixel values to determine camera extrinsic parameters that meet preset error conditions.
[0074] Specifically, the pixel coordinates can be transformed into a coordinate system according to the initial calibration parameters of the adjacent cameras (including the calibrated initial camera extrinsic parameters and initial camera intrinsic parameters, etc.), so as to determine the pixel values corresponding to the pixel coordinates in each image to be stitched, and then the camera extrinsic parameters of the adjacent cameras are optimized according to the photometric error of the pixel values to determine the camera extrinsic parameters (pose) that meet the preset error conditions. Figure 2 and Figure 3 As shown, by reducing the pixel error in the common view area of adjacent cameras (i.e., minimizing the error at the pixel level and optimizing the position and posture of the on-board surround view camera), the artifacts in the common view area are eliminated, thereby optimizing the image fusion effect.
[0075] Among them, this embodiment can be applied in a situation where a complete calibration scenario is not available (for example, without the help of a professional calibration checkerboard), that is, it can use favorable conditions in the current environment to find reference targets with rich textures, such as lane lines, zebra crossings, and roadside lines, to complete the recalibration process of cameras such as vehicle-mounted surround view cameras, reducing the need for cumbersome operations, venues, and professionals.
[0076] Among them, in the bird's-eye view of surround stitching, the focus is on the common viewing area of adjacent cameras. When the image fusion of the common viewing area meets the subjective and objective indicators, the camera calibration process can be considered completed.
[0077] Optionally, determining pixel coordinates of a reference target located in the common view area according to the common view area image includes:
[0078] The pixel coordinates of the frame area of the reference object are extracted from the common viewing area image.
[0079] Specifically, when determining the pixel coordinates of the reference target, the pixel coordinates of the border area of the reference target can be extracted from the common view area image, that is, the edge texture with obvious characteristics of the reference target is extracted, which is conducive to calculating the photometric error at the corresponding pixel positions between adjacent cameras without involving all pixel positions in the common view area in the calculation.
[0080] In this optional embodiment, by extracting the pixel coordinates of the border area of the reference target from the common view area image, the edge texture with obvious characteristics of the reference target can be extracted, which is beneficial to calculating the photometric error at the corresponding pixel positions between adjacent cameras without requiring all pixel positions in the common view area to participate in the calculation, which is beneficial to reducing computing power consumption.
[0081] Optionally, after determining the pixel coordinates of a reference target located in the common view area according to the common view area image, the method further includes:
[0082] Determine a pixel gradient of the common view area image, and filter pixels of the common view area image according to the pixel gradient;
[0083] And / or, determining the distance between each pixel in the common viewing area image and the frame area, and filtering the pixels of the common viewing area image according to the distance.
[0084] Specifically, after determining the pixel coordinates of the reference target, the corresponding pixel coordinates need to be filtered, for example:
[0085] (1) Determine the pixel gradient of the common view area image, exclude the smooth area with a small gradient, and filter the pixels of the common view area image, thereby reducing the number of pixel coordinates of the weakly correlated common view area image.
[0086] (2) Determine the distance between the pixels in the common view area image and the border area, exclude pixels that are far away and at the edge, and filter the pixels in the common view area image, thereby reducing the number of pixel coordinates in the weakly correlated common view area.
[0087] In this optional embodiment, filtering the pixels of the common view area image according to the pixel gradient can reduce the number of pixel coordinates of the weakly correlated common view area and improve the accuracy of the camera calibration method.
[0088] Optionally, performing coordinate system transformation on the pixel coordinates to determine pixel values corresponding to the pixel coordinates in each of the images to be stitched includes:
[0089] The pixel coordinates are back-projected into the camera coordinate system of the adjacent camera to determine the camera space coordinates corresponding to the pixel coordinates.
[0090] Specifically, the pixel coordinates are first back-projected to the camera coordinate system of the adjacent camera, and the camera space coordinates corresponding to the pixel coordinates can be obtained through coordinate transformation; among them, the camera coordinate system is a coordinate system defined relative to the camera itself, and the origin is usually located at the optical center of the camera. The world coordinates of the object can be converted to the camera coordinate system through the external parameters of the camera.
[0091] The camera space coordinates are converted into world coordinates in three-dimensional space through an inverse extrinsic transformation.
[0092] Specifically, after determining the camera space coordinates, the camera space coordinates are converted into world coordinates in three-dimensional space through an inverse extrinsic transformation; wherein the world coordinate system is a global coordinate system used to describe the position of objects in three-dimensional space.
[0093] The world coordinates are converted into the camera coordinate system, and then projected from the camera coordinate system to the pixel coordinate system corresponding to each of the images to be stitched, to obtain the target coordinates of the pixel coordinates in the pixel coordinate system corresponding to each of the images to be stitched.
[0094] Specifically, after determining the world coordinates, the world coordinates are converted to the camera coordinate system, and then projected from the camera coordinate system to the pixel coordinate system corresponding to each image to be stitched, so that the target coordinates of the pixel coordinates in the pixel coordinate system corresponding to each image to be stitched can be obtained; wherein the pixel coordinate system is used to represent the pixel position in the image, and each pixel coordinate represents a discrete position in the image.
[0095] Determine the pixel value corresponding to the target coordinate in each of the images to be stitched.
[0096] Specifically, after the target coordinates are determined, the pixel values corresponding to the target coordinates in each of the images to be stitched may be acquired according to the target coordinates.
[0097] When the pixel coordinates of the bird's-eye view are transformed into the world coordinate system, the height is set to 1m by default (because the height is unknown when the pixel coordinates are back-projected into the world coordinates, giving the height is helpful for calculating the three-dimensional world coordinates).
[0098] In this optional embodiment, the pixel coordinates are converted between the camera coordinate system, the world coordinate system and the pixel coordinate system according to the initial calibration parameters of the adjacent cameras, so that the pixel values corresponding to the target coordinates in each image to be stitched can be obtained, which is beneficial to the subsequent solution of the photometric error of the pixel value. Compared with the existing optical flow method, it can reduce the amount of calculation, thereby reducing computing power consumption.
[0099] Optionally, the optimizing the camera extrinsic parameters of the adjacent cameras according to the photometric errors between the pixel values includes:
[0100] A corresponding L1 norm is determined according to the photometric error.
[0101] Specifically, the L1 norm of the photometric error of the pixel values at corresponding coordinates of adjacent cameras is first determined (used to measure the degree of difference between two images, usually expressed as the sum of the absolute values of the differences between the pixel values at corresponding positions of the two images), and the L1 norm is used for the photometric error calculation after the next external parameter optimization.
[0102] An overdetermined equation formed by the pixel coordinates of each pixel in the common viewing area image is solved to determine the external parameter disturbance.
[0103] Specifically, an overdetermined equation (the number of equations is greater than the number of unknowns) formed by the pixel coordinates of each pixel in the common viewing area image is solved to determine the external parameter disturbance; by constructing an overdetermined equation, the optimization time can be shortened.
[0104] The disturbance amount is updated according to the extrinsic parameter disturbance, and the camera extrinsic parameter is updated according to the L1 norm and the disturbance amount to obtain a new camera extrinsic parameter.
[0105] Specifically, the disturbance amount is updated according to the extrinsic parameter disturbance, a new disturbance is left-multiplied on the extrinsic parameter, and the camera extrinsic parameter is updated according to the L1 norm and the updated extrinsic parameter disturbance to obtain a new camera extrinsic parameter.
[0106] The photometric error is re-determined according to the new camera extrinsic parameter, and the above steps are iterated and looped until the photometric error corresponding to the determined camera extrinsic parameter is less than a preset threshold.
[0107] Specifically, after determining the new camera extrinsic parameters, the photometric error is re-determined according to the new camera extrinsic parameters, and the above steps are iterated until the photometric error corresponding to the determined camera extrinsic parameters is less than a preset threshold value; in the iterative loop, the camera extrinsic parameters corresponding to the photometric error with a smaller error can be continuously saved, and the global optimal solution can be retained even if a local optimal solution appears during the iteration process. In addition, even if some disturbances or abnormal situations occur during the iteration process, stable results can be retained to avoid the algorithm from failing due to local fluctuations.
[0108] Among them, the L1 norm is used to measure the difference between the grayscale values or color values of the two images in the calculation of the photometric error. For example, the difference between the pixel values of the two images is summed up by the L1 norm to calculate the photometric error at each pixel position. In each iteration, the photometric error is calculated based on the estimated value of the current camera extrinsic parameter, and the perturbation of the extrinsic parameter is updated to minimize the photometric error.
[0109] In this optional embodiment, the camera extrinsic parameters corresponding to the photometric error that meets the preset threshold are determined in an iterative manner, which can retain the global optimal solution and retain stable results under abnormal conditions, thereby avoiding the algorithm from failing due to local fluctuations.
[0110] Optionally, solving the overdetermined equation formed by the pixel coordinates of each pixel in the common viewing area image includes:
[0111] A Jacobian matrix and a Hessian matrix are constructed to solve the overdetermined equation, wherein the Jacobian matrix is used to describe the local linear approximation of the overdetermined equation, and the Hessian matrix is used for the overdetermined equation to converge to an optimal solution.
[0112] Specifically, the derivatives of the expression with respect to the camera pose are analyzed, and the first-order derivative matrix (such as the Jacobian matrix) and the second-order derivative matrix (such as the Hessian matrix) are constructed. The overdetermined equations can be solved using the ldlt interface of OpenCV (for example, using the least squares method).
[0113] The Jacobian matrix is a matrix of first-order derivatives of a function, which is usually used to describe the local linear approximation of a multivariable function, thus providing the basis for the least squares method, for example, in the least squares method, an overdetermined equation is fitted by minimizing the sum of squares of the residuals (the difference between the observed values and the model predicted values).
[0114] Among them, the Hessian matrix is the second-order derivative matrix of a function, which is usually used in optimization algorithms. By providing higher-order information, it helps overdetermined equations converge to the optimal solution faster.
[0115] In this optional embodiment, the Jacobian matrix and the Hessian matrix are used to solve the overdetermined equation, which can effectively shorten the external parameter optimization time and improve the camera calibration efficiency.
[0116] Optionally, the camera calibration method further includes:
[0117] The common viewing area image is scaled to determine a multi-layer image.
[0118] Specifically, the image in the common view area is scaled using the image pyramid principle to obtain images at different levels.
[0119] Among them, the principle of image pyramid is to use the multi-scale information of the image for analysis and processing. By constructing a pyramid, the characteristics of the image can be observed and analyzed at different scales, thereby improving the efficiency and robustness of image processing; through the Gaussian pyramid (generated by downsampling at each level, the image of each level is half the size of the image of the previous level), the overall structure and contour characteristics of the image can be observed at different scales, and through the Laplacian pyramid (generated by the Gaussian pyramid, the image of each level is the result of subtracting the image of the previous level from the image of the previous level after upsampling), the detailed information and edge features of the image can be obtained.
[0120] The multi-level image is processed layer by layer to optimize the camera extrinsic parameters layer by layer.
[0121] Specifically, the multi-level images are processed layer by layer, and then the camera extrinsic parameters corresponding to the multi-level images are optimized layer by layer, that is, the camera extrinsic parameters are optimized layer by layer (changing the image resolution). For each layer of images, the camera extrinsic parameters of adjacent cameras are optimized according to the photometric error between the pixel values corresponding to the pixel coordinates in each image to be stitched, so as to obtain the camera extrinsic parameters that meet the preset error conditions, so that the final optimization effect is optimal.
[0122] During the scaling process, the camera internal parameters also need to be scaled synchronously.
[0123] In this optional embodiment, by optimizing the camera extrinsic parameters corresponding to the multi-level images layer by layer, the optimization effect of images of different levels and scales can be optimized, which is beneficial to improving the accuracy of the camera calibration method.
[0124] Optionally, the step of stitching the images to be stitched from adjacent cameras to generate a stitched image including a common viewing area of the adjacent cameras includes:
[0125] The images to be stitched are back-projected to the corresponding camera coordinate system to obtain an intermediate image.
[0126] Specifically, the images to be stitched are first back-projected to the corresponding camera coordinate system to obtain an intermediate image.
[0127] The intermediate image is converted into a corresponding pixel coordinate system according to preset camera intrinsic parameters to obtain a target image, wherein the preset camera intrinsic parameters are intrinsic parameters of a virtual camera with a preset lens orientation.
[0128] Specifically, after the intermediate image is determined, the intermediate image is converted to a corresponding pixel coordinate system according to preset camera intrinsic parameters (intrinsic parameters of a virtual camera with a preset lens orientation), thereby obtaining a target image (eg, a bird's-eye view).
[0129] The target image is subjected to image stitching and common view area fusion to obtain the stitched image.
[0130] Specifically, after the target image is determined, image stitching and common view area fusion are performed on the target image (the common view area fusion may be included in the image stitching), so as to obtain a stitched image.
[0131] like Figure 4 As shown, a camera calibration device 400 provided in an embodiment of the present invention includes:
[0132] A generating module 410 is used to stitch the images to be stitched from adjacent cameras to generate a stitched image including a common viewing area of the adjacent cameras, wherein the adjacent cameras include two or more cameras with overlapping shooting areas;
[0133] A processing module 420 is used to separate a common view area image from the stitched image, and determine the pixel coordinates of a reference target located in the common view area according to the common view area image;
[0134] The calibration module 430 is used to perform coordinate system transformation on the pixel coordinates, determine the pixel values corresponding to the pixel coordinates in each of the images to be stitched, and optimize the camera extrinsic parameters of the adjacent cameras according to the photometric errors between the pixel values to determine the camera extrinsic parameters that meet the preset error conditions.
[0135] like Figure 5 As shown, an electronic device 500 provided by an embodiment of the present invention includes a memory 510 and a processor 520; the memory 510 is used to store a computer program; the processor 520 is used to implement the camera calibration method as described above when executing the computer program.
[0136] In other words, an electronic device 500 includes a memory 510 and a processor 520 coupled to the memory 510; the memory 510 is configured to store a computer program; and the processor 520 is configured to perform the following operations when executing the computer program:
[0137] Stitching the images to be stitched from adjacent cameras to generate a stitched image containing a common viewing area of the adjacent cameras, wherein the adjacent cameras include two or more cameras with overlapping shooting areas;
[0138] Separating a common view area image from the stitched image, and determining pixel coordinates of a reference target located in the common view area according to the common view area image;
[0139] The pixel coordinates are transformed into a coordinate system to determine the pixel values corresponding to the pixel coordinates in each of the images to be stitched, and the camera extrinsic parameters of the adjacent cameras are optimized according to the photometric errors between the pixel values to determine the camera extrinsic parameters that meet the preset error conditions.
[0140] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the camera calibration method as described above is implemented.
[0141] In other words, a non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor performs the following operations:
[0142] Stitching the images to be stitched from adjacent cameras to generate a stitched image containing a common viewing area of the adjacent cameras, wherein the adjacent cameras include two or more cameras with overlapping shooting areas;
[0143] Separating a common view area image from the stitched image, and determining pixel coordinates of a reference target located in the common view area according to the common view area image;
[0144] The pixel coordinates are transformed into a coordinate system to determine the pixel values corresponding to the pixel coordinates in each of the images to be stitched, and the camera extrinsic parameters of the adjacent cameras are optimized according to the photometric errors between the pixel values to determine the camera extrinsic parameters that meet the preset error conditions.
[0145] An electronic device 500 that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device 500 is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 500 can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present invention described and / or required herein.
[0146] The electronic device 500 includes a computing unit, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) or a computer program loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for the operation of the device can also be stored. The computing unit, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.
[0147] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc. In the present application, the unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present invention. In addition, each functional unit in each embodiment of the present invention can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.
[0148] Although the present invention is disclosed as above, the protection scope of the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the protection scope of the present invention.
Claims
1. A camera calibration method, characterized in that: include: Stitching the images to be stitched from adjacent cameras to generate a stitched image containing a common viewing area of the adjacent cameras, wherein the adjacent cameras include two or more cameras with overlapping shooting areas; Separating a common view area image from the stitched image, and determining pixel coordinates of a reference target located in the common view area according to the common view area image; The pixel coordinates are transformed into a coordinate system to determine the pixel values corresponding to the pixel coordinates in each of the images to be stitched, and the camera extrinsic parameters of the adjacent cameras are optimized according to the photometric errors between the pixel values to determine the camera extrinsic parameters that meet the preset error conditions.
2. The camera calibration method according to claim 1, characterized in that: Determining the pixel coordinates of the reference target located in the common viewing area according to the common viewing area image includes: The pixel coordinates of the frame area of the reference object are extracted from the common viewing area image.
3. The camera calibration method according to claim 2, characterized in that: After determining the pixel coordinates of the reference target located in the common viewing area according to the common viewing area image, the method further includes: Determine a pixel gradient of the common view area image, and filter pixels of the common view area image according to the pixel gradient; And / or, determining the distance between each pixel in the common viewing area image and the frame area, and filtering the pixels of the common viewing area image according to the distance.
4. The camera calibration method according to claim 1, characterized in that: The performing coordinate system transformation on the pixel coordinates to determine the pixel values corresponding to the pixel coordinates in each of the images to be stitched comprises: Back-projecting the pixel coordinates into the camera coordinate system of the adjacent camera to determine the camera space coordinates corresponding to the pixel coordinates; Convert the camera space coordinates into world coordinates in three-dimensional space by inverse extrinsic transformation; The world coordinates are converted into the camera coordinate system, and then projected from the camera coordinate system to the pixel coordinate system corresponding to each of the images to be stitched, to obtain the target coordinates of the pixel coordinates in the pixel coordinate system corresponding to each of the images to be stitched; Determine the pixel value corresponding to the target coordinate in each of the images to be stitched.
5. The camera calibration method according to claim 4, characterized in that: The optimizing the camera extrinsic parameters of the adjacent cameras according to the photometric errors between the pixel values includes: Determine a corresponding L1 norm according to the photometric error; Solving an overdetermined equation formed by pixel coordinates of each pixel in the common viewing area image to determine an external parameter disturbance; updating a disturbance amount according to the extrinsic parameter disturbance, and updating the camera extrinsic parameter according to the L1 norm and the disturbance amount to obtain a new camera extrinsic parameter; The photometric error is re-determined according to the new camera extrinsic parameter, and the above steps are iterated and looped until the photometric error corresponding to the determined camera extrinsic parameter is less than a preset threshold.
6. The camera calibration method according to claim 5, characterized in that: The solving of the overdetermined equation formed by the pixel coordinates of each pixel in the common viewing area image includes: A Jacobian matrix and a Hessian matrix are constructed to solve the overdetermined equation, wherein the Jacobian matrix is used to describe the local linear approximation of the overdetermined equation, and the Hessian matrix is used for the overdetermined equation to converge to an optimal solution.
7. The camera calibration method according to any one of claims 1 to 6, characterized in that: Also includes: Performing scaling processing on the common viewing area image to determine a multi-layer image; The multi-level image is processed layer by layer to optimize the camera extrinsic parameters layer by layer.
8. The camera calibration method according to any one of claims 1 to 6, characterized in that: The step of stitching the images to be stitched from adjacent cameras to generate a stitched image including the common viewing area of the adjacent cameras includes: Back-projecting the images to be stitched into the corresponding camera coordinate system to obtain an intermediate image; The intermediate image is converted into a corresponding pixel coordinate system according to a preset camera intrinsic parameter to obtain a target image, wherein the preset camera intrinsic parameter is an intrinsic parameter of a virtual camera with a preset lens orientation; The target image is subjected to image stitching and common view area fusion to obtain the stitched image.
9. A camera calibration device, characterized in that: include: A generation module, used for stitching the images to be stitched from adjacent cameras to generate a stitched image containing a common viewing area of the adjacent cameras, wherein the adjacent cameras include two or more cameras with overlapping shooting areas; A processing module, used to separate a common view area image from the stitched image, and determine the pixel coordinates of a reference target located in the common view area according to the common view area image; The calibration module is used to perform coordinate system transformation on the pixel coordinates, determine the pixel values corresponding to the pixel coordinates in each of the images to be stitched, and optimize the camera extrinsic parameters of the adjacent cameras according to the photometric errors between the pixel values to determine the camera extrinsic parameters that meet the preset error conditions.
10. An electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the camera calibration method according to any one of claims 1 to 8 when executing the computer program.
11. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the camera calibration method according to any one of claims 1 to 8 is implemented.
Citation Information
Patent Citations
Tiled splicing method based on corrected scanning image
CN114463186A
Method and device for correcting external parameters of camera
CN115546312A
Vehicle-mounted panoramic look-around automatic calibration method, system and device and storage medium
CN116416321A
External parameter calibration method and device for fisheye camera, equipment and medium
CN117611678A
Joint calibration method, system and equipment for vehicle-mounted camera and medium
CN117611688A
Cited By
Camera external parameter calibration result correction method and system based on OpenCV Eigen
CN119379815A
