Imaging device calibration method, equipment, storage medium and product

Abstract

One or more embodiments of the invention provide an imaging device calibration method, equipment, a storage medium and a product. The method comprises the following steps: acquiring a plurality of calibration images, wherein the calibration images are obtained by shooting a calibration pattern by a to-be-calibrated imaging device; establishing a pairing relationship between a three-dimensional coordinate and a two-dimensional coordinate of a feature point in the calibration pattern, wherein the two-dimensional coordinate is the two-dimensional coordinate of the feature point in the calibration image; calibrating the imaging device by using the pairing relationship to obtain initial internal and external parameters of the imaging device; according to the method, the initial internal and external parameters of the imaging device are optimized by taking minimization of the re-projection error and constraint of the optical center deviation as optimization objectives and utilizing the pairing relationship, so that the physical rationality and the global fitting effect of the optimized internal and external parameters are improved.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of calibration technology, and in particular, to a calibration method for an imaging device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the rapid development of imaging technology, imaging devices have been widely used in the fields of industry, medicine, autonomous driving, and consumer electronics. In order to ensure the high-precision imaging effect of the imaging device during use, it is a key step to calibrate its imaging parameters (i.e., internal and external parameters). During the calibration process, noise and systematic deviations are inevitably present in the calibration data, which may lead to errors in the parameter estimation results and even cause the optimization process to fall into a local optimal solution, thereby reducing the calibration accuracy. Summary of the invention

[0003] In view of this, one or more embodiments of the present specification provide a calibration method for an imaging device, an electronic device, a computer-readable storage medium, and a computer program product.

[0004] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0005] According to a first aspect of one or more embodiments of this specification, a calibration method for an imaging device is provided, comprising:

[0006] Acquire multiple calibration images, where the calibration images are obtained by photographing the calibration pattern by the imaging device to be calibrated;

[0007] Establishing a pairing relationship between the three-dimensional coordinates and the two-dimensional coordinates of the feature point in the calibration pattern, wherein the two-dimensional coordinates are the two-dimensional coordinates of the feature point in the calibration image;

[0008] Using the pairing relationship, calibrating the imaging device to obtain initial internal and external parameters of the imaging device;

[0009] Taking minimizing the reprojection error and constraining the optical center deviation as the optimization goal, the initial internal and external parameters of the imaging device are optimized by utilizing the pairing relationship.

[0010] According to a second aspect of the embodiments of this specification, there is provided an electronic device, including:

[0011] processor;

[0012] a memory for storing processor-executable instructions;

[0013] When the processor executes the executable instructions, it is used to implement the method described in the first aspect.

[0014] According to a third aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0015] According to a fourth aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.

[0016] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0017] In the embodiments of this specification, a plurality of calibration images are first obtained. The calibration images are obtained by photographing the calibration pattern by the imaging device to be calibrated, which lays the foundation for the subsequent calculation of internal and external parameters; then, a pairing relationship is established between the three-dimensional coordinates of the feature points in the calibration pattern and the two-dimensional coordinates of the feature points in the calibration image, so as to associate the real three-dimensional space points with the projection points in the image; then, the imaging device is calibrated using the pairing relationship to obtain the initial internal and external parameters of the imaging device; finally, the initial internal and external parameters of the imaging device are optimized using the pairing relationship with the optimization goal of minimizing the reprojection error and constraining the optical center deviation. With the goal of minimizing the reprojection error and combining the constraint of the optical center deviation, the calibration process can effectively avoid the optimization from falling into the local optimum while reducing the influence of noise, and the optimized optical center position can be closer to the center of the image, improving the physical rationality and global fitting effect of the optimized internal and external parameters, thereby significantly improving the calibration accuracy and stability.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 The present invention is a flowchart of a calibration method for an imaging device provided by an exemplary embodiment.

[0020] Figure 2 The present invention is a flowchart of another calibration method for an imaging device provided by an exemplary embodiment.

[0021] Figure 3 A flowchart for optimizing initial internal and external parameters of an imaging device is provided by an exemplary embodiment.

[0022] Figure 4 It is a schematic structural diagram of an electronic device provided by an exemplary embodiment.

[0023] Figure 5 It is a block diagram of a calibration device for an imaging device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0024] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0025] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0026] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0027] The imaging device mentioned in the embodiments of this specification may be a physical imaging device. The imaging device may be configured to detect electromagnetic radiation (e.g., visible light, infrared light, and / or ultraviolet light) and generate image data based on the detected electromagnetic radiation. The imaging device may include a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor that generates an electrical signal in response to the wavelength of light. The generated electrical signal may be processed to generate image data. The image data generated by the imaging device may include one or more images, and one or more images may be static images (e.g., photos), dynamic images (e.g., videos), or a suitable combination thereof. The image data may be multi-color (e.g., RGB, CMYK, HSV) or monochrome (e.g., grayscale, black, brown). The imaging device may include a lens configured to direct light to the image sensor.

[0028] The imaging device may be a camera. The camera may be a dynamic camera or a video camera that captures dynamic image data (e.g., video). The camera may be a static camera that captures static images (e.g., photos). The camera may capture both dynamic image data and static images. The camera may switch between capturing dynamic image data and static images. Although certain embodiments provided herein are described in the context of a camera, it should be understood that the present disclosure may be applicable to any suitable imaging device, and any description of the camera herein may also be applicable to any suitable imaging device, and any description of the camera herein may also be applicable to other types of imaging devices. A camera may be used to generate multiple 2D images of a 3D scene (e.g., an environment, one or more objects, etc.). These images generated by the camera may represent a projection of the 3D scene onto a 2D image plane. Therefore, each point in the 2D image corresponds to a 3D spatial coordinate in the scene. The camera may include optical elements (e.g., lenses, reflectors, filters, etc.). The camera may capture color images, grayscale images, infrared images, and the like.

[0029] Among them, the calibration of the imaging device is a very important process in computer vision and photogrammetry. Its purpose is to determine the internal and external parameters of the imaging device so that the information of the three-dimensional world can be restored from the two-dimensional image. Through the calibration process, we can understand the internal geometric structure of the imaging device, the distortion characteristics of the lens, and the spatial relationship between the imaging device and the world coordinate system. After calibration, the accuracy of the imaging device in different applications can be improved, especially in the fields of three-dimensional reconstruction, target detection, robot vision, augmented reality, etc.

[0030] The main goal of imaging device calibration is to obtain the following parameters:

[0031] (1) Internal parameters, which describe the internal properties of the imaging device. The function of internal parameters is to map 3D world coordinates to 2D image coordinates. The internal parameters mainly include:

[0032] Focal length: The distance from the optical center of the imaging device to the image plane, usually expressed by the horizontal and vertical components of the focal length.

[0033] Principal point (or optical center): The origin of the image coordinate system, usually the optical center of the imaging device.

[0034] Distortion coefficient: describes the distortion of the imaging device lens, usually including radial distortion and tangential distortion.

[0035] (2) Extrinsic parameters describe the position and orientation of the imaging device in the three-dimensional world, usually represented by a rotation matrix and a displacement vector. Extrinsic parameters are used to transform a three-dimensional point in the world coordinate system into a three-dimensional point in the imaging device coordinate system, and then project it into the image coordinate system through intrinsic parameters.

[0036] In the calibration process of related technologies, noise and systematic deviations are inevitably present in the calibration data, which may lead to errors in parameter estimation results and even cause the optimization process to fall into a local optimal solution, thereby reducing the calibration accuracy. Based on this, the embodiments of this specification provide a calibration method for an imaging device, which can be performed by any electronic device with computing capabilities, including but not limited to servers, virtual servers, tablet computers, personal digital assistants (PDAs), laptop computers, and desktop computers. Please refer to Figure 1 , the method comprising:

[0037] In S101, a plurality of calibration images are acquired, where the calibration images are obtained by photographing a calibration pattern by an imaging device to be calibrated.

[0038] The acquisition of calibration images is the starting point of the entire imaging device calibration process. In this step, a calibration pattern with a specific geometric structure (such as a checkerboard, dot array, or other regular pattern) can be displayed on the display, and the calibration pattern can be photographed using the imaging device to be calibrated to obtain multiple calibration images containing the calibration pattern. Using a display to display the calibration pattern is flexible and repeatable

[0039] Exemplarily, the multiple calibration images are obtained by photographing the calibration pattern displayed on the display by an imaging device in at least one of different positions, different postures and different lighting conditions, so that these calibration images cover the calibration pattern in multiple positions, multiple perspectives and multiple lighting within the camera field of view, ensuring the diversity and coverage of the calibration data, and laying the foundation for subsequent internal and external parameter calculations.

[0040] For example, in order to improve the accuracy of the calibration result, the clarity of the collected calibration image must meet the preset clarity requirement to avoid failure in feature point extraction due to motion blur or loss of focus.

[0041] In S102, a pairing relationship between the three-dimensional coordinates and the two-dimensional coordinates of the feature point in the calibration pattern is established, where the two-dimensional coordinates are the two-dimensional coordinates of the feature point in the calibration image.

[0042] In this step, according to the known characteristics of the calibration pattern in the display, a three-dimensional coordinate system of the feature points in the calibration pattern is defined (usually the display plane is used as the reference plane of the three-dimensional coordinate system). Then, the two-dimensional coordinates of the feature points in the calibration pattern are extracted from the collected calibration image, and a pairing relationship between these two-dimensional coordinates and the three-dimensional coordinates is established. Further, by obtaining the mapping process between the three-dimensional coordinates of the feature points and the two-dimensional coordinates paired therewith (that is, the following calibration process), a projection model of the imaging device (such as a pinhole model) can be obtained. The projection model is a geometric model that describes how the camera maps points in the three-dimensional world to the two-dimensional image plane. The projection model needs to consider internal parameters (such as focal length, optical center position, distortion parameters) and external parameters (relative position and posture of the imaging device and the display coordinate system), that is, the projection model is constructed by the internal and external parameters of the imaging device. At this time, the internal and external parameters are unknown quantities to be calibrated.

[0043] This step establishes the mathematical basis for the calibration process, associating the 3D coordinates of a real 3D space point with the 2D coordinates of its projection point in the image.

[0044] In S103, the imaging device is calibrated using the pairing relationship to obtain initial internal and external parameters of the imaging device.

[0045] The initial internal and external parameters are solved through the pairing relationship established in the previous step, that is, the pairing relationship between the two-dimensional coordinates of each feature point extracted from the calibration image and the known three-dimensional coordinates; the internal parameters include the focal length, optical center position, distortion coefficient and other parameters that describe the imaging characteristics of the camera; the external parameters include the position of the imaging device in three-dimensional space and the translation and rotation posture. This process can usually adopt the camera calibration algorithm in related technologies (such as the DLT method or Zhang Zhengyou calibration method), and preliminarily estimate the internal and external parameters through linear optimization or simple nonlinear optimization, providing an initial solution for subsequent fine optimization.

[0046] In S104, the initial internal and external parameters of the imaging device are optimized by using the pairing relationship with the optimization goal of minimizing the reprojection error and constraining the optical center deviation.

[0047] Based on the initial internal and external parameters, this step takes minimizing the reprojection error as the main optimization goal, and combines the constraints of the optical center position deviation to further optimize the internal and external parameters of the camera. The reprojection error refers to the error between the current two-dimensional coordinates obtained after the three-dimensional coordinates of the feature points of the calibration pattern are projected onto the image plane by the projection model of the imaging device, and their actual two-dimensional coordinates. During the optimization process, the error is minimized by adjusting the internal and external parameters. And further considering the influence of the optical center position on the imaging quality, by constraining the optical center position, it is ensured that the optimized optical center is close to the center of the image, thereby improving the physical rationality and global fitting effect of the calibration results. By comprehensively considering the reprojection error and the optical center deviation, the accuracy and stability of the calibration results are improved, and the imaging error problem that may be caused by excessive optical center position offset can be avoided. It is suitable for scenes that are sensitive to the optical center position.

[0048] In some embodiments, the optical center position in the optical system has an important influence on the imaging quality. The optical center deviation may introduce imaging errors. Especially in a telephoto camera, the optical center deviation problem is more prominent due to its sensitivity to imaging parameters. Therefore, when calibrating a telephoto camera, the calibration method of the imaging device provided in the embodiments of this specification may be used, such as Figure 2 As shown, a calibration method for an imaging device of a telephoto camera is provided, the method comprising:

[0049] In S201, a plurality of calibration images are obtained, where the calibration images are obtained by the imaging device to be calibrated photographing the calibration pattern displayed on the display. Please refer to the description of S101 above, which will not be repeated here.

[0050] In S202, a pairing relationship between the three-dimensional coordinates and the two-dimensional coordinates of the feature point in the calibration pattern is established, where the two-dimensional coordinates are the two-dimensional coordinates of the feature point in the calibration image. See the description of S101 above, which will not be repeated here.

[0051] In S203, the imaging device is calibrated using the pairing relationship to obtain initial internal and external parameters of the imaging device. See the description of S101 above, which will not be repeated here. The initial internal and external parameters of the imaging device include focal length, which serves as a judgment basis for the next step.

[0052] In S204, it is determined whether the focal length of the imaging device meets the telephoto condition.

[0053] Exemplarily, the telephoto condition may be that the focal length of the imaging device is greater than or equal to a preset value, and the preset value includes, for example, 50 mm, 100 mm, or 200 mm, but is not limited thereto.

[0054] In S205, if the focal length of the imaging device meets the telephoto condition, the initial internal and external parameters of the imaging device are optimized by using the pairing relationship with the optimization goal of minimizing the reprojection error and constraining the optical center deviation.

[0055] In S206, if the focal length of the imaging device satisfies the non-telephoto condition, the calibration process ends.

[0056] In this embodiment, in view of the imaging characteristics of the telephoto camera, the calibration process faces problems such as significant noise impact on parameter estimation and high sensitivity to optical center deviation. This method optimizes and calibrates the imaging device that meets the telephoto condition, with the goal of minimizing the reprojection error, and combines the constraint of the optical center deviation, so that the calibration process can effectively avoid the optimization from falling into the local optimum while reducing the influence of noise, and the optimized optical center position is closer to the image center, improving the physical rationality and global fitting effect of the optimized internal and external parameters, thereby significantly improving the calibration accuracy and stability, and making the calibration results show higher robustness and reliability in practical applications.

[0057] In some embodiments, if there are multiple feature points in the calibration pattern, the electronic device can establish pairing relationships corresponding to multiple feature points in S102 and S202. During the calibration process, there are many feature points, and the pairing relationships of some feature points may be abnormal due to noise, occlusion or other factors. If these abnormal feature points are not eliminated, the optimization difficulty of reprojection error and optical center deviation will be increased in the subsequent optimization process. Therefore, after obtaining the initial internal and external parameters of the imaging device, the electronic device can project the three-dimensional coordinates of each feature point using the projection model indicated by the initial internal and external parameters of the imaging device to obtain the current two-dimensional coordinates of each feature point; if the difference between the current two-dimensional coordinates of each feature point and its actual two-dimensional coordinates in the calibration image is greater than the preset difference, the feature point and its corresponding pairing relationship are eliminated, that is, the two-dimensional coordinates and three-dimensional coordinates of the feature point do not participate in the subsequent optimization process. Effectively identifying and filtering out abnormal values ​​or feature points with large noise can not only reduce the interference of abnormal data on the subsequent optimization process, but also enhance the convergence and stability of the projection model of the imaging device, ensure that the subsequent optimization process is closer to the global optimal solution, and thus improve the accuracy and reliability of the calibration results.

[0058] In some embodiments, the optimization process of the initial internal and external parameters of the imaging device is exemplarily described below:

[0059] See also Figure 3The electronic device can use the two-dimensional coordinates of the feature points in the calibration image as the target two-dimensional coordinates in the optimization process, determine the target optical center position in the optimization process based on the image center of the calibration image, and the target two-dimensional coordinates and the target optical center position provide a clear reference for the optimization, and use the projection model indicated by the initial internal and external parameters of the imaging device as the current projection model in the first round of iteration (S301), and perform the following optimization process:

[0060] In S302, the three-dimensional coordinates of the feature points are projected using the current projection model to obtain current two-dimensional coordinates.

[0061] In S303, the reprojection error is determined based on the difference between the current two-dimensional coordinates and the target two-dimensional coordinates of the feature point, and the optical center deviation is obtained according to the deviation between the current optical center position indicated by the current projection model and the target optical center position.

[0062] The reprojection error quantifies the accuracy of the current projection model's fitting of the two-dimensional coordinates, while the optical center deviation evaluates the rationality of the optical center position. The two comprehensively measure the fitness of the current projection model.

[0063] Specifically, for multiple feature points, after respectively determining the difference between the current two-dimensional coordinates and the target two-dimensional coordinates of each feature point, there are multiple differences, and the multiple differences can be statistically processed to obtain a reprojection error. This application does not impose any restrictions on the specific method of statistical processing, and can be specifically set according to the actual application scenario. For example, the sum of multiple differences, the average value, median, maximum value, minimum value, standard deviation, and variance of multiple differences can be determined as the reprojection error.

[0064] In S304, it is determined whether the following conditions are met: the reprojection error and the optical center deviation meet a preset convergence condition or the current number of iterations reaches a preset maximum number of iterations.

[0065] Exemplarily, the preset convergence conditions include: the optical center deviation is less than or equal to the preset deviation threshold, and the reprojection error is less than or equal to the preset error threshold. The specific values ​​of the preset deviation threshold, the preset error threshold, and the preset maximum number of iterations can be set according to the actual application scenario, and this embodiment does not impose any restrictions on this, so as to adapt to the calibration requirements of different scenarios.

[0066] For example, the optical center deviation includes the first deviation in the horizontal direction and the second deviation in the vertical direction. Assuming that the resolution of the calibration image is 1920*1080, the optical center deviation is expected to be within Image.Width×0.05, that is, the preset first deviation threshold corresponding to 1920 is 96 pixels (that is, 1920×0.05), and the preset second deviation threshold corresponding to 1080 is 54 pixels (that is, 1080×0.05).

[0067] In S305, if at least one of the reprojection error and the optical center deviation does not meet the preset convergence condition and the current number of iterations does not reach the preset maximum number of iterations, the internal and external parameters of the imaging device are updated with the optimization goal of minimizing the reprojection error and constraining the optical center deviation, and the projection model indicated by the updated internal and external parameters is used as the current projection model in the next round of iteration.

[0068] In this step, the projection model is gradually optimized by dynamically adjusting the internal and external parameters to ensure the rationality of the optical center position and the minimization of the reprojection error, thereby improving the accuracy and stability of the calibration.

[0069] Exemplarily, the electronic device may construct a reprojection error term in the objective function based on the reprojection error, and construct an optical center deviation regularization term in the objective function based on the optical center deviation and a preset regularization term weight; wherein the optical center deviation regularization term includes the product between the L2 norm of the optical center deviation and the preset regularization term weight, wherein the optical center deviation includes a first deviation in the horizontal direction and a second deviation in the vertical direction, and the L2 norm of the optical center deviation includes the sum of the square of the first deviation and the square of the second deviation. Then, the electronic device updates the internal and external parameters of the imaging device by optimizing the objective function including the reprojection error term and the optical center deviation regularization term.

[0070] Among them, the reprojection error term ensures that the error between the current two-dimensional coordinates of the feature point and the target two-dimensional coordinates is minimized, thereby improving the ability of the internal and external parameters to accurately describe the actual projection relationship; the optical center deviation regularization term constrains the optical center position through the L2 norm of the optical center deviation, making it closer to the center of the image, and enhancing the geometric rationality of the calibration result. The introduction of the regularization term weight provides flexibility, allowing the optimization process to strike a balance between the accuracy of the reprojection error and the constraint of the optical center position. Finally, by optimizing the objective function containing the reprojection error term and the optical center deviation regularization term (for example, minimizing the value of the objective function), the interference of noise and deviation can be effectively reduced, the calibration accuracy of the internal and external parameters can be improved, and the convergence and robustness of the optimization can be ensured at the same time, providing reliable support for the high-precision application of actual imaging equipment.

[0071] For example, the objective function can be expressed as follows: N represents the number of feature points; Represents the current two-dimensional coordinates of the i-th feature point; Represents the target two-dimensional coordinates of the i-th feature point; Represents the difference between the current 2D coordinates and the target 2D coordinates of the i-th feature point, represents the reprojection error; λ represents the regularization weight, which is used to balance the optimization effects of the reprojection error and the optical center deviation; c currentrepresents the current optical center position indicated by the current projection model, c target represents the target optical center position, ‖c current -c target ‖ 2 represents the optical center deviation, which is constrained by the regularization term.

[0072] Exemplarily, the regularization term weight is less than or equal to 1, and the regularization term weight is positively correlated with the reprojection error determined in the first round of iteration. When the reprojection error determined in the first round of iteration of the above optimization process is large, the optimization process may cause the optical center position to deviate from expectations, so a stronger optical center deviation constraint (that is, setting a larger λ) is required to ensure that the optimization process does not sacrifice the rationality of the optical center position. When the reprojection error determined in the first round of iteration of the above optimization process is small, it means that the current projection model is close to the correct solution, and the constraint on the optical center position can be appropriately relaxed (that is, setting a smaller λ) to focus more on reducing the reprojection error. The regularization term weight λ determines the degree of influence of the optical center deviation on the objective function. By dynamically adjusting the regularization term weight, the relationship between minimizing the reprojection error and the optical center constraint can be flexibly balanced under different calibration conditions.

[0073] Exemplarily, in the process of optimizing the objective function, a variety of different optimization algorithms can be combined and applied to improve the optimization efficiency and result quality. For example, multiple optimization algorithms (such as gradient direction, Newton direction, etc.) can be used to simultaneously calculate multiple optimization directions. During the optimization process, each optimization algorithm will calculate an optimization direction based on the current parameters and the characteristics of the objective function. The electronic device can select an optimization direction from them, such as selecting the direction that can minimize the objective function value, or selecting the most stable direction (avoiding numerical divergence). By comparing the performance of different optimization algorithms at the same time in each iteration, it is possible to more quickly determine which optimization algorithm is more suitable for the current round of iterations, and different optimization algorithms have different search capabilities and convergence paths, which can reduce the possibility of falling into local extreme values.

[0074] In S306, if the reprojection error and the optical center deviation both meet the preset convergence conditions or the current number of iterations reaches the preset maximum number of iterations, the optimization process ends.

[0075] Through the above steps, the initial internal and external parameters of the imaging device can be iteratively optimized to achieve minimization of the reprojection error and effective constraint of the optical center deviation, thereby improving the global accuracy and geometric rationality of the calibration.

[0076] In some embodiments, the regularization term weight used in the first optimization process may be pre-set, for example, it may be determined based on the size of the reprojection error determined in the first iteration of the first optimization process, or it may be an empirical value set by a developer, or it may be determined by other algorithms (such as cross-validation, grid search, L-curve analysis, or Bayesian optimization algorithms). This embodiment does not impose any restrictions on this.

[0077] Exemplarily, after the above optimization process ends, if at least one of the reprojection error and the optical center deviation obtained in the last round of iteration in the optimization process does not meet the preset convergence condition, the electronic device can adjust the regularization term weight used in the optimization process, and re-perform the above optimization process with the adjusted regularization term weight. This embodiment dynamically adjusts the regularization term weight to flexibly deal with conflicts between different objectives in the optimization process, thereby achieving balanced optimization of the reprojection error and the optical center deviation.

[0078] In one possible case, if the optical center deviation obtained in the last iteration of the optimization process is greater than the preset deviation threshold, the weight of the regularization term used in the optimization process is increased; in this case, the optical center position is not constrained enough from the image center, indicating that the optimization process focuses more on minimizing the reprojection error and ignores the geometric rationality of the optical center position. By increasing the weight of the regularization term and improving the influence of the optical center deviation regularization term in the objective function, the constraint on the optical center position can be strengthened, prompting the next round of optimization to better correct the optical center deviation.

[0079] In another possible situation, if the optical center deviation obtained in the last iteration of the optimization process is less than or equal to the preset deviation threshold, but the reprojection error obtained in the last iteration is greater than the preset error threshold, the weight of the regularization term used in the optimization process is reduced. In this case, the optical center deviation has met the requirements, but the reprojection error is still large, indicating that the optimization process may pay too much attention to the constraints of the optical center position, resulting in insufficient attention to the actual fitting accuracy. At this time, by reducing the weight of the regularization term and reducing the influence of the optical center deviation regularization term, more optimization weights can be allocated to the reprojection error term, prompting the next optimization process to better reduce the reprojection error.

[0080] Through the above dynamic adjustment, the optimization process can be adaptively adjusted between convergence and accuracy, thereby improving the global optimality and robustness of the final result. This adjustment mechanism can also prevent the optimization from falling into the local optimum, further improving the accuracy and geometric rationality of the calibration results.

[0081] In other embodiments, effective efficiency and optimization quality can be improved by step-by-step optimization. The initial internal and external parameter optimization process for the imaging device can include a first step-by-step optimization process, a second step-by-step optimization process, and a joint optimization process.

[0082] The first step optimization process is to first fix the other parameters of the imaging device except the optical center parameter, and then optimize only the optical center parameter, which helps to correct the optical center deviation more accurately. This method can ensure the optimization independence of the optical center parameter and reduce the influence of other parameter errors on the optical center optimization.

[0083] The second step optimization process is to first fix the optical center parameters determined in the first step optimization process, and then optimize the internal and external parameters of the imaging device except the optical center parameters, which can avoid the excessive influence of the optical center deviation on the reprojection error, thereby more accurately adjusting other internal and external parameters such as focal length, principal point offset, and distortion coefficient.

[0084] The joint optimization process is to jointly optimize all the internal and external parameters of the imaging device determined in the first and second sub-step optimization processes. Based on the parameters determined by the first and second sub-step optimizations, joint optimization can further coordinate the relationship between the internal and external parameters, thereby improving the global consistency of the overall optimization results. This process integrates the local optimal results of the first two steps and ultimately achieves the global optimal solution.

[0085] Specifically, first, the electronic device uses the two-dimensional coordinates of the feature points in the calibration image as the target two-dimensional coordinates in all optimization processes, and determines the target optical center position in all optimization processes based on the image center of the calibration image.

[0086] Next, the electronic device uses the projection model indicated by the initial internal and external parameters of the imaging device as the current projection model in the following first round of iterations, and performs the following first step optimization process: the three-dimensional coordinates of the feature points are projected using the current projection model to obtain the current two-dimensional coordinates; based on the difference between the current two-dimensional coordinates of the feature points and the target two-dimensional coordinates, the reprojection error is determined, and the optical center deviation is obtained according to the deviation between the current optical center position indicated by the current projection model and the target optical center position. Determine whether the following conditions are met: the reprojection error and the optical center deviation meet the preset convergence conditions or the current number of iterations reaches the preset maximum number of iterations. If not, fix the other parameters except the optical center parameters in the internal and external parameters of the imaging device, and take minimizing the reprojection error and constraining the optical center deviation as the optimization goal, update the optical center parameters, and use the projection model indicated by the optical center parameters and the above-mentioned other fixed parameters as the current projection model in the next round of iterations. If so, end the first step optimization process.

[0087] Next, the electronic device uses the projection model indicated by the optical center parameters determined in the first sub-step optimization process and the other fixed parameters as the current projection model in the following first round of iterations, and performs the following second sub-step optimization process: the three-dimensional coordinates of the feature points are projected using the current projection model to obtain the current two-dimensional coordinates; based on the difference between the current two-dimensional coordinates of the feature points and the target two-dimensional coordinates, the reprojection error is determined, and the optical center deviation is obtained according to the deviation between the current optical center position indicated by the current projection model and the target optical center position. Determine whether the following conditions are met: the reprojection error and the optical center deviation meet the preset convergence conditions or the current number of iterations reaches the preset maximum number of iterations. If not, fix the optical center parameters determined in the first sub-step optimization process, with minimizing the reprojection error and constraining the optical center deviation as the optimization goal, update the other parameters except the optical center parameters in the internal and external parameters of the imaging device, and use the fixed optical center parameters and the projection model indicated by the updated other parameters as the current projection model in the next round of iterations. If so, end the second sub-step optimization process.

[0088] Finally, the electronic device uses the projection model indicated by the optical center parameters determined in the first sub-step optimization process and the other parameters determined in the second sub-step optimization process as the current projection model in the following first round of iterations, and performs the following joint optimization process: the three-dimensional coordinates of the feature points are projected using the current projection model to obtain the current two-dimensional coordinates; based on the difference between the current two-dimensional coordinates of the feature points and the target two-dimensional coordinates, the reprojection error is determined, and the optical center deviation is obtained according to the deviation between the current optical center position indicated by the current projection model and the target optical center position. Determine whether the following conditions are met: the reprojection error and the optical center deviation meet the preset convergence conditions or the current number of iterations reaches the preset maximum number of iterations. If not, all internal and external parameters determined by the imaging device in the first sub-step optimization process and the second sub-step optimization process are updated with the optimization goal of minimizing the reprojection error and constraining the optical center deviation, and the projection model indicated by the updated internal and external parameters is used as the current projection model in the next round of iterations. If so, end the joint optimization process.

[0089] In this embodiment, by optimizing step by step, the optical center parameter and other parameters are optimized separately, which can effectively reduce the interference between parameters and make the optimization process more stable. Compared with the complex process of directly optimizing all parameters at the same time, the step-by-step optimization can gradually narrow the optimization space, reduce the amount of calculation, and improve the optimization efficiency. Finally, combined with joint optimization, the calculation complexity can be significantly reduced while ensuring the optimization accuracy.

[0090] In an exemplary embodiment, in a virtual shooting scene, a display (such as an LED screen) is usually used to display a virtual scene, which is combined with a real scene in front of the display to form a complete shooting picture. When the imaging device shoots the display and the real scene in front of the display, it is necessary to ensure that the geometric relationship and optical characteristics of the virtual scene and the real scene in the picture are highly consistent, so as to achieve seamless integration of visual effects. Before actual shooting, the calibration pattern can be displayed on the display, and the internal and external parameters of the imaging device can be accurately calibrated by the above calibration method.

[0091] Specifically, a specific calibration pattern, such as a checkerboard or dot pattern, is first displayed on the display. Since the display is easy to control, the three-dimensional coordinates of its feature points can be accurately determined by a three-dimensional scanning modeling method. The imaging device then shoots the calibration pattern on the display and collects multiple calibration images to capture the optical imaging characteristics of the imaging device at different positions and postures. Then, using the above calibration method, a pairing relationship is established between the three-dimensional coordinates of the feature points and their two-dimensional coordinates in the calibration image, and the pairing relationship is used to complete the calibration and optimization process of the internal and external parameters of the imaging device. Especially for telephoto imaging devices, multiple rounds of optimization can effectively reduce the reprojection error and constrain the optical center deviation, thereby ensuring the high accuracy of the calibration results.

[0092] Through this method, in a virtual shooting scene, the imaging model of the imaging device can be accurately matched with the geometric relationship between the virtual content on the display and the real scene. This calibration process can effectively improve the overall effect of the fusion of virtual and real in virtual shooting, reduce the complexity of post-processing, and improve the realism and consistency of visual presentation.

[0093] The various technical features in the above embodiments can be arbitrarily combined as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, any combination of the various technical features in the above embodiments also falls within the scope of this specification.

[0094] In some embodiments, the embodiments of this specification also provide an electronic device, including: a processor; a memory for storing processor executable instructions; wherein the processor implements any of the above methods by running the executable instructions.

[0095] Figure 4 is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 4At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408, and a non-volatile memory 410, and may also include hardware required for other functions. One or more embodiments of this specification may be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into the memory 408 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0096] Please refer to Figure 5 , the calibration device of the imaging device can be applied to Figure 4 The device shown in the figure is used to implement the technical solution of this specification. Among them, the calibration device of the imaging device may include:

[0097] The calibration image acquisition module 501 is used to acquire a plurality of calibration images, wherein the calibration images are obtained by photographing a calibration pattern by an imaging device to be calibrated.

[0098] The pairing relationship establishing module 502 is used to establish a pairing relationship between the three-dimensional coordinates and the two-dimensional coordinates of the feature point in the calibration pattern, where the two-dimensional coordinates are the two-dimensional coordinates of the feature point in the calibration image.

[0099] The calibration module 503 is used to calibrate the imaging device by using the pairing relationship to obtain initial internal and external parameters of the imaging device.

[0100] The optimization module 504 is used to optimize the initial internal and external parameters of the imaging device by using the pairing relationship, with the goal of minimizing the reprojection error and constraining the optical center deviation.

[0101] In one implementation, the initial internal and external parameters of the imaging device include a focal length. The optimization module 504 is specifically configured to optimize the initial internal and external parameters of the imaging device using the pairing relationship, if the focal length of the imaging device meets the telephoto condition, with minimizing the reprojection error and constraining the optical center deviation as the optimization goal.

[0102] In one implementation, the optimization module 504 is specifically used to use the two-dimensional coordinates of the feature point in the calibration image as the target two-dimensional coordinates in the optimization process, determine the target optical center position in the optimization process based on the image center of the calibration image, and use the projection model indicated by the initial internal and external parameters of the imaging device as the current projection model in the first round of iteration, and perform the following optimization process: use the current projection model to project the three-dimensional coordinates of the feature point to obtain the current two-dimensional coordinates; determine the reprojection error based on the difference between the current two-dimensional coordinates of the feature point and the target two-dimensional coordinates, and obtain the optical center deviation based on the deviation between the current optical center position indicated by the current projection model and the target optical center position; if at least one of the reprojection error and the optical center deviation does not meet the preset convergence condition and the current number of iterations does not reach the preset maximum number of iterations, minimize the reprojection error and constrain the optical center deviation as the optimization target, update the internal and external parameters of the imaging device, and use the projection model indicated by the updated internal and external parameters as the current projection model in the next round of iteration.

[0103] In one implementation, the optimization module 504 is specifically used to construct a reprojection error term in the objective function based on the reprojection error, and to construct an optical center deviation regularization term in the objective function based on the optical center deviation and a preset regularization term weight; and to update the internal and external parameters of the imaging device by optimizing the objective function including the reprojection error term and the optical center deviation regularization term.

[0104] In one implementation, the optical center deviation regularization term includes the product of the L2 norm of the optical center deviation and the preset regularization term weight, wherein the optical center deviation includes a first deviation in the horizontal direction and a second deviation in the vertical direction, and the L2 norm of the optical center deviation includes the sum of the square of the first deviation and the square of the second deviation.

[0105] In one implementation, the regularization term weight is less than or equal to 1, and the regularization term weight is positively correlated with the reprojection error determined in the first round of iterations.

[0106] In one implementation, the method also includes a regularization term weight adjustment module, which is used to adjust the regularization term weight used in the optimization process after the optimization process is completed, if at least one of the reprojection error and the optical center deviation obtained in the last round of iteration of the optimization process does not meet the preset convergence condition, and re-perform the optimization process with the adjusted regularization term weight.

[0107] In one implementation, the regularization term weight adjustment module is specifically used to increase the regularization term weight used in the optimization process if the optical center deviation obtained in the last round of iteration is greater than a preset deviation threshold; if the optical center deviation obtained in the last round of iteration is less than or equal to the preset deviation threshold, but the reprojection error obtained in the last round of iteration is greater than a preset error threshold, reduce the regularization term weight used in the optimization process.

[0108] In one implementation, the optimization process includes a first step optimization process, a second step optimization process, and a joint optimization process.

[0109] In the first step optimization process, the optimization module 504 is specifically used to fix other parameters of the internal and external parameters of the imaging device except the optical center parameter, so as to minimize the reprojection error and constrain the optical center deviation as the optimization goal, and update the optical center parameter.

[0110] In the second step-by-step optimization process, the optimization module 504 is specifically used to fix the optical center parameters determined in the first step-by-step optimization process, to minimize the reprojection error and constrain the optical center deviation as the optimization goal, and to update other parameters of the internal and external parameters of the imaging device except the optical center parameters.

[0111] In the joint optimization process, the optimization module 504 is specifically used to update all internal and external parameters of the imaging device determined in the first step optimization process and the second step optimization process with minimizing the reprojection error and constraining the optical center deviation as the optimization goal.

[0112] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0113] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0114] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0115] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.

[0116] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A calibration method for an imaging device, comprising: Acquire multiple calibration images, where the calibration images are obtained by photographing the calibration pattern by the imaging device to be calibrated; Establishing a pairing relationship between the three-dimensional coordinates and the two-dimensional coordinates of the feature point in the calibration pattern, wherein the two-dimensional coordinates are the two-dimensional coordinates of the feature point in the calibration image; Using the pairing relationship, calibrating the imaging device to obtain initial internal and external parameters of the imaging device; Taking minimizing the reprojection error and constraining the optical center deviation as the optimization goal, the initial internal and external parameters of the imaging device are optimized by utilizing the pairing relationship.

2. The method according to claim 1, wherein the initial internal and external parameters of the imaging device include focal length; The optimization objective is to minimize the reprojection error and constrain the optical center deviation, and to optimize the initial internal and external parameters of the imaging device by using the pairing relationship, including: If the focal length of the imaging device meets the telephoto condition, the initial internal and external parameters of the imaging device are optimized by using the pairing relationship with the optimization goal of minimizing the reprojection error and constraining the optical center deviation.

3. The method according to claim 1, wherein the optimization objective is to minimize the reprojection error and constrain the optical center deviation, and the initial internal and external parameters of the imaging device are optimized by using the pairing relationship, comprising: The two-dimensional coordinates of the feature points in the calibration image are used as the target two-dimensional coordinates in the optimization process, the target optical center position in the optimization process is determined based on the image center of the calibration image, and the projection model indicated by the initial internal and external parameters of the imaging device is used as the current projection model in the first round of iteration, and the following optimization process is performed: Projecting the three-dimensional coordinates of the feature points using the current projection model to obtain current two-dimensional coordinates; Determine a reprojection error based on a difference between the current two-dimensional coordinates of the feature point and the target two-dimensional coordinates, and obtain an optical center deviation according to a deviation between a current optical center position indicated by the current projection model and the target optical center position; If at least one of the reprojection error and the optical center deviation does not satisfy the preset convergence condition and the current number of iterations does not reach the preset maximum number of iterations, the internal and external parameters of the imaging device are updated with the optimization goal of minimizing the reprojection error and constraining the optical center deviation, and the projection model indicated by the updated internal and external parameters is used as the current projection model in the next round of iteration.

4. The method according to claim 3, wherein the step of updating the internal and external parameters of the imaging device with the optimization goal of minimizing the reprojection error and constraining the optical center deviation comprises: Constructing a reprojection error term in an objective function based on the reprojection error, and constructing an optical center deviation regularization term in the objective function based on the optical center deviation and a preset regularization term weight; By optimizing the objective function including the reprojection error term and the optical center deviation regularization term, the internal and external parameters of the imaging device are updated.

5. According to the method of claim 4, the optical center deviation regularization term comprises the product of the L2 norm of the optical center deviation and the preset regularization term weight, wherein, The optical center deviation includes a first deviation in a horizontal direction and a second deviation in a vertical direction, and an L2 norm of the optical center deviation includes a sum of a square of the first deviation and a square of the second deviation.

6. According to the method of claim 4, the regularization term weight is less than or equal to 1, and the regularization term weight is positively correlated with the reprojection error determined in the first round of iterations.

7. The method according to claim 4, further comprising: After the optimization process is completed, if at least one of the reprojection error and the optical center deviation obtained in the last round of iteration of the optimization process does not meet the preset convergence condition, the regularization term weight used in the optimization process is adjusted, and the optimization process is repeated with the adjusted regularization term weight.

8. The method according to claim 7, wherein if at least one of the reprojection error and the optical center deviation obtained in the last round of iteration of the optimization process does not satisfy the preset convergence condition, adjusting the regularization term weight used in the optimization process comprises: If the optical center deviation obtained in the last round of iteration is greater than a preset deviation threshold, increasing the weight of the regularization term used in the optimization process; If the optical center deviation obtained in the last round of iteration is less than or equal to the preset deviation threshold, but the reprojection error obtained in the last round of iteration is greater than the preset error threshold, the regularization term weight used in the optimization process is reduced.

9. The method according to claim 3, wherein the optimization process comprises a first step optimization process, a second step optimization process and a joint optimization process; In the first step optimization process, the updating of the internal and external parameters of the imaging device with minimizing the reprojection error and constraining the optical center deviation as the optimization goal includes: Fixing other parameters except the optical center parameter among the internal and external parameters of the imaging device, minimizing the reprojection error and constraining the optical center deviation as the optimization goal, and updating the optical center parameter; In the second step optimization process, the internal and external parameters of the imaging device are updated with the optimization goal of minimizing the reprojection error and constraining the optical center deviation, including: Fixing the optical center parameter determined in the first step optimization process, taking minimizing the reprojection error and constraining the optical center deviation as the optimization goal, and updating other parameters of the internal and external parameters of the imaging device except the optical center parameter; In the joint optimization process, the updating of the internal and external parameters of the imaging device with minimizing the reprojection error and constraining the optical center deviation as the optimization goal includes: Taking minimizing the reprojection error and constraining the optical center deviation as the optimization goal, all internal and external parameters of the imaging device determined in the first step optimization process and the second step optimization process are updated.

10. An electronic device, comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method according to any one of claims 1 to 9 by executing the executable instructions.

11. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 9.

12. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

Images