Imaging device calibration method, equipment, storage medium and product

Abstract

One or more embodiments of this specification provide a calibration method, device, storage medium, and product for an imaging device. The method includes: acquiring multiple calibration images, where the calibration images are obtained by photographing a calibration pattern with the imaging device to be calibrated; establishing a pairing relationship between the three-dimensional coordinates of a feature point in the calibration pattern and the two-dimensional coordinates of the feature point in the calibration image; utilizing the pairing relationship to calibrate the imaging device to obtain initial intrinsic and extrinsic parameters of the imaging device; and utilizing the pairing relationship to optimize the initial intrinsic and extrinsic parameters of the imaging device with the optimization goal of minimizing reprojection error and constraining optical center deviation, thereby improving the physical rationality and global fitting effect of the optimized intrinsic and extrinsic parameters.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of calibration technology, and in particular, to a calibration method for an imaging device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the rapid development of imaging technology, imaging devices have been widely used in fields such as industry, medicine, autonomous driving, and consumer electronics. To ensure high-precision imaging during use, calibration of imaging parameters (i.e., intrinsic and extrinsic parameters) is a critical step. During the calibration process, noise and systematic biases inevitably exist in the calibration data. These can lead to errors in parameter estimation and even cause the optimization process to fall into a local optimum, thereby reducing calibration accuracy. Summary of the Invention

[0003] In view of this, one or more embodiments of this specification provide a calibration method for an imaging device, an electronic device, a computer-readable storage medium, and a computer program product.

[0004] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0005] According to a first aspect of one or more embodiments of this specification, a calibration method for an imaging device is provided, comprising:

[0006] Acquire multiple calibration images, where the calibration images are obtained by photographing the calibration pattern by the imaging device to be calibrated;

[0007] Establishing a pairing relationship between the three-dimensional coordinates and the two-dimensional coordinates of the feature point in the calibration pattern, where the two-dimensional coordinates are the two-dimensional coordinates of the feature point in the calibration image;

[0008] Using the pairing relationship, calibrating the imaging device to obtain initial internal and external parameters of the imaging device;

[0009] With minimizing the reprojection error and constraining the optical center deviation as the optimization goal, the initial internal and external parameters of the imaging device are optimized using the pairing relationship.

[0010] According to a second aspect of the embodiments of this specification, there is provided an electronic device, including:

[0011] processor;

[0012] a memory for storing processor-executable instructions;

[0013] Wherein, when the processor executes the executable instructions, it is used to implement the method described in the first aspect.

[0014] According to a third aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0015] According to a fourth aspect of the embodiments of this specification, a computer program product is provided, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.

[0016] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0017] In the embodiment of this specification, a plurality of calibration images are first obtained. The calibration images are obtained by photographing the calibration pattern by the imaging device to be calibrated, laying the foundation for the subsequent calculation of internal and external parameters; then, a pairing relationship is established between the three-dimensional coordinates of the feature points in the calibration pattern and the two-dimensional coordinates of the feature points in the calibration image, so as to associate the real three-dimensional space points with the projection points in the image; then, the imaging device is calibrated using the pairing relationship to obtain the initial internal and external parameters of the imaging device; finally, the initial internal and external parameters of the imaging device are optimized using the pairing relationship with the optimization goal of minimizing the reprojection error and constraining the optical center deviation. With the goal of minimizing the reprojection error and combining the constraint of the optical center deviation, the calibration process can effectively avoid the optimization from falling into the local optimum while reducing the influence of noise, and the optimized optical center position can be closer to the center of the image, improving the physical rationality and global fitting effect of the optimized internal and external parameters, thereby significantly improving the calibration accuracy and stability.

[0018] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flowchart of a calibration method for an imaging device provided by an exemplary embodiment.

[0020] Figure 2 This is a flowchart of another calibration method for an imaging device provided by an exemplary embodiment.

[0021] Figure 3 A flowchart of optimizing initial internal and external parameters of an imaging device is provided by an exemplary embodiment.

[0022] Figure 4 It is a structural diagram of an electronic device provided by an exemplary embodiment.

[0023] Figure 5 A block diagram of a calibration device for an imaging device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0024] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments are not intended to represent all implementations consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of one or more embodiments of this specification, as detailed in the appended claims.

[0025] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than those described in this specification. In addition, a single step described in this specification may be broken down into multiple steps for description in other embodiments, and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0026] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0027] The imaging device mentioned in the embodiments of this specification may be a physical imaging device. The imaging device may be configured to detect electromagnetic radiation (e.g., visible light, infrared light, and / or ultraviolet light) and generate image data based on the detected electromagnetic radiation. The imaging device may include a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor that generates an electrical signal in response to the wavelength of light. The generated electrical signal may be processed to generate image data. The image data generated by the imaging device may include one or more images, and the one or more images may be static images (e.g., photos), dynamic images (e.g., videos), or a suitable combination thereof. The image data may be multi-color (e.g., RGB, CMYK, HSV) or monochrome (e.g., grayscale, black, sepia). The imaging device may include a lens configured to direct light onto the image sensor.

[0028] The imaging device may be a camera. The camera may be a dynamic camera or a video camera that captures dynamic image data (e.g., video). The camera may be a static camera that captures static images (e.g., photographs). The camera may capture both dynamic image data and static images. The camera may switch between capturing dynamic image data and static images. Although certain embodiments provided herein are described in the context of a camera, it should be understood that the present disclosure may be applicable to any suitable imaging device, and any description herein regarding a camera may also be applicable to any suitable imaging device, and any description herein regarding a camera may also be applicable to other types of imaging devices. A camera may be used to generate multiple 2D images of a 3D scene (e.g., an environment, one or more objects, etc.). These images generated by the camera may represent projections of the 3D scene onto a 2D image plane. Therefore, each point in the 2D image corresponds to a 3D spatial coordinate in the scene. The camera may include optical elements (e.g., a lens, a reflector, a filter, etc.). The camera may capture color images, grayscale images, infrared images, etc.

[0029] Imaging device calibration is a crucial process in computer vision and photogrammetry. Its purpose is to determine the internal and external parameters of the imaging device so that information about the three-dimensional world can be recovered from two-dimensional images. This process provides insights into the internal geometry of the imaging device, the distortion characteristics of the lens, and the spatial relationship between the imaging device and the world coordinate system. Calibration can improve the accuracy of imaging devices in various applications, particularly in 3D reconstruction, object detection, robotic vision, and augmented reality.

[0030] The main goal of imaging device calibration is to obtain the following parameters:

[0031] (1) Internal parameters, which describe the internal properties of the imaging device. The function of internal parameters is to map 3D world coordinates to 2D image coordinates. The internal parameters mainly include:

[0032] Focal length: The distance from the optical center of the imaging device to the image plane, usually expressed by the horizontal and vertical components of the focal length.

[0033] Principal point (or optical center): The origin of the image coordinate system, usually the optical center of the imaging device.

[0034] Distortion coefficient: describes the distortion of the imaging device lens, usually including radial distortion and tangential distortion.

[0035] (2) Extrinsic parameters describe the position and orientation of the imaging device in the three-dimensional world, usually represented by a rotation matrix and a displacement vector. Extrinsic parameters are used to transform a three-dimensional point in the world coordinate system into a three-dimensional point in the imaging device coordinate system, which is then projected into the image coordinate system using intrinsic parameters.

[0036] In the calibration process of related technologies, noise and systematic deviations are inevitably present in the calibration data, which may lead to errors in the parameter estimation results and even cause the optimization process to fall into a local optimal solution, thereby reducing the calibration accuracy. Based on this, the embodiments of this specification provide a calibration method for an imaging device, which can be performed by any electronic device with computing capabilities, including but not limited to servers, virtual servers, tablet computers, personal digital assistants (PDAs), laptop computers, desktop computers, etc. Please refer to Figure 1 , the method comprising:

[0037] In S101 , a plurality of calibration images are acquired, where the calibration images are obtained by photographing a calibration pattern by an imaging device to be calibrated.

[0038] The acquisition of calibration images is the starting point of the entire imaging device calibration process. In this step, a calibration pattern with a specific geometric structure (such as a checkerboard, dot array, or other regular pattern) can be displayed on the monitor, and the calibration pattern can be photographed with the imaging device to be calibrated to obtain multiple calibration images containing the calibration pattern. Using a monitor to display the calibration pattern is flexible and repeatable.

[0039] Exemplarily, multiple calibration images are obtained by photographing the calibration pattern displayed on the display by an imaging device in at least one of different positions, different postures and different lighting conditions, so that these calibration images cover the calibration pattern in multiple positions, multiple perspectives and multiple lighting conditions within the camera's field of view, ensuring the diversity and coverage of the calibration data, and laying the foundation for subsequent internal and external parameter calculations.

[0040] For example, in order to improve the accuracy of the calibration results, the clarity of the collected calibration image must meet the preset clarity requirement to avoid failure in feature point extraction due to motion blur or loss of focus.

[0041] In S102 , a pairing relationship is established between the three-dimensional coordinates and the two-dimensional coordinates of the feature point in the calibration pattern, where the two-dimensional coordinates are the two-dimensional coordinates of the feature point in the calibration image.

[0042] In this step, based on the known characteristics of the calibration pattern in the display, a three-dimensional coordinate system of the feature points in the calibration pattern is defined (usually the display plane is used as the reference plane of the three-dimensional coordinate system). Then, the two-dimensional coordinates of the feature points in the calibration pattern are extracted from the captured calibration image, and a pairing relationship between these two-dimensional coordinates and the three-dimensional coordinates is established. Further, by obtaining the mapping process between the three-dimensional coordinates of the feature points and the two-dimensional coordinates paired with them (that is, the following calibration process), a projection model of the imaging device (such as a pinhole model) can be obtained. The projection model is a geometric model that describes how the camera maps points in the three-dimensional world to the two-dimensional image plane. The projection model needs to take into account internal parameters (such as focal length, optical center position, distortion parameters) and external parameters (the relative position and posture of the imaging device and the display coordinate system), that is, the projection model is constructed by the internal and external parameters of the imaging device. At this time, the internal and external parameters are unknown quantities to be calibrated.

[0043] This step establishes the mathematical foundation for the calibration process, associating the 3D coordinates of a real 3D space point with the 2D coordinates of its projection point in the image.

[0044] In S103 , the imaging device is calibrated using the pairing relationship to obtain initial internal and external parameters of the imaging device.

[0045] The initial internal and external parameters are solved by using the pairing relationship established in the previous step—that is, the pairing relationship between the two-dimensional coordinates of each feature point extracted from the calibration image and the known three-dimensional coordinates. Internal parameters include parameters describing the camera's imaging characteristics, such as focal length, optical center position, and distortion coefficients. External parameters include the position of the imaging device in three-dimensional space and its translational and rotational posture. This process can typically use camera calibration algorithms from related technologies (such as the DLT method or Zhang Zhengyou calibration method) to preliminarily estimate the internal and external parameters through linear optimization or simple nonlinear optimization, providing an initial solution for subsequent refined optimization.

[0046] In S104 , the initial internal and external parameters of the imaging device are optimized by utilizing the pairing relationship with the optimization goal of minimizing the reprojection error and constraining the optical center deviation.

[0047] Based on the initial internal and external parameters, this step takes minimizing the reprojection error as the main optimization goal, and combines the constraints of the optical center position deviation to further optimize the internal and external parameters of the camera. The reprojection error refers to the error between the current two-dimensional coordinates obtained after the three-dimensional coordinates of the feature points of the calibration pattern are projected onto the image plane by the projection model of the imaging device, and their actual two-dimensional coordinates. During the optimization process, the error is minimized by adjusting the internal and external parameters. And further considering the influence of the optical center position on the imaging quality, by constraining the optical center position, it is ensured that the optimized optical center is close to the center position of the image, thereby improving the physical rationality and global fitting effect of the calibration results. By comprehensively considering the reprojection error and the optical center deviation, the accuracy and stability of the calibration results are improved, and the imaging error problem that may be caused by excessive optical center position offset can be avoided. It is suitable for scenes that are sensitive to the optical center position.

[0048] In some embodiments, the optical center position in an optical system has a significant impact on image quality. Optical center deviation may introduce imaging errors. This is especially true in telephoto cameras, where optical center deviation is more prominent due to their sensitivity to imaging parameters. Therefore, when calibrating a telephoto camera, the calibration method for the imaging device described in the embodiments of this specification may be used, such as Figure 2 As shown, a calibration method for an imaging device of a telephoto camera is provided, the method comprising:

[0049] In S201, a plurality of calibration images are obtained, where the calibration images are obtained by photographing the calibration pattern displayed on the display by the imaging device to be calibrated. Please refer to the description of S101 above, which will not be repeated here.

[0050] In S202, a pairing relationship is established between the three-dimensional coordinates of the feature point in the calibration pattern and the two-dimensional coordinates, where the two-dimensional coordinates are the two-dimensional coordinates of the feature point in the calibration image.

[0051] In S203, the imaging device is calibrated using the pairing relationship to obtain the initial intrinsic and extrinsic parameters of the imaging device. See the description of S101 above and will not be repeated here. The initial intrinsic and extrinsic parameters of the imaging device include the focal length, which serves as the basis for the next step.

[0052] In S204 , it is determined whether the focal length of the imaging device meets the telephoto condition.

[0053] Illustratively, the telephoto condition may be that the focal length of the imaging device is greater than or equal to a preset value, where the preset value includes, for example, 50 mm, 100 mm, or 200 mm, but is not limited thereto.

[0054] In S205 , if the focal length of the imaging device satisfies the telephoto condition, the initial internal and external parameters of the imaging device are optimized using the pairing relationship with the optimization goal of minimizing the reprojection error and constraining the optical center deviation.

[0055] In S206 , if the focal length of the imaging device satisfies the non-telephoto condition, the calibration process ends.

[0056] In this embodiment, the calibration process for telephoto cameras faces challenges related to significant noise impact on parameter estimation and high sensitivity to optical center deviation. This method optimizes and calibrates an imaging device that meets telephoto requirements, minimizing reprojection error. By incorporating constraints based on optical center deviation, the calibration process effectively avoids local optimization while reducing the impact of noise. Furthermore, the optimized optical center position is closer to the image center, improving the physical rationality of the optimized internal and external parameters and the global fit. This significantly enhances calibration accuracy and stability, resulting in more robust and reliable calibration results in practical applications.

[0057] In some embodiments, if there are multiple feature points in the calibration pattern, the electronic device can establish pairing relationships corresponding to each of the multiple feature points in S102 and S202. During the calibration process, the number of feature points is large, and some feature point pairing relationships may be abnormal due to noise, occlusion, or other factors. If these abnormal feature points are not eliminated, the optimization difficulty of reprojection error and optical center deviation will increase during the subsequent optimization process. Therefore, after obtaining the initial intrinsic and extrinsic parameters of the imaging device, the electronic device can project the three-dimensional coordinates of each feature point using the projection model indicated by the initial intrinsic and extrinsic parameters of the imaging device to obtain the current two-dimensional coordinates of each feature point. If the difference between the current two-dimensional coordinates of each feature point and its actual two-dimensional coordinates in the calibration image is greater than a preset difference, the feature point and its corresponding pairing relationship are eliminated. In other words, the two-dimensional and three-dimensional coordinates of the feature point are not included in the subsequent optimization process. This effectively identifies and filters out outliers or feature points with high noise. This not only reduces the interference of abnormal data on the subsequent optimization process, but also enhances the convergence and stability of the imaging device's projection model, ensuring that the subsequent optimization process is closer to the global optimal solution, thereby improving the accuracy and reliability of the calibration results.

[0058] In some embodiments, the optimization process of the initial internal and external parameters of the imaging device is exemplarily described below:

[0059] See also Figure 3The electronic device may use the two-dimensional coordinates of the feature points in the calibration image as the target two-dimensional coordinates in the optimization process, and determine the target optical center position in the optimization process based on the image center of the calibration image. The target two-dimensional coordinates and the target optical center position provide a clear reference for the optimization, and use the projection model indicated by the initial internal and external parameters of the imaging device as the current projection model in the first round of iteration (S301) to perform the following optimization process:

[0060] In S302 , the three-dimensional coordinates of the feature points are projected using the current projection model to obtain current two-dimensional coordinates.

[0061] In S303 , a reprojection error is determined based on the difference between the current two-dimensional coordinates and the target two-dimensional coordinates of the feature point, and an optical center deviation is obtained according to the deviation between the current optical center position indicated by the current projection model and the target optical center position.

[0062] The reprojection error quantifies the accuracy of the current projection model's fit to the 2D coordinates, while the optical center deviation evaluates the rationality of the optical center's position. Together, they provide a comprehensive measure of the current projection model's fit.

[0063] Specifically, for multiple feature points, after determining the difference between the current two-dimensional coordinates and the target two-dimensional coordinates of each feature point, there are multiple differences. The multiple differences can be statistically processed to obtain a reprojection error. This application does not impose any restrictions on the specific method of statistical processing, and specific settings can be made based on actual application scenarios. For example, the sum of the multiple differences, the average, median, maximum, minimum, standard deviation, and variance of the multiple differences can be determined as the reprojection error.

[0064] In S304 , it is determined whether the following conditions are met: the reprojection error and the optical center deviation meet a preset convergence condition or the current number of iterations reaches a preset maximum number of iterations.

[0065] Exemplarily, the preset convergence conditions include: the optical center deviation is less than or equal to a preset deviation threshold, and the reprojection error is less than or equal to a preset error threshold. The specific values ​​of the preset deviation threshold, the preset error threshold, and the preset maximum number of iterations can be set based on the actual application scenario and are not limited in this embodiment to accommodate calibration requirements in different scenarios.

[0066] For example, the optical center deviation includes the first deviation in the horizontal direction and the second deviation in the vertical direction. Assuming that the resolution of the calibration image is 1920*1080, the expected optical center deviation is within Image.Width×0.05, that is, the preset first deviation threshold corresponding to 1920 is 96 pixels (that is, 1920×0.05), and the preset second deviation threshold corresponding to 1080 is 54 pixels (that is, 1080×0.05).

[0067] In S305, if at least one of the reprojection error and the optical center deviation does not meet the preset convergence condition and the current number of iterations does not reach the preset maximum number of iterations, the internal and external parameters of the imaging device are updated with the optimization goal of minimizing the reprojection error and constraining the optical center deviation, and the projection model indicated by the updated internal and external parameters is used as the current projection model in the next round of iteration.

[0068] In this step, the projection model is gradually optimized by dynamically adjusting the internal and external parameters to ensure the rationality of the optical center position and the minimization of the reprojection error, thereby improving the accuracy and stability of the calibration.

[0069] Exemplarily, the electronic device may construct a reprojection error term in the objective function based on the reprojection error, and construct an optical center deviation regularization term in the objective function based on the optical center deviation and a preset regularization term weight; wherein the optical center deviation regularization term includes the product of the L2 norm of the optical center deviation and the preset regularization term weight, wherein the optical center deviation includes a first deviation in the horizontal direction and a second deviation in the vertical direction, and the L2 norm of the optical center deviation includes the sum of the square of the first deviation and the square of the second deviation. The electronic device then updates the internal and external parameters of the imaging device by optimizing the objective function including the reprojection error term and the optical center deviation regularization term.

[0070] Among them, the reprojection error term ensures that the error between the current two-dimensional coordinates of the feature point and the target two-dimensional coordinates is minimized, thereby improving the ability of the internal and external parameters to accurately describe the actual projection relationship; the optical center deviation regularization term constrains the optical center position through the L2 norm of the optical center deviation, making it closer to the image center, thereby enhancing the geometric rationality of the calibration result. The introduction of the regularization term weight provides flexibility, enabling the optimization process to strike a balance between the accuracy of the reprojection error and the constraint of the optical center position. Ultimately, by optimizing the objective function containing the reprojection error term and the optical center deviation regularization term (for example, minimizing the value of the objective function), the interference of noise and deviation can be effectively reduced, the calibration accuracy of the internal and external parameters can be improved, and the convergence and robustness of the optimization can be ensured at the same time, providing reliable support for the high-precision application of actual imaging equipment.

[0071] For example, the objective function can be expressed as follows: N represents the number of feature points; Represents the current two-dimensional coordinates of the i-th feature point; Represents the target two-dimensional coordinates of the i-th feature point; Represents the difference between the current two-dimensional coordinates and the target two-dimensional coordinates of the i-th feature point, represents the reprojection error; λ represents the regularization weight, which is used to balance the optimization effects of the reprojection error and the optical center deviation; c currentIndicates the current optical center position indicated by the current projection model, c target represents the target optical center position, ‖c current -c target ‖ 2 represents the optical center deviation, which is constrained by the regularization term.

[0072] Exemplarily, the regularization term weight is less than or equal to 1, and the regularization term weight is positively correlated with the reprojection error determined in the first round of iteration. When the reprojection error determined in the first round of iteration of the above optimization process is large, the optimization process may cause the optical center position to deviate from expectations, so a stronger optical center deviation constraint (that is, setting a larger λ) is required to ensure that the optimization process does not sacrifice the rationality of the optical center position. When the reprojection error determined in the first round of iteration of the above optimization process is small, it means that the current projection model is close to the correct solution, and the constraint on the optical center position can be appropriately relaxed (that is, setting a smaller λ) to focus more on reducing the reprojection error. The regularization term weight λ determines the degree of influence of the optical center deviation on the objective function. By dynamically adjusting the regularization term weight, the relationship between minimizing the reprojection error and the optical center constraint can be flexibly balanced under different calibration conditions.

[0073] For example, in the process of optimizing the objective function, a variety of different optimization algorithms can be combined and applied to improve the optimization efficiency and result quality. For example, a variety of optimization algorithms (such as gradient direction, Newton direction, etc.) can be used to calculate multiple optimization directions at the same time. During the optimization process, each optimization algorithm will calculate an optimization direction based on the current parameters and the characteristics of the objective function. The electronic device can select an optimization direction from them, such as the direction that can minimize the objective function value, or the most stable direction (to avoid numerical divergence). By comparing the performance of different optimization algorithms at the same time in each iteration, it is possible to more quickly determine which optimization algorithm is more suitable for the current round of iteration, and different optimization algorithms have different search capabilities and convergence paths, which can reduce the possibility of falling into local extreme values.

[0074] In S306 , if both the reprojection error and the optical center deviation satisfy the preset convergence conditions or the current number of iterations reaches the preset maximum number of iterations, the optimization process ends.

[0075] Through the above steps, the initial internal and external parameters of the imaging device can be iteratively optimized to minimize the reprojection error and effectively constrain the optical center deviation, thereby improving the global accuracy and geometric rationality of the calibration.

[0076] In some embodiments, the regularization term weight used in the first optimization process can be pre-set, for example, it can be determined based on the size of the reprojection error determined in the first round of iteration of the first optimization process, or it can be an empirical value set by the developer, or it can be determined by other algorithms (such as cross-validation, grid search, L-curve analysis or Bayesian optimization algorithms). This embodiment does not impose any restrictions on this.

[0077] For example, after the above-described optimization process ends, if at least one of the reprojection error and optical center deviation obtained in the final iteration of the optimization process does not meet a preset convergence condition, the electronic device may adjust the regularization term weight used in the optimization process and re-perform the above-described optimization process using the adjusted regularization term weight. This embodiment dynamically adjusts the regularization term weight to flexibly address conflicts between different objectives during the optimization process, thereby achieving balanced optimization of the reprojection error and optical center deviation.

[0078] In one possible scenario, if the optical center deviation obtained in the final iteration of the optimization process exceeds a preset deviation threshold, the weight of the regularization term used in that optimization process is increased. In this case, the optical center position is insufficiently constrained from the image center, indicating that the optimization process focuses more on minimizing the reprojection error and ignores the geometric rationality of the optical center position. By increasing the regularization term weight and enhancing the influence of the optical center deviation regularization term in the objective function, the constraints on the optical center position can be strengthened, prompting the next optimization round to better correct the optical center deviation.

[0079] In another possible scenario, if the optical center deviation obtained in the last iteration of the optimization process is less than or equal to the preset deviation threshold, but the reprojection error obtained in the last iteration is greater than the preset error threshold, the weight of the regularization term used in this optimization process is reduced. In this case, the optical center deviation has met the requirements, but the reprojection error is still large, indicating that the optimization process may have focused too much on the constraints of the optical center position, resulting in insufficient attention to the actual fitting accuracy. In this case, by reducing the weight of the regularization term and reducing the influence of the optical center deviation regularization term, more optimization weight can be allocated to the reprojection error term, prompting the next optimization process to better reduce the reprojection error.

[0080] Through this dynamic adjustment, the optimization process can adaptively adjust between convergence and accuracy, thereby improving the global optimality and robustness of the final result. This adjustment mechanism can also prevent the optimization from falling into local optimality, further improving the accuracy and geometric rationality of the calibration results.

[0081] In other embodiments, effective efficiency and optimization quality can be improved through step-by-step optimization. The initial internal and external parameter optimization process for the imaging device can include a first step-by-step optimization process, a second step-by-step optimization process, and a joint optimization process.

[0082] The first step of the optimization process involves fixing all internal and external parameters of the imaging device except the optical center parameter, and then optimizing only the optical center parameter, which helps to more accurately correct optical center deviations. This approach ensures the independence of the optical center parameter optimization and reduces the impact of errors in other parameters on the optical center optimization.

[0083] The second-step optimization process is to first fix the optical center parameters determined in the first-step optimization process, and then optimize the internal and external parameters of the imaging device except the optical center parameters. This can avoid the excessive influence of the optical center deviation on the reprojection error, thereby more accurately adjusting other internal and external parameters such as focal length, principal point offset, and distortion coefficient.

[0084] The joint optimization process combines all internal and external parameters of the imaging device determined in both the first and second optimization steps. Building upon the parameters determined in the first and second optimization steps, the joint optimization further coordinates the relationships between these parameters, thereby improving the global consistency of the overall optimization results. This process integrates the locally optimal results from the first two steps to ultimately achieve a globally optimal solution.

[0085] Specifically, first, the electronic device uses the two-dimensional coordinates of the feature points in the calibration image as the target two-dimensional coordinates in all optimization processes, and determines the target optical center position in all optimization processes based on the image center of the calibration image.

[0086] Next, the electronic device uses the projection model indicated by the initial internal and external parameters of the imaging device as the current projection model in the following first round of iterations, and performs the following first step optimization process: the three-dimensional coordinates of the feature points are projected using the current projection model to obtain the current two-dimensional coordinates; based on the difference between the current two-dimensional coordinates of the feature points and the target two-dimensional coordinates, the reprojection error is determined, and the optical center deviation is obtained according to the deviation between the current optical center position indicated by the current projection model and the target optical center position. Determine whether the following conditions are met: the reprojection error and the optical center deviation meet the preset convergence conditions or the current number of iterations reaches the preset maximum number of iterations. If not, fix the other parameters except the optical center parameters in the internal and external parameters of the imaging device, and take minimizing the reprojection error and constraining the optical center deviation as the optimization goal, update the optical center parameters, and use the projection model indicated by the optical center parameters and the above-mentioned other fixed parameters as the current projection model in the next round of iterations. If so, end the first step optimization process.

[0087] Next, the electronic device uses the projection model indicated by the optical center parameters determined in the first sub-step optimization process and the other fixed parameters as the current projection model in the following first round of iteration, and performs the following second sub-step optimization process: using the current projection model to project the three-dimensional coordinates of the feature point to obtain the current two-dimensional coordinates; based on the difference between the current two-dimensional coordinates of the feature point and the target two-dimensional coordinates, the reprojection error is determined, and the optical center deviation is obtained according to the deviation between the current optical center position indicated by the current projection model and the target optical center position. Determine whether the following conditions are met: the reprojection error and the optical center deviation meet the preset convergence conditions or the current number of iterations reaches the preset maximum number of iterations. If not, fix the optical center parameters determined in the first sub-step optimization process, with minimizing the reprojection error and constraining the optical center deviation as the optimization goals, update the other parameters of the internal and external parameters of the imaging device except the optical center parameters, and use the fixed optical center parameters and the projection model indicated by the updated other parameters as the current projection model in the next round of iteration. If so, end the second sub-step optimization process.

[0088] Finally, the electronic device uses the projection model indicated by the optical center parameters determined in the first sub-step optimization process and the other parameters determined in the second sub-step optimization process as the current projection model in the following first round of iteration, and performs the following joint optimization process: the three-dimensional coordinates of the feature point are projected using the current projection model to obtain the current two-dimensional coordinates; based on the difference between the current two-dimensional coordinates of the feature point and the target two-dimensional coordinates, the reprojection error is determined, and the optical center deviation is obtained according to the deviation between the current optical center position indicated by the current projection model and the target optical center position. Determine whether the following conditions are met: the reprojection error and the optical center deviation meet the preset convergence conditions or the current number of iterations reaches the preset maximum number of iterations. If not, all internal and external parameters of the imaging device determined in the first and second sub-step optimization processes are updated with the optimization goal of minimizing the reprojection error and constraining the optical center deviation, and the projection model indicated by the updated internal and external parameters is used as the current projection model in the next round of iteration. If so, end the joint optimization process.

[0089] In this embodiment, step-by-step optimization, which first optimizes the optical center parameter and other parameters separately, effectively reduces interference between parameters and makes the optimization process more stable. Compared to the complex process of optimizing all parameters simultaneously, step-by-step optimization can gradually narrow the optimization space, reduce the computational workload, and improve optimization efficiency. Finally, combined with joint optimization, it can significantly reduce computational complexity while maintaining optimization accuracy.

[0090] In an exemplary embodiment, in a virtual shooting scene, a display (e.g., an LED screen) is typically used to display the virtual scene, which is combined with the real scene in front of the display to form a complete shooting image. When the imaging device captures the display and the real scene in front of the display, it is necessary to ensure that the geometric relationship and optical properties of the virtual scene and the real scene in the image are highly consistent, thereby achieving a seamless visual integration. Before actual shooting, a calibration pattern can be displayed on the display, and the internal and external parameters of the imaging device can be accurately calibrated using the above-mentioned calibration method.

[0091] Specifically, a specific calibration pattern, such as a checkerboard or dot pattern, is first displayed on the display. Since the display is easy to control, the three-dimensional coordinates of its feature points can be accurately determined through a three-dimensional scanning modeling method. The imaging device then captures the calibration pattern on the display and collects multiple calibration images to capture the optical imaging characteristics of the imaging device at different positions and postures. The above-mentioned calibration method is then used to establish a pairing relationship between the three-dimensional coordinates of the feature points and their two-dimensional coordinates in the calibration image, and the pairing relationship is used to complete the calibration and optimization process of the internal and external parameters of the imaging device. In particular, for telephoto imaging devices, multiple rounds of optimization can effectively reduce the reprojection error and constrain the optical center deviation, thereby ensuring high accuracy of the calibration results.

[0092] This method allows the imaging device's imaging model to accurately match the geometric relationship between the virtual content on the display and the real scene in a virtual shooting scene. This calibration process effectively improves the overall fusion of virtual and real objects in virtual shooting, reduces the complexity of post-processing, and improves the realism and consistency of visual presentation.

[0093] The various technical features in the above embodiments can be combined arbitrarily as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, the arbitrary combination of the various technical features in the above embodiments also falls within the scope of disclosure of this specification.

[0094] In some embodiments, an embodiment of this specification further provides an electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements any of the above methods by running the executable instructions.

[0095] Figure 4 This is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 4At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408, and a non-volatile memory 410. Of course, it may also include hardware required for other functions. One or more embodiments of this specification can be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into the memory 408 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0096] Please refer to Figure 5 , the calibration device of the imaging device can be applied to Figure 4 The device shown in the figure is used to implement the technical solution of this specification. The calibration device of the imaging device may include:

[0097] The calibration image acquisition module 501 is used to acquire multiple calibration images, where the calibration images are obtained by photographing a calibration pattern by an imaging device to be calibrated.

[0098] The pairing relationship establishing module 502 is configured to establish a pairing relationship between the three-dimensional coordinates of a feature point in the calibration pattern and the two-dimensional coordinates of the feature point in the calibration image.

[0099] The calibration module 503 is configured to calibrate the imaging device using the pairing relationship to obtain initial internal and external parameters of the imaging device.

[0100] The optimization module 504 is configured to optimize the initial internal and external parameters of the imaging device by utilizing the pairing relationship, with the goal of minimizing the reprojection error and constraining the optical center deviation.

[0101] In one implementation, the initial intrinsic and extrinsic parameters of the imaging device include a focal length. The optimization module 504 is specifically configured to optimize the initial intrinsic and extrinsic parameters of the imaging device using the pairing relationship, with minimizing reprojection error and constraining optical center deviation as optimization objectives, if the focal length of the imaging device satisfies a telephoto condition.

[0102] In one implementation, the optimization module 504 is specifically used to use the two-dimensional coordinates of the feature point in the calibration image as the target two-dimensional coordinates in the optimization process, determine the target optical center position in the optimization process based on the image center of the calibration image, and use the projection model indicated by the initial internal and external parameters of the imaging device as the current projection model in the first round of iteration, and perform the following optimization process: use the current projection model to project the three-dimensional coordinates of the feature point to obtain the current two-dimensional coordinates; based on the difference between the current two-dimensional coordinates of the feature point and the target two-dimensional coordinates, determine the reprojection error, and obtain the optical center deviation based on the deviation between the current optical center position indicated by the current projection model and the target optical center position; if at least one of the reprojection error and the optical center deviation does not meet the preset convergence condition and the current number of iterations does not reach the preset maximum number of iterations, minimize the reprojection error and constrain the optical center deviation as the optimization target, update the internal and external parameters of the imaging device, and use the projection model indicated by the updated internal and external parameters as the current projection model in the next round of iteration.

[0103] In one implementation, the optimization module 504 is specifically used to construct a reprojection error term in the objective function based on the reprojection error, and to construct an optical center deviation regularization term in the objective function based on the optical center deviation and a preset regularization term weight; by optimizing the objective function including the reprojection error term and the optical center deviation regularization term, the internal and external parameters of the imaging device are updated.

[0104] In one implementation, the optical center deviation regularization term includes the product of the L2 norm of the optical center deviation and the preset regularization term weight, wherein the optical center deviation includes a first deviation in the horizontal direction and a second deviation in the vertical direction, and the L2 norm of the optical center deviation includes the sum of the square of the first deviation and the square of the second deviation.

[0105] In one implementation, the regularization term weight is less than or equal to 1, and the regularization term weight is positively correlated with the reprojection error determined in the first round of iterations.

[0106] In one implementation, the method further includes a regularization term weight adjustment module for adjusting the regularization term weight used in the optimization process after the optimization process is completed, if at least one of the reprojection error and the optical center deviation obtained in the last round of iteration of the optimization process does not meet the preset convergence condition, and re-performing the optimization process with the adjusted regularization term weight.

[0107] In one implementation, the regularization term weight adjustment module is specifically used to increase the regularization term weight used in the optimization process if the optical center deviation obtained in the last round of iteration is greater than a preset deviation threshold; if the optical center deviation obtained in the last round of iteration is less than or equal to the preset deviation threshold, but the reprojection error obtained in the last round of iteration is greater than a preset error threshold, reduce the regularization term weight used in the optimization process.

[0108] In one implementation, the optimization process includes a first step optimization process, a second step optimization process, and a joint optimization process.

[0109] In the first step optimization process, the optimization module 504 is specifically used to fix the other parameters of the internal and external parameters of the imaging device except the optical center parameter, so as to minimize the reprojection error and constrain the optical center deviation as the optimization goal, and update the optical center parameter.

[0110] In the second step-by-step optimization process, the optimization module 504 is specifically used to fix the optical center parameters determined in the first step-by-step optimization process, so as to minimize the reprojection error and constrain the optical center deviation as the optimization goal, and update other parameters of the internal and external parameters of the imaging device except the optical center parameters.

[0111] In the joint optimization process, the optimization module 504 is specifically used to update all internal and external parameters of the imaging device determined in the first step optimization process and the second step optimization process with minimizing the reprojection error and constraining the optical center deviation as the optimization goal.

[0112] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0113] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0114] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0115] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.

[0116] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A calibration method for an imaging device, comprising: Acquire multiple calibration images, where the calibration images are obtained by photographing the calibration pattern by the imaging device to be calibrated; Establishing a pairing relationship between the three-dimensional coordinates and the two-dimensional coordinates of the feature point in the calibration pattern, where the two-dimensional coordinates are the two-dimensional coordinates of the feature point in the calibration image; Using the pairing relationship, calibrating the imaging device to obtain initial internal and external parameters of the imaging device; With minimizing the reprojection error and constraining the optical center deviation as the optimization goal, the pairing relationship is used to optimize the initial internal and external parameters of the imaging device; the optimization of the initial internal and external parameters of the imaging device includes: in each round of iterative optimization process, constructing a reprojection error term in the objective function based on the reprojection error determined in the current round, and constructing an optical center deviation regularization term in the objective function based on the optical center deviation determined in the current round and a preset regularization term weight, and updating the internal and external parameters of the imaging device by optimizing the objective function including the reprojection error term and the optical center deviation regularization term; wherein the regularization term weight is less than or equal to 1, and the regularization term weight is positively correlated with the reprojection error determined in the first round of iteration.

2. The method according to claim 1, wherein the initial internal and external parameters of the imaging device include focal length; The optimization objective is to minimize the reprojection error and constrain the optical center deviation, and to optimize the initial internal and external parameters of the imaging device using the pairing relationship, including: If the focal length of the imaging device meets the telephoto condition, the initial internal and external parameters of the imaging device are optimized using the pairing relationship with the optimization goal of minimizing the reprojection error and constraining the optical center deviation.

3. The method according to claim 1, wherein the optimization objective is to minimize the reprojection error and constrain the optical center deviation, and to optimize the initial internal and external parameters of the imaging device using the pairing relationship, comprising: The two-dimensional coordinates of the feature points in the calibration image are used as the target two-dimensional coordinates in the optimization process. The target optical center position in the optimization process is determined based on the image center of the calibration image. The projection model indicated by the initial internal and external parameters of the imaging device is used as the current projection model in the first round of iteration. The following optimization process is performed: Projecting the three-dimensional coordinates of the feature point using the current projection model to obtain current two-dimensional coordinates; determining a reprojection error based on a difference between the current two-dimensional coordinates of the feature point and the target two-dimensional coordinates, and obtaining an optical center deviation according to a deviation between a current optical center position indicated by the current projection model and the target optical center position; If at least one of the reprojection error and the optical center deviation does not meet the preset convergence condition and the current number of iterations does not reach the preset maximum number of iterations, the internal and external parameters of the imaging device are updated with the optimization goal of minimizing the reprojection error and constraining the optical center deviation, and the projection model indicated by the updated internal and external parameters is used as the current projection model in the next round of iteration.

4. The method according to claim 1, wherein the optical center deviation regularization term comprises the product of the L2 norm of the optical center deviation and the preset regularization term weight, wherein: The optical center deviation includes a first deviation in the horizontal direction and a second deviation in the vertical direction, and the L2 norm of the optical center deviation includes the sum of the square of the first deviation and the square of the second deviation.

5. The method according to claim 1, further comprising: After the optimization process is completed, if at least one of the reprojection error and the optical center deviation obtained in the last round of iteration of the optimization process does not meet the preset convergence condition, the regularization term weight used in the optimization process is adjusted, and the optimization process is repeated with the adjusted regularization term weight.

6. The method according to claim 5, wherein if at least one of the reprojection error and the optical center deviation obtained in the last iteration of the optimization process does not meet the preset convergence condition, adjusting the regularization term weight used in the optimization process comprises: If the optical center deviation obtained in the last round of iteration is greater than a preset deviation threshold, increasing the weight of the regularization term used in the optimization process; If the optical center deviation obtained in the last round of iteration is less than or equal to the preset deviation threshold, but the reprojection error obtained in the last round of iteration is greater than the preset error threshold, the regularization term weight used in the optimization process is reduced.

7. The method according to claim 3, wherein the optimization process comprises a first step optimization process, a second step optimization process and a joint optimization process; In the first step optimization process, the updating of the internal and external parameters of the imaging device with minimizing the reprojection error and constraining the optical center deviation as the optimization goal includes: Fixing the other parameters of the internal and external parameters of the imaging device except the optical center parameter, and updating the optical center parameter with the optimization goal of minimizing the reprojection error and constraining the optical center deviation; In the second step optimization process, the updating of the internal and external parameters of the imaging device with minimizing the reprojection error and constraining the optical center deviation as the optimization goal includes: Fixing the optical center parameter determined in the first step optimization process, taking minimizing the reprojection error and constraining the optical center deviation as the optimization goal, and updating the other parameters of the internal and external parameters of the imaging device except the optical center parameter; In the joint optimization process, the updating of the internal and external parameters of the imaging device with minimizing the reprojection error and constraining the optical center deviation as the optimization goal includes: With minimizing the reprojection error and constraining the optical center deviation as the optimization goal, all internal and external parameters of the imaging device determined in the first step optimization process and the second step optimization process are updated.

8. An electronic device comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method according to any one of claims 1 to 7 by executing the executable instructions.

9. A computer-readable storage medium having computer instructions stored thereon, wherein when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images