Camera distortion calibration method and device based on NURBS, storage medium and product

Abstract

The invention relates to the field of computer vision, in particular to a camera distortion calibration method and device based on NURBS, a storage medium and a product. The method comprises the following steps: obtaining a calibration image, wherein the calibration image is an image obtained by shooting a calibration plate through a camera; according to the calibration image, an NURBS curved surface model is constructed, the NURBS curved surface model is used for representing a mapping relation between two-dimensional pixel coordinates of a plurality of feature points on the calibration plate and a light vector, and the light vector is used for representing a light direction pointing to the feature points from the optical center of the camera; and performing distortion correction on other images shot by the camera according to the constructed NURBS curved surface model. According to the embodiment of the invention, the NURBS curved surface model is constructed, and the model can accurately represent the mapping relation between the two-dimensional pixel coordinates of the feature points on the calibration plate and the light vector, and can fit non-uniform distortion in any form, thereby remarkably improving the precision of camera distortion correction.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision, and in particular to a camera distortion calibration method, apparatus, storage medium, and product based on non-uniform rational B-splines (NURBS). Background Technology

[0002] In the field of computer vision, camera distortion calibration is a fundamental task for accurate measurement and data acquisition. Its purpose is to establish a precise mapping relationship between the image's planar projection and the real three-dimensional space by acquiring the camera's intrinsic and extrinsic parameters. However, current technologies have many shortcomings in camera distortion calibration.

[0003] Traditional camera distortion calibration methods are primarily based on pinhole camera models, compensating for lens distortion by introducing correction terms such as radial distortion, tangential distortion, and thin prism distortion. While these methods improve calibration accuracy to some extent, they still have limitations. For example, the pinhole camera model assumes that lens distortion is mainly concentrated in the central region, neglecting the complex distortion in the edge regions, leading to insufficient calibration accuracy. Furthermore, these methods typically rely on specific calibration equipment and complex optimization algorithms, making them difficult to adapt to different types of cameras, especially non-professional cameras such as mobile phone cameras and surveillance video cameras. Currently, a reasonable and effective camera distortion calibration method has not yet been provided in the relevant technologies. Summary of the Invention

[0004] In view of this, this disclosure presents a camera distortion calibration method, apparatus, storage medium, and product based on NURBS.

[0005] According to one aspect of this disclosure, a NURBS-based camera distortion calibration method is provided, the method comprising:

[0006] Acquire a calibration image, which is an image of the calibration board obtained by taking a picture with a camera;

[0007] Based on the calibration image, a NURBS surface model is constructed. The NURBS surface model is used to represent the mapping relationship between the two-dimensional pixel coordinates of multiple feature points on the calibration board and the ray vector. The ray vector is used to represent the ray direction from the optical center of the camera to the feature point.

[0008] Based on the constructed NURBS surface model, distortion correction is performed on other images captured by the camera.

[0009] In one possible implementation, constructing the NURBS surface model based on the calibration image includes:

[0010] Based on the calibration image, extract the two-dimensional pixel coordinates of multiple feature points on the calibration board in the imaging plane coordinate system, wherein the imaging plane coordinate system is a two-dimensional coordinate system defined based on the imaging plane of the camera;

[0011] Obtain the ray vectors corresponding to the multiple feature points in the camera coordinate system, wherein the camera coordinate system is a three-dimensional coordinate system with the camera optical center as the origin;

[0012] The NURBS surface model is constructed based on the two-dimensional pixel coordinates of multiple feature points and the corresponding ray vectors.

[0013] In another possible implementation, obtaining the ray vectors corresponding to the multiple feature points in the camera coordinate system includes:

[0014] Obtain the three-dimensional coordinates of multiple feature points in the calibration plate coordinate system, wherein the calibration coordinate system is a three-dimensional coordinate system defined based on the plane of the calibration plate;

[0015] For each feature point, the three-dimensional coordinates of the feature point in the camera coordinate system are determined based on the three-dimensional coordinates of the feature point and the camera optical center in the calibration plate coordinate system.

[0016] The corresponding ray vector is determined based on the three-dimensional coordinates of each feature point in the camera coordinate system.

[0017] In another possible implementation, constructing the NURBS surface model based on the two-dimensional pixel coordinates of the plurality of feature points and the corresponding ray vectors includes:

[0018] The two-dimensional pixel coordinates of each feature point are determined as the parameter coordinates of the NURBS surface model, and the parameter coordinates are used to represent the position of the feature point in the parameter space of the NURBS surface model;

[0019] The ray vector corresponding to each feature point is determined as the coordinates of the control point of the NURBS surface model, and a corresponding weight is set for each control point. The control point is used to represent the shape of the NURBS surface model, and the weight is used to represent the degree of influence of the control point on the NURBS surface.

[0020] The NURBS surface model is constructed based on the parameter coordinates corresponding to each feature point, the coordinates of the control point, and the weight.

[0021] In another possible implementation, the distortion correction of other images captured by the camera based on the constructed NURBS surface model includes:

[0022] Based on other images captured by the camera, extract the two-dimensional pixel coordinates of each pixel in the other images;

[0023] For the two-dimensional pixel coordinates of each pixel, the corresponding corrected ray vector is determined by the constructed NURBS surface model;

[0024] The distortion-corrected image is synthesized based on the corrected ray vector corresponding to each pixel.

[0025] In another possible implementation, the step of synthesizing the distortion-corrected image based on the corrected ray vector corresponding to each pixel includes:

[0026] The corrected ray vector corresponding to each pixel is projected onto the ideal imaging plane to obtain the corresponding projection coordinates;

[0027] Based on the camera parameters of the camera, the projected coordinates corresponding to each pixel are converted into corrected two-dimensional pixel coordinates;

[0028] The distortion-corrected image is synthesized based on the corrected two-dimensional pixel coordinates of each pixel.

[0029] According to another aspect of this disclosure, a NURBS-based camera distortion calibration device is provided, the device comprising:

[0030] The acquisition module is used to acquire calibration images, which are images of the calibration board obtained by taking pictures with a camera.

[0031] The construction module is used to construct a NURBS surface model based on the calibration image. The NURBS surface model is used to represent the mapping relationship between the two-dimensional pixel coordinates of multiple feature points on the calibration board and the ray vector. The ray vector is used to represent the direction of light rays from the optical center of the camera to the feature point.

[0032] The correction module is used to perform distortion correction on other images captured by the camera based on the constructed NURBS surface model.

[0033] In one possible implementation, the building module is further configured to:

[0034] Based on the calibration image, extract the two-dimensional pixel coordinates of multiple feature points on the calibration board in the imaging plane coordinate system, wherein the imaging plane coordinate system is a two-dimensional coordinate system defined based on the imaging plane of the camera;

[0035] Obtain the ray vectors corresponding to the multiple feature points in the camera coordinate system, wherein the camera coordinate system is a three-dimensional coordinate system with the camera optical center as the origin;

[0036] The NURBS surface model is constructed based on the two-dimensional pixel coordinates of multiple feature points and the corresponding ray vectors.

[0037] In another possible implementation, the building module is also used for:

[0038] Obtain the three-dimensional coordinates of multiple feature points in the calibration plate coordinate system, wherein the calibration coordinate system is a three-dimensional coordinate system defined based on the plane of the calibration plate;

[0039] For each feature point, the three-dimensional coordinates of the feature point in the camera coordinate system are determined based on the three-dimensional coordinates of the feature point and the camera optical center in the calibration plate coordinate system.

[0040] The corresponding ray vector is determined based on the three-dimensional coordinates of each feature point in the camera coordinate system.

[0041] In another possible implementation, the building module is also used for:

[0042] The two-dimensional pixel coordinates of each feature point are determined as the parameter coordinates of the NURBS surface model, and the parameter coordinates are used to represent the position of the feature point in the parameter space of the NURBS surface model;

[0043] The ray vector corresponding to each feature point is determined as the coordinates of the control point of the NURBS surface model, and a corresponding weight is set for each control point. The control point is used to represent the shape of the NURBS surface model, and the weight is used to represent the degree of influence of the control point on the NURBS surface.

[0044] The NURBS surface model is constructed based on the parameter coordinates corresponding to each feature point, the coordinates of the control point, and the weight.

[0045] In another possible implementation, the correction module is further configured to:

[0046] Based on other images captured by the camera, extract the two-dimensional pixel coordinates of each pixel in the other images;

[0047] For the two-dimensional pixel coordinates of each pixel, the corresponding corrected ray vector is determined by the constructed NURBS surface model;

[0048] The distortion-corrected image is synthesized based on the corrected ray vector corresponding to each pixel.

[0049] In another possible implementation, the correction module is further configured to:

[0050] The corrected ray vector corresponding to each pixel is projected onto the ideal imaging plane to obtain the corresponding projection coordinates;

[0051] Based on the camera parameters of the camera, the projected coordinates corresponding to each pixel are converted into corrected two-dimensional pixel coordinates;

[0052] The distortion-corrected image is synthesized based on the corrected two-dimensional pixel coordinates of each pixel.

[0053] According to another aspect of this disclosure, a NURBS-based camera distortion calibration apparatus is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0054] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0055] According to another aspect of this disclosure, a computer program product is provided, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0056] This disclosure provides a NURBS-based camera distortion calibration method. The method acquires a calibration image (an image obtained by photographing a calibration board with a camera) and constructs a NURBS surface model based on the calibration image. This NURBS surface model accurately represents the mapping relationship between the two-dimensional pixel coordinates of multiple feature points on the calibration board and ray vectors, where the ray vectors represent the direction of light rays from the camera's optical center to the feature points. Furthermore, the constructed NURBS surface model is used to correct distortion in other images captured by the camera. This method offers the following advantages: Firstly, it achieves high-precision distortion correction: the NURBS surface model can accurately represent the mapping relationship between the two-dimensional pixel coordinates of feature points on the calibration board and ray vectors, and can fit arbitrary forms of non-uniform distortion, thereby significantly improving the accuracy of camera distortion correction. Secondly, it is highly adaptable: this method is compatible with multiple calibration methods and is applicable to cameras of different types and accuracy requirements, including non-professional cameras (such as mobile phone cameras, surveillance video cameras, etc.), solving the problem of poor calibration results for non-professional cameras using traditional methods. On the other hand, it offers fast and robust parameter estimation: combined with the original algorithm, it achieves fast and robust parameter estimation, reducing computational complexity and dependence on initial parameters, thus improving calibration efficiency. Furthermore, it boasts strong versatility: this method does not rely on prior camera knowledge and does not require complex calibration equipment, exhibiting strong versatility and practicality. Finally, it enhances image processing accuracy: through precise distortion correction, it significantly improves image processing accuracy, providing higher-quality data support for subsequent image analysis, 3D reconstruction, and other tasks.

[0057] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0058] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0059] Figure 1 A schematic diagram of the structure of a computing device provided in an exemplary embodiment of the present disclosure is shown.

[0060] Figure 2 A flowchart of a NURBS-based camera distortion calibration method provided in an exemplary embodiment of this disclosure is shown.

[0061] Figure 3 A flowchart of a NURBS-based camera distortion calibration method provided in another exemplary embodiment of this disclosure is shown.

[0062] Figure 4A schematic diagram of a calibration experimental apparatus provided in an exemplary embodiment of this disclosure is shown.

[0063] Figure 5 A schematic diagram of a scanning scenario provided by an exemplary embodiment of this disclosure is shown.

[0064] Figure 6 A schematic diagram of a real-world point cloud provided by an exemplary embodiment of this disclosure is shown.

[0065] Figure 7 A schematic diagram of reference point selection provided by an exemplary embodiment of this disclosure is shown.

[0066] Figure 8 A schematic diagram showing a data comparison of two correction methods provided in an exemplary embodiment of this disclosure is illustrated.

[0067] Figure 9 The diagram illustrates the offset distance and pixel offset direction of a mobile phone lens provided in an exemplary embodiment of this disclosure before and after calibration.

[0068] Figure 10 The diagram illustrates the offset distance and pixel offset direction of a mobile phone lens provided in an exemplary embodiment of this disclosure before and after calibration.

[0069] Figure 11 The diagram illustrates the offset distance and pixel offset direction of a C-type camera lens before and after calibration, according to an exemplary embodiment of this disclosure.

[0070] Figure 12 This is a block diagram illustrating an apparatus according to an exemplary embodiment. Detailed Implementation

[0071] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0072] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0073] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0074] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are used only to distinguish one element / operation from another. Therefore, without departing from the teachings of the conception of embodiments of this disclosure, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0075] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0076] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0077] In recent years, with the rapid development of high-precision sensor technology, computer vision-based non-contact monitoring methods have rapidly emerged and been widely applied in the field of high-precision monitoring. This innovative technology accurately acquires spatial information of target objects in a non-contact manner, and with the help of advanced image processing and 3D reconstruction technologies, it can efficiently achieve high-precision digital modeling and in-depth analysis of complex scenes. It is widely used in cultural heritage protection, construction management, smart city operation and maintenance, geological disaster monitoring, and virtual and augmented reality, providing efficient and accurate non-contact monitoring methods for tasks such as ancient building health monitoring, construction quality inspection, urban operation status perception, and geological disaster early warning and assessment. Compared with traditional contact or manual surveying methods, non-contact monitoring technology has significant advantages such as high efficiency, non-destructiveness, high precision, and real-time performance, effectively solving the problems of traditional surveying methods being time-consuming, labor-intensive, costly, and difficult to adapt to complex environments. Furthermore, this technology also demonstrates extremely important application value in fields such as autonomous driving, navigation and positioning, and general-purpose robots.

[0078] Camera calibration, a cornerstone of computer vision, plays an irreplaceable and crucial role in precise measurement and data acquisition. During camera calibration, by accurately acquiring the camera's intrinsic and extrinsic parameters, a precise mapping relationship can be established between the image plane projection and the real three-dimensional space. This technology not only significantly improves the accuracy of image processing but also provides a solid foundation for the fusion and analysis of multi-view image data, demonstrating significant application value in numerous fields. The camera calibration process typically includes two core stages: camera modeling and camera calibration. The goal of camera modeling is to accurately describe the physical characteristics and optical behavior of the sensor using a set of mathematical parameters, while camera calibration estimates the specific values ​​of these parameters through direct calculation or optimization iteration. In a complete camera mathematical model, the intrinsic parameters detail the mapping relationship of light rays projected onto the imaging plane through the lens system, while the extrinsic parameters define the camera's spatial position and orientation in the world coordinate system.

[0079] The pinhole camera model, as a fundamental and widely used camera model, holds significant importance in both theory and practice. However, due to factors such as lens manufacturing errors, assembly deviations, and the inherent inhomogeneity of materials, pinhole camera models often exhibit certain errors in practical applications. To improve model accuracy, correction terms such as radial distortion, tangential distortion, and thin prism distortion have been introduced. For professional cameras, due to high manufacturing precision requirements, their lenses generally conform to mathematical models of radial distortion, tangential distortion, and thin prism distortion, achieving high accuracy through calibration. However, non-professional cameras such as mobile phone cameras, surveillance video cameras, and mass-market digital cameras, due to manufacturing process and cost limitations, often do not fully conform to these mathematical models. Their errors exhibit a degree of randomness, leading to poor calibration results, and the massive amounts of data collected are insufficient to directly meet the demands of high-precision monitoring.

[0080] To address this issue, this disclosure proposes a novel detection and calibration model for distortion in ordinary camera images. This model, based on the NURBS surface principle for camera modeling, can flexibly fit arbitrary forms of non-uniform distortion and is compatible with various calibration methods. Combined with the original algorithm of this disclosure, fast and robust parameter estimation can be achieved. In tests across multiple real-world scenarios, the method provided by this disclosure has demonstrated superior performance, offering not only a novel solution for precise visual measurement in the construction field but also showing broad application prospects in emerging fields such as smart cities, autonomous driving, and intelligent security, potentially driving technological progress and development in related industries.

[0081] First, the application scenarios involved in this disclosure will be introduced. Please refer to... Figure 1 The diagram illustrates a structural schematic of a computing device provided in an exemplary embodiment of this disclosure.

[0082] Computing devices can be terminals or servers. Terminals include mobile terminals or fixed terminals, such as mobile phones, tablets, laptops, and desktop computers. A server can be a single server, a server cluster consisting of several servers, or a cloud computing service center.

[0083] The computing device includes a processor 10, a memory 20, and a communication interface 30. Those skilled in the art will understand that... Figure 1 The structure shown does not constitute a limitation on the computing device and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. Wherein:

[0084] The processor 10 is the control center of the computing device. It connects various parts of the computing device through various interfaces and lines. By running or executing software programs and / or modules stored in the memory 20, and by calling data stored in the memory 20, it performs various functions of the computing device and processes data, thereby controlling the computing device as a whole. The processor 10 can be implemented by a central processing unit (CPU) or a graphics processing unit (GPU).

[0085] The memory 20 can be used to store software programs and modules. The processor 10 executes various functional applications and data processing by running the software programs and modules stored in the memory 20. The memory 20 may mainly include a program storage area and a data storage area. The program storage area may store the operating system 21, the acquisition module 22, the construction module 23, the correction module 24, and the application program 25 required for at least one function, etc.; the data storage area may store data created according to the use of the computing device, etc. The memory 20 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Accordingly, memory 20 may also include a memory controller to provide processor 10 with access to memory 20.

[0086] The processor 10 performs the following functions by running the acquisition module 22: acquiring a calibration image, which is an image obtained by the camera from the calibration board; the processor 10 performs the following functions by running the construction module 23: constructing a NURBS surface model based on the calibration image, the NURBS surface model is used to represent the mapping relationship between the two-dimensional pixel coordinates of multiple feature points on the calibration board and the ray vector, the ray vector is used to represent the direction of the ray from the optical center of the camera to the feature point; the processor 10 performs the following functions by running the correction module 24: performing distortion correction on other images captured by the camera based on the constructed NURBS surface model.

[0087] The methods provided in the embodiments of this disclosure will now be described using several exemplary embodiments.

[0088] Please refer to Figure 2 It illustrates a flowchart of a NURBS-based camera distortion calibration method provided in an exemplary embodiment of this disclosure. This embodiment uses this method for... Figure 1 The following example uses a computing device as an illustration. The method includes the following steps.

[0089] Step 201: Obtain the calibration image, which is an image of the calibration board obtained by taking a picture with a camera.

[0090] A calibration image is an image obtained by taking a picture of a calibration board with a camera. It is used for parameter estimation and distortion correction during the camera calibration process. The calibration image contains the image of the calibration board, which is usually an object with known geometric features (such as a checkerboard or an array of dots). Its feature points (such as the corners of the checkerboard or the center point of the array of dots) are clearly visible in the image.

[0091] A camera is an imaging device used to capture calibration images. It focuses light from a scene onto an imaging plane to form an image.

[0092] A computing device can capture an optical image of a calibration board using a camera's imaging system, converting it into a digital image. In some embodiments, the calibration board is placed facing the camera, ensuring that the plane of the calibration board is approximately perpendicular to the camera's optical axis and that the feature points of the calibration board are within the camera's field of view. The calibration board is captured using the camera's imaging system, and the camera's focal length and exposure parameters are adjusted to make the feature points of the calibration board clearly visible in the image. The captured calibration image is then stored digitally in a computer or other storage device for subsequent image processing and analysis.

[0093] Step 202: Based on the calibration image, construct a NURBS surface model. The NURBS surface model is used to represent the mapping relationship between the two-dimensional pixel coordinates of multiple feature points on the calibration board and the ray vector. The ray vector is used to represent the direction of the ray from the optical center of the camera to the feature point.

[0094] NURBS surface models are mathematical models used to represent complex surfaces. They define the shape of a surface using a set of control points, weights, and basis functions, enabling flexible fitting of various complex geometries. In camera calibration, NURBS surface models are used to represent the mapping relationship between the two-dimensional pixel coordinates of multiple feature points on a calibration board and ray vectors.

[0095] Feature points on a calibration plate refer to points with well-defined geometric positions and shapes. They are typically used as known reference points during camera calibration and are key points for establishing the relationship between the image plane and three-dimensional space. These points have a known geometric arrangement on the calibration plate and can be clearly detected in the calibration image.

[0096] The positions of feature points on the calibration board are predefined and typically follow a regular arrangement, such as the corners of a checkerboard or the center points of a dot array. Feature points are usually located at the boundaries of black and white squares on the calibration board or other high-contrast areas, facilitating detection by image processing algorithms. The positions of feature points on the calibration board are fixed and do not change due to variations in the board's placement angle or ambient lighting conditions. The number and density of feature points affect the accuracy and reliability of the calibration. Generally, a greater number and more uniform distribution of feature points result in more accurate calibration results.

[0097] In a checkerboard calibration board, corner points are the intersections of the black and white squares. These points form a regular grid arrangement on the calibration board and are the most commonly used feature point type in camera calibration. In a dot array calibration board, feature points are the center points of the dots. These points are typically extracted using a center detection algorithm. Besides checkerboard and dot arrays, other shapes of calibration boards can also be used, such as striped calibration boards and random dot matrix calibration boards, whose feature point shapes and arrangements will differ.

[0098] The two-dimensional pixel coordinates of each feature point represent its position on the camera's imaging plane, expressed in two-dimensional coordinates. These coordinates are the location of the feature point detected from the calibration image using image processing algorithms.

[0099] A ray vector is a vector representing the direction of light rays from the camera's optical center to a feature point. It is a three-dimensional vector representing the direction of the light ray in the camera coordinate system. In camera calibration, ray vectors are used to establish the geometric relationship between the image plane and three-dimensional space.

[0100] The computing device can construct a NURBS surface model based on feature point information in the calibration image using mathematical modeling methods. In some embodiments, the two-dimensional pixel coordinates of multiple feature points are extracted from the calibration image, typically using image processing algorithms. The ray vectors corresponding to the multiple feature points in the camera coordinate system are then determined based on the geometric features of the calibration board. Control points, weights, and basis function parameters of the NURBS surface model are fitted based on the extracted two-dimensional pixel coordinates and corresponding ray vectors. The accuracy of the fitted NURBS surface model is verified to ensure it accurately represents the mapping relationship between the two-dimensional pixel coordinates of the feature points on the calibration board and the ray vectors. It should be noted that the details of constructing the NURBS surface model can be found in the relevant descriptions in the embodiments below, and will not be elaborated upon here.

[0101] Step 203: Based on the constructed NURBS surface model, perform distortion correction on other images captured by the camera.

[0102] Distortion correction is the process of processing images captured by a camera to eliminate or reduce image distortion. Distortion refers to image distortion caused by the optical characteristics of the camera lens (such as radial distortion, tangential distortion, etc.). The purpose of distortion correction is to restore the true geometry of the image and improve its accuracy and quality.

[0103] Other images are those captured by the camera, besides the calibration images, that require distortion correction. These other images can be target images from real-world application scenarios, such as images of building structures or cityscapes.

[0104] In some embodiments, for other images that need correction, the two-dimensional pixel coordinates of each pixel in the other images are extracted; the corrected ray vector corresponding to each two-dimensional pixel coordinate is calculated using a NURBS surface model; and the corrected ray vector is projected onto an ideal imaging plane to determine the distortion-corrected image. It should be noted that details regarding distortion correction can be found in the relevant descriptions in the embodiments below, and will not be elaborated upon here.

[0105] This embodiment utilizes a calibration board to obtain camera intrinsic parameters and distortion parameters, and establishes a mapping relationship between two-dimensional pixel coordinates and ray vectors by constructing a NURBS surface model, thereby achieving distortion correction. Please refer to... Figure 3 This document illustrates a flowchart of a NURBS-based camera distortion calibration method provided in another exemplary embodiment of this disclosure. This embodiment uses this method for... Figure 1 The following example uses a computing device as an illustration. The method includes the following steps.

[0106] Step 301: Acquire calibration images and obtain the two-dimensional pixel coordinates of multiple feature points on the calibration board.

[0107] The computing device can acquire calibration images through a camera; based on the calibration images, it can extract the two-dimensional pixel coordinates of multiple feature points on the calibration board in the imaging plane coordinate system, which is a two-dimensional coordinate system defined based on the camera's imaging plane.

[0108] An imaging plane coordinate system is a two-dimensional coordinate system established with the camera's imaging plane as a reference, used to describe the positional relationships of pixels in an image. The imaging plane coordinate system is a two-dimensional Cartesian coordinate system, with its origin typically located at the top left corner of the image (common in computer vision) or the center of the image (possibly used in some optical systems). The horizontal axis (usually the x-axis) represents the horizontal direction of the image, and the vertical axis (usually the y-axis) represents the vertical direction. Each pixel in the image can be represented by a pair of coordinate values ​​(x, y) in the imaging plane coordinate system.

[0109] Taking a checkerboard calibration board as an example, and corner points as feature points, firstly, a uniform checkerboard calibration board is selected, where the side length of each black and white square is a known unit length. The calibration board is placed facing the camera, ensuring that the plane of the calibration board is approximately perpendicular to the optical axis of the camera, and then a calibration image is captured. Illustratively, the two-dimensional pixel coordinates of all corner points on the calibration board in the imaging plane coordinate system are extracted using a checkerboard feature point detection algorithm.

[0110] {(x pix,ij ,y pix,ij )}

[0111] Where, (x pix,ij ,y pix,ij () represents the two-dimensional pixel coordinates of the corner point in the i-th row and j-th column of the imaging plane coordinate system, where i represents the corner point index in the x-direction of the calibrated image, i∈[1,N]. x ], N x The x-axis represents the number of corner points on the checkerboard calibration board in the x-direction; j represents the corner point index of the calibration image in the j-direction, j∈[1,N]. y ], N y This indicates the number of corner points of the chessboard calibration board in the y-direction.

[0112] The computing device can obtain the three-dimensional coordinates of multiple feature points on the calibration plate in the calibration plate coordinate system, as well as the three-dimensional coordinates of the camera optical center in the calibration plate coordinate system. The calibration coordinate system is a three-dimensional coordinate system defined based on the plane of the calibration plate.

[0113] The calibration coordinate system is a three-dimensional Cartesian coordinate system defined with the plane of the calibration plate as a reference. It is used to describe the positions of feature points on the calibration plate and the optical center of the camera within this coordinate system. The origin of the calibration coordinate system is usually located at a specific point on the calibration plate (e.g., the center of the calibration plate or a feature point). The coordinate axes are determined by the plane of the calibration plate and the distribution of feature points. Typically, the three coordinate axes of the calibration coordinate system are defined as follows: X-axis: usually parallel to the plane of the calibration plate, pointing horizontally. Y-axis: usually parallel to the plane of the calibration plate, pointing vertically. Z-axis: perpendicular to the plane of the calibration plate, pointing directly upwards.

[0114] Indicatively, in the calibration plate coordinate system (with the calibration plate plane as the (Z=0) plane), the extracted three-dimensional coordinates of the corner points are as follows:

[0115] {(X ij ,Y ij ,0)}

[0116] Among them, (X) ij ,Y ij (,0) represents the three-dimensional coordinates of the corner point in the i-th row and j-th column of the calibration plate coordinate system. These coordinates are uniformly distributed and spaced at unit intervals, and can be written as:

[0117] X ij = (i-1)·Δx,Y ij = (j-1)·Δy,Z ij =0

[0118] Where Δx and Δy are the known physical dimensions of the checkerboard cells, i.e., the side length of each black and white square. These dimensions are known and used to convert the coordinates on the calibration board into actual physical dimensions.

[0119] At the same time, it is necessary to measure (or know in advance) the three-dimensional coordinates of the camera optical center O in the calibration plate coordinate system:

[0120] O=(O X O Y O Z )

[0121] Among them, (O) X O Y O Z The coordinates of the camera's optical center in the calibration plate coordinate system represent the true ray vector in the camera coordinate system and are crucial for camera calibration and 3D reconstruction.

[0122] Step 302: Obtain the ray vectors corresponding to multiple feature points in the camera coordinate system.

[0123] The computing device acquires the ray vectors corresponding to multiple feature points in the camera coordinate system, which may include the following steps: establishing a camera coordinate system, which is a three-dimensional coordinate system with the camera optical center as the origin; for each feature point, determining the three-dimensional coordinates of the feature point in the camera coordinate system based on the feature point and the three-dimensional coordinates of the camera optical center in the calibration board coordinate system; determining the corresponding ray vector based on the three-dimensional coordinates of each feature point in the camera coordinate system. Specifically, the three-dimensional coordinates of the feature point in the camera coordinate system can be the difference between the three-dimensional coordinates of the feature point in the calibration board coordinate system and the three-dimensional coordinates of the camera optical center in the calibration board coordinate system. The ray vector corresponding to the feature point is the difference between the three-dimensional coordinates of the feature point in the camera coordinate system and the three-dimensional coordinates of the camera optical center in the camera coordinate system.

[0124] A camera coordinate system is a three-dimensional Cartesian coordinate system with the camera's optical center as its origin. It is used to describe the position and orientation of a point in space relative to the camera. The camera coordinate system is a three-dimensional Cartesian coordinate system with its origin located at the camera's optical center (i.e., the camera's optical center). Typically, the three axes of the camera coordinate system are defined as follows: X-axis: usually points to the right of the camera. Y-axis: usually points to the bottom of the camera. Z-axis: points in the direction of the camera's optical axis, i.e., directly in front of the camera.

[0125] Schematic diagram: A camera coordinate system is established with the camera's optical center O as the origin and the optical axis aligned with the Z-axis. Let the two-dimensional pixel coordinates of a certain corner point be... for:

[0126]

[0127] The three-dimensional coordinates P of the corner point in the camera coordinate system ij It can be represented as:

[0128]

[0129] Since the checkerboard plane is assumed to be Z=0, the 3D coordinates of the corner point in the camera coordinate system are the 3D coordinates of the corner point in the calibration board coordinate system minus the 3D coordinates of the camera optical center O in the calibration board coordinate system (if rotation exists, the camera's extrinsic parameters need to be considered separately). The resulting ray vector r... ij It can be represented as:

[0130] r ij =P ij -O cam Among them, O cam Let r be the three-dimensional coordinates of the camera's optical center in the camera coordinate system. Considering rotation, the ray vector r... ij This can be further expressed as:

[0131] r ij =(P ij -O cam )·R

[0132] Where R is the rotation matrix, which is part of the camera's extrinsic parameters and is used to describe the camera's rotation.

[0133] Step 303: Construct a NURBS surface model based on the two-dimensional pixel coordinates of multiple feature points and the corresponding ray vectors.

[0134] In image processing, the two-dimensional pixel coordinates of feature points can usually only be observed directly from the image. To establish the mapping relationship between two-dimensional pixel coordinates and ray vectors, this embodiment introduces a NURBS surface model, which interpolates or approximates a given data point through a series of control points and weights, thereby constructing the NURBS surface model.

[0135] In some embodiments, constructing a NURBS surface model based on the two-dimensional pixel coordinates of multiple feature points and their corresponding ray vectors may include the following steps: determining the two-dimensional pixel coordinates of each feature point as the parameter coordinates of the NURBS surface model, where the parameter coordinates represent the position of the feature point in the parameter space of the NURBS surface model; determining the ray vector corresponding to each feature point as the coordinates of a control point of the NURBS surface model, and assigning a corresponding weight to each control point, where the control point represents the shape of the NURBS surface model, and the weight represents the degree of influence of the control point on the NURBS surface; and constructing the NURBS surface model based on the parameter coordinates of each feature point, the coordinates of the control points, and the weights.

[0136] Parametric coordinates are a concept in NURBS surface models used to represent the positions of feature points in parameter space. In a NURBS surface model, the position of each point is determined by two parameters, u and v, which define the point's location on the surface. For each feature point, its two-dimensional pixel coordinates can be mapped to the parametric coordinates of the NURBS surface model.

[0137] Control points are another key concept in NURBS surface models, used to define the shape of the surface. Control points are points in three-dimensional space that influence the positions of points on the surface through specific mathematical formulas. When constructing a NURBS surface model, the ray vector corresponding to each feature point can be used as the coordinates of the control points.

[0138] Weights are parameters in a NURBS surface model used to adjust the influence of control points on the surface. Each control point has a corresponding weight; the larger the weight, the stronger the influence of the control point on the surface. Weights are usually positive numbers and can be used to control the smoothness of the surface and the accuracy of feature points. Illustratively, weights can be assigned by the confidence level of a corner detection algorithm. Corner detection is a common image processing technique, and corner detection algorithms typically output confidence values ​​for feature points. The confidence value indicates how "confident" the algorithm is in detecting that feature point, and is usually a value between 0 and 1. This means that if the corner detection algorithm has a high confidence level (close to 1) for a feature point, then the weight of the control point corresponding to that feature point will be relatively large. This indicates that the control point has a significant impact on the shape of the NURBS surface because it is considered a reliable feature point. If the corner detection algorithm has a low confidence level (close to 0) for a feature point, then the weight of the control point corresponding to that feature point will be relatively small. This indicates that the control point has a smaller impact on the shape of the NURBS surface because it is considered a less reliable feature point.

[0139] Schematic, the 2D pixel coordinates of all corner points are considered as scatter points on the parameter plane (u,v) of the NURBS surface, and the ray vectors in the camera coordinate system are considered as 3D output values. These 3D output values ​​serve as the coordinates of the control points (or fitting targets) of the NURBS surface model. The goal is to construct a NURBS surface model that can accurately interpolate or approximate the corresponding ray vectors at given parameter coordinates. Parameter coordinates u ij Defined as:

[0140]

[0141] in, Let R be the two-dimensional pixel coordinates of the corner point in the i-th row and j-th column in the imaging plane coordinate system. The coordinates of the control points are the three-dimensional output value R. ij Defined as:

[0142] R ij =r ij =(r x,ij ,r y,ij ,r z,ij )

[0143] Where, r ij Let be the ray vector of the corner point in the i-th row and j-th column in the camera coordinate system.

[0144] Therefore, each feature point is assigned the coordinates R of a corresponding control point. ij and weight w ij This will give you a set of data for constructing a NURBS surface model (r x,ij ,ry,ij ,r z,ij ,w ij This data will be used to construct a NURBS surface model, thereby achieving a precise mapping from two-dimensional pixel coordinates to ray vectors.

[0145] Schematic, a typical bidirectional NURBS surface model S(u,v) is defined as follows:

[0146]

[0147] Where, N i,p (u) and M j,q (v) represents the B-spline basis functions in the u and v directions, respectively. The orders p and q of the basis functions are typically 3 or 4. B-spline basis functions are piecewise polynomial functions used to define NURBS surface models, mapping the geometric information of control points to points in the parameter space. The order of the basis functions is a crucial parameter, determining their polynomial degree and influencing the smoothness and shape control capabilities of the NURBS surface model. ij Represents the coordinates R of the control point ij The corresponding weights. Weight w ij Used to adjust the degree of influence of control points on the shape of NURBS surface models. R ij ∈R 3 Let r be the coordinates of the control point and the corresponding ray vector r in the camera coordinate system. ij R ij The basic shape of the NURBS surface model is defined. (u,v) are parametric coordinates, which typically vary within the interval [0,1]×[0,1]. i and j are variables used to index the control points, the direction n represents the number of control points in the u direction, and m represents the number of control points in the v direction.

[0148] In some embodiments, when constructing a NURBS surface model, an interpolation-based construction method or other suitable methods can be used. The interpolation-based construction method represents adjusting the coordinates and weights of control points so that the NURBS surface model passes through all given data points, including the two-dimensional pixel coordinates of feature points and their corresponding ray vectors. This means that by appropriately selecting the coordinates and weights of the control points, the NURBS surface model can guarantee rigid interpolation at feature points, thereby achieving high-precision surface reconstruction.

[0149] Step 304: Based on other images captured by the camera, determine the corrected ray vector corresponding to each pixel in the other images.

[0150] In some embodiments, the computing device can extract the two-dimensional pixel coordinates of each pixel in other images captured by the camera; for each pixel's two-dimensional pixel coordinates, the corresponding corrected ray vector is determined by the constructed NURBS surface model.

[0151] In other words, after completing the above calibration process, for any new pixel p pix =(x pix ,y pix The corrected ray vector in the camera coordinate system can be obtained through the following steps:

[0152] 1. Determine the corresponding parameter coordinates:

[0153] Based on the given p pix Determine the corresponding parametric coordinates (u*, v*) in the surface parameter domain:

[0154] (u*,v*)=argmin (u,v) ||S(u,v)-p pix ||2

[0155] If, during the construction of the NURBS surface model, an explicit or approximate inverse mapping f from two-dimensional pixel coordinates to parametric coordinates has been established... -1 Then it can be solved directly:

[0156] (u*,v*)=f -1 (x pix ,y pix )

[0157] After determining the parameter coordinates (u*, v*), substitute them into the constructed NURBS surface model:

[0158]

[0159] This yields the corrected ray vector r(u,v), which is the vector for pixel p. pix The true ray vector in the camera coordinate system after eliminating the effects of lens distortion.

[0160] Step 305: Synthesize the distortion-corrected image based on the corrected ray vector corresponding to each pixel.

[0161] In some embodiments, the corrected ray vector corresponding to each pixel is projected onto an ideal imaging plane to obtain the corresponding projection coordinates. Based on the camera parameters, the projection coordinates corresponding to each pixel are converted into corrected two-dimensional pixel coordinates; based on the corrected two-dimensional pixel coordinates of each pixel, a distortion-corrected image is synthesized.

[0162] In an idealized optical system, the ideal imaging plane is an imaginary plane located at the camera's focal length. All light rays parallel to the camera's optical axis converge on this plane after passing through the lens system to form a sharp image. This plane is the ideal position in the camera design for receiving images, assuming that all light rays can be perfectly focused there without any distortion or defocus. In other words, the ideal imaging plane describes the plane where light rays should converge without distortion. Projected coordinates refer to the intersection point of the corrected ray vector and the ideal imaging plane.

[0163] Camera parameters refer to a series of parameters used to describe the internal characteristics of a camera and its position and orientation relative to the world coordinate system. Camera parameters can include intrinsic parameters, principal point coordinates, etc. The intrinsic parameters matrix is ​​a matrix describing the internal optical characteristics of the camera. Principal point coordinates refer to the position of the optical center on the camera's imaging plane, which is the center point where light rays converge on the image sensor. In an ideal situation without distortion, the principal point is the geometric center of the image.

[0164] Schematic illustration: After obtaining the corrected ray vector r(u,v), it can be further projected back onto the ideal imaging plane to obtain the distortion-corrected image. The ideal imaging plane is constructed at Z = f (focal length f) in the camera coordinate system. Then, for any corrected ray vector r(u,v) = (r... x ,r y ,r z Its projected coordinates p′ on the ideal imaging plane (x,y) for:

[0165]

[0166] Based on the camera parameters (such as intrinsic parameter matrix, principal point coordinates, etc.), the projected coordinates p′ (x,y) The image is then converted to corrected 2D pixel coordinates, resulting in the 2D pixel coordinates of the distortion-corrected image. Since these corrected 2D pixel coordinates may be non-integer, image interpolation methods are needed to synthesize the distortion-corrected image. For example, bicubic convolution interpolation, a commonly used image interpolation technique, can effectively handle non-integer pixel coordinates, thereby generating a smooth and high-quality image. Through these steps, the corrected image is obtained, where the position of each pixel is adjusted according to the corrected ray vector, thus eliminating the effects of lens distortion.

[0167] The advantage of the method provided in this disclosure is that, compared to related technologies, it can effectively correct random errors caused by non-professional cameras that do not perfectly conform to ideal mathematical models. To verify the superiority of this disclosure over related technologies, three experimental scenarios were designed, and these scenarios were precisely scanned using a laser scanner with an accuracy of ±2 mm at a distance of 10 meters.

[0168] Please refer to Figure 4 The diagram illustrates a schematic of a calibration experimental apparatus provided in an exemplary embodiment of this disclosure. This apparatus is used for camera distortion calibration. The diagram comprises two parts: 1. Left image: Shows a laboratory or workshop environment with multiple workbenches and chairs. Pipes and lights are visible on the ceiling, giving the environment a somewhat industrial feel. This scene is used to capture calibration images to obtain feature points on a calibration board. 2. Right image: Shows a classroom or conference room environment with multiple long tables and chairs. A tripod is positioned above the tables, and a camera is mounted on the tripod. The camera in this scene is used to photograph the calibration board and obtain calibration images. These images demonstrate the experimental apparatus and environment used for camera distortion calibration, obtaining calibration images by photographing the calibration board.

[0169] Please refer to Figure 5 The diagram illustrates a scanning scene provided by an exemplary embodiment of this disclosure. Four images are shown, each depicting a different scanning scene: 1. Top left: An office scene with a prominent white light source on the screen, which may be used for calibration. The office contains multiple workbenches and chairs. 2. Top right: A warehouse scene with multiple option buttons on the screen for controlling the scanning process or selecting different scanning modes. The warehouse contains shelves and some packaging materials. 3. Bottom left: A corridor scene with multiple option buttons on the screen for controlling the scanning process or selecting different scanning modes. The corridor contains multiple doors and windows. 4. Bottom right: A laboratory scene with multiple option buttons on the screen, possibly for controlling the scanning process or selecting different scanning modes. The laboratory contains multiple workbenches and equipment. The point cloud data output by the laser scanner is considered an accurate representation of the real-world scene. Please refer to... Figure 6The diagram illustrates a real-world scene point cloud representation provided by an exemplary embodiment of this disclosure. Four point cloud maps are shown, each depicting a real-world scene from a different perspective: 1. Top left: A point cloud map showing the layout of an office, including workbenches, chairs, and some equipment. 2. Top right: A point cloud map showing the layout of a warehouse, including shelves and packaging materials. 3. Bottom left: A point cloud map showing the layout of a corridor, including doors, windows, and some decorations. 4. Bottom right: A point cloud map showing the layout of a laboratory, including lab benches and some equipment.

[0170] Please refer to Figure 7 The diagram illustrates a reference point selection method provided in an exemplary embodiment of this disclosure. Two different scenarios are shown: 1. Left image: An office scene with several prominent feature points used as reference points during camera correction. 2. Right image: A classroom or conference room scene with a tripod and a camera mounted on it. Numbers (1 to 18) are marked on the table in front of the camera to identify specific feature points for use in subsequent correction.

[0171] In these scenarios, cameras were installed and original photos were acquired. Then, distortion correction was performed on the original photos using embodiments of this disclosure and related technologies, resulting in two sets of corrected photos (hereinafter referred to as Photo A and Photo B). A series of reference points with distinct features were selected in the original photos, and the specific locations and coordinates of these reference points were determined in the point cloud data of the real scene. Simultaneously, the camera coordinates were located in the point cloud data, and a reference vector from the image sensor to the reference points was constructed. A planar coordinate system was established on the corrected photos, and the pixel coordinates corresponding to the reference points in Photo A and Photo B were determined, and two-dimensional vectors from these pixel coordinates to the image center were calculated. Subsequently, these two-dimensional vectors were converted into direction vectors in three-dimensional space, resulting in direction vector A and direction vector B. By calculating the angles between these direction vectors and the reference vectors, the error between the corrected photo scene and the real scene can be evaluated.

[0172] The accuracy of the two correction methods is quantified by comparing the angles between direction vector A and the reference vector (hereinafter referred to as angle A) and the angle between direction vector B and the reference vector (hereinafter referred to as angle B). A data comparison diagram of the two correction methods is shown below. Figure 8 As shown, Figure 8The images presented here show four different 3D bar charts, each comparing the results of correction using two different methods in different real-world scenarios. Each bar chart displays two different colored bars: yellow represents the optimized value after correction using this method, and blue represents the original value after correction using the traditional method. As can be seen from the figures, angle A is significantly smaller than angle B in most cases, indicating that the scene calibrated using the embodiments of this disclosure is closer to the real-world scene than the scene calibrated using related technologies. Figure 9 , Figure 10 and Figure 11 The diagram illustrates offset distance maps and pixel offset direction maps of different device lenses provided in the exemplary embodiments of this disclosure before and after calibration. By visually rendering the offset direction and offset distance maps before and after calibration, the impact of lens errors on imaging can be intuitively shown. Specifically, the pixel offset distance map before and after calibration maps distances to arrow colors, with red corresponding to offsets within 1 pixel, and blue indicating larger offset distances as the distance approaches the target pixel. The offset direction map maps the angle between the offset direction and true north to arrow colors.

[0173] Figure 9 This disclosure illustrates the offset distance and pixel offset direction maps of a mobile phone lens (Image A) before and after calibration, according to an exemplary embodiment. The left image (offset distance map) shows the pixel offset distance before calibration, with colors varying from red (offset within 1 pixel) to blue (larger offset distances), indicating the degree and distribution of lens distortion. The right image (pixel offset direction map) shows the pixel offset direction before calibration; the color of the arrows indicates the angle between the offset direction and true north, helping to understand the directionality of distortion.

[0174] Figure 10 This illustration shows the offset distance map and pixel offset direction map of a mobile phone lens B provided in an exemplary embodiment of this disclosure before and after calibration. The left image (offset distance map) also shows the pixel offset distance before calibration, color coding, and... Figure 9 The same image shows the distortion of the B phone's lens. The right image (pixel offset direction map) shows the pixel offset direction before calibration, providing information about the distortion direction of the B phone's lens.

[0175] Figure 11 This illustration shows the offset distance map and pixel offset direction map of a C-type camera lens provided in an exemplary embodiment of this disclosure before and after calibration. The left image (offset distance map) shows the pixel offset distance before calibration, with color coding similar to the previous two images. Figure 1 The first image shows the degree of distortion of the C-type camera lens. The second image (pixel offset direction map) shows the pixel offset direction before calibration, providing information about the distortion direction of the C-type camera lens.

[0176] These figures demonstrate that, due to limitations in manufacturing and assembly processes, mobile phone and camera lenses may experience significant stress, leading to lens deformation and concentric circular distortion error fluctuations. This distortion cannot be described using traditional radial or tangential distortion models and therefore cannot be eliminated by conventional methods. The embodiments of this disclosure, by employing a NURBS surface model, effectively correct this complex distortion, thereby significantly improving image quality. These visualized offset direction and offset distance graphs provide a clear view of the changes in lens distortion before and after calibration, thus evaluating the effectiveness of the correction method.

[0177] In summary, optical systems often exhibit various distortions due to lens manufacturing errors, assembly deviations, and material inhomogeneities. Pinhole models and similar models can only correct regular distortions, not irregular distortions caused by lens manufacturing errors. NURBS surface models, due to their flexibility and accuracy, offer significant advantages in defining and rendering complex curves and surfaces. The node vector values ​​and spacing of NURBS surface models can be arbitrary, enabling precise definition and rendering of complex curves and surfaces. NURBS surface models are defined using control points and weights, allowing for precise interpolation at these control points. Adjusting the weights further fine-tunes the influence of control points on the surface shape, achieving high-precision modeling. In this embodiment, establishing a mapping relationship between two-dimensional pixel coordinates and ray vectors by constructing a NURBS surface model yields the following beneficial effects: Firstly, it adapts to complex distortions: Because NURBS surface models can accurately adapt to complex geometries, they can simulate and correct irregular distortions not caused by cameras. On the other hand, precise mapping: By using two-dimensional pixel coordinates as parametric coordinates on the NURBS surface model and ray vectors as coordinates of control points on the NURBS surface model, precise mapping from the image plane to three-dimensional space can be achieved. Furthermore, distortion correction: The NURBS surface model can accurately describe and correct distortions in an image because it can adjust control points and weights according to the actual distortion, thereby generating a corrected image.

[0178] This embodiment uses a set of control points and weights to interpolate or approximate given data points, allowing the NURBS surface model to accurately interpolate or approximate ray vectors at feature point locations. By selecting an interpolation-type construction method (or other methods), the NURBS surface model can completely pass through the given ray vector data points. The corrected ray is calculated using the NURBS surface interpolation model values, obtaining the true ray vector of that pixel in the camera coordinate system after eliminating the effects of lens distortion. Finally, it is projected back onto the ideal imaging plane to obtain the corrected image, achieving irregular distortion correction. This method not only corrects irregular distortion caused by lens manufacturing errors but also improves image quality, providing more accurate and reliable visual information for optical system applications.

[0179] The following are device embodiments of the present disclosure. For parts not described in detail in the device embodiments, please refer to the technical details disclosed in the above method embodiments.

[0180] This disclosure provides a NURBS-based camera distortion calibration device. This device can be implemented as all or part of a computing device through software, hardware, or a combination of both. The device includes an acquisition module, a construction module, and a calibration module.

[0181] The acquisition module is used to acquire calibration images, which are images of the calibration board obtained by taking pictures with a camera.

[0182] The building module is used to construct a NURBS surface model based on the calibration image. The NURBS surface model is used to represent the mapping relationship between the two-dimensional pixel coordinates of multiple feature points on the calibration board and the ray vector. The ray vector is used to represent the direction of the ray from the camera optical center to the feature point.

[0183] The correction module is used to correct distortion in other images captured by the camera based on the constructed NURBS surface model.

[0184] In one possible implementation, the building module is also used for:

[0185] Based on the calibration image, extract the two-dimensional pixel coordinates of multiple feature points on the calibration board in the imaging plane coordinate system. The imaging plane coordinate system is a two-dimensional coordinate system defined based on the camera's imaging plane.

[0186] Obtain the ray vectors corresponding to multiple feature points in the camera coordinate system, which is a three-dimensional coordinate system with the camera optical center as the origin;

[0187] A NURBS surface model is constructed based on the two-dimensional pixel coordinates of multiple feature points and the corresponding ray vectors.

[0188] In another possible implementation, the building module is also used for:

[0189] Obtain the three-dimensional coordinates of multiple feature points in the calibration plate coordinate system, which is a three-dimensional coordinate system defined based on the plane of the calibration plate;

[0190] For each feature point, the three-dimensional coordinates of the feature point in the camera coordinate system are determined based on the three-dimensional coordinates of the feature point and the camera optical center in the calibration plate coordinate system.

[0191] The corresponding ray vector is determined based on the three-dimensional coordinates of each feature point in the camera coordinate system.

[0192] In another possible implementation, the building module is also used for:

[0193] The two-dimensional pixel coordinates of each feature point are determined as the parametric coordinates of the NURBS surface model. The parametric coordinates are used to represent the position of the feature point in the parameter space of the NURBS surface model.

[0194] The ray vector corresponding to each feature point is determined as the coordinates of the control point of the NURBS surface model, and a corresponding weight is set for each control point. The control point is used to represent the shape of the NURBS surface model, and the weight is used to represent the degree of influence of the control point on the NURBS surface.

[0195] A NURBS surface model is constructed based on the parameter coordinates of each feature point, the coordinates of the control points, and their weights.

[0196] In another possible implementation, the correction module is also used for:

[0197] Extract the two-dimensional pixel coordinates of each pixel from other images captured by the camera;

[0198] For the two-dimensional pixel coordinates of each pixel, the corresponding corrected ray vector is determined by the constructed NURBS surface model;

[0199] The distortion-corrected image is synthesized based on the corrected ray vector corresponding to each pixel.

[0200] In another possible implementation, the correction module is also used for:

[0201] The corrected ray vector corresponding to each pixel is projected onto the ideal imaging plane to obtain the corresponding projection coordinates;

[0202] Based on the camera parameters, the projected coordinates of each pixel are converted into corrected two-dimensional pixel coordinates.

[0203] The distortion-corrected image is synthesized based on the corrected two-dimensional pixel coordinates of each pixel.

[0204] It should be noted that the above embodiments only illustrate the division of the above functional modules when implementing the device. In actual applications, the above functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0205] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0206] This disclosure also provides a NURBS-based camera distortion calibration device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0207] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0208] This disclosure also provides a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program implements the steps of the above method when executed by a processor.

[0209] Figure 12 This is a block diagram illustrating an apparatus 1900 according to an exemplary embodiment. For example, apparatus 1900 may be provided as a computing device. (Refer to...) Figure 12 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0210] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM macOS X TM Unix TM Linux TM FreeBSD TMOr similar.

[0211] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of the device 1900 to perform the above-described method.

[0212] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0213] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0214] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information of computer-readable program instructions. These electronic circuits can execute computer-readable program instructions to implement various aspects of this disclosure.

[0215] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0216] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0217] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0218] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0219] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A camera distortion calibration method based on non-uniform rational B-spline NURBS, characterized in that, The method includes: Acquire a calibration image, which is an image of the calibration board obtained by taking a picture with a camera; Based on the calibration image, a NURBS surface model is constructed. The NURBS surface model is used to represent the mapping relationship between the two-dimensional pixel coordinates of multiple feature points on the calibration board and the ray vector. The ray vector is used to represent the direction of light rays from the optical center of the camera to the feature point. Based on the constructed NURBS surface model, distortion correction is performed on other images captured by the camera.

2. The method according to claim 1, characterized in that, The step of constructing a NURBS surface model based on the calibration image includes: Based on the calibration image, extract the two-dimensional pixel coordinates of multiple feature points on the calibration board in the imaging plane coordinate system, wherein the imaging plane coordinate system is a two-dimensional coordinate system defined based on the imaging plane of the camera; Obtain the ray vectors corresponding to the multiple feature points in the camera coordinate system, wherein the camera coordinate system is a three-dimensional coordinate system with the camera optical center as the origin; The NURBS surface model is constructed based on the two-dimensional pixel coordinates of multiple feature points and the corresponding ray vectors.

3. The method according to claim 2, characterized in that, The step of obtaining the ray vectors corresponding to the multiple feature points in the camera coordinate system includes: Obtain the three-dimensional coordinates of multiple feature points in the calibration plate coordinate system, wherein the calibration coordinate system is a three-dimensional coordinate system defined based on the plane of the calibration plate; For each feature point, the three-dimensional coordinates of the feature point in the camera coordinate system are determined based on the three-dimensional coordinates of the feature point and the camera optical center in the calibration plate coordinate system. The corresponding ray vector is determined based on the three-dimensional coordinates of each feature point in the camera coordinate system.

4. The method according to claim 2, characterized in that, The step of constructing the NURBS surface model based on the two-dimensional pixel coordinates of multiple feature points and the corresponding ray vectors includes: The two-dimensional pixel coordinates of each feature point are determined as the parameter coordinates of the NURBS surface model, and the parameter coordinates are used to represent the position of the feature point in the parameter space of the NURBS surface model; The ray vector corresponding to each feature point is determined as the coordinates of the control point of the NURBS surface model, and a corresponding weight is set for each control point. The control point is used to represent the shape of the NURBS surface model, and the weight is used to represent the degree of influence of the control point on the NURBS surface. The NURBS surface model is constructed based on the parameter coordinates corresponding to each feature point, the coordinates of the control point, and the weight.

5. The method according to any one of claims 1 to 4, characterized in that, The distortion correction of other images captured by the camera based on the constructed NURBS surface model includes: Based on other images captured by the camera, extract the two-dimensional pixel coordinates of each pixel in the other images; For the two-dimensional pixel coordinates of each pixel, the corresponding corrected ray vector is determined by the constructed NURBS surface model; The distortion-corrected image is synthesized based on the corrected ray vector corresponding to each pixel.

6. The method according to claim 5, characterized in that, The step of synthesizing the distortion-corrected image based on the corrected ray vector corresponding to each pixel includes: The corrected ray vector corresponding to each pixel is projected onto the ideal imaging plane to obtain the corresponding projection coordinates; Based on the camera parameters of the camera, the projected coordinates corresponding to each pixel are converted into corrected two-dimensional pixel coordinates; The distortion-corrected image is synthesized based on the corrected two-dimensional pixel coordinates of each pixel.

7. A camera distortion calibration device based on NURBS, characterized in that, The device includes: The acquisition module is used to acquire calibration images, which are images of the calibration board obtained by taking pictures with a camera. The construction module is used to construct a NURBS surface model based on the calibration image. The NURBS surface model is used to represent the mapping relationship between the two-dimensional pixel coordinates of multiple feature points on the calibration board and the ray vector. The ray vector is used to represent the direction of light rays from the optical center of the camera to the feature point. The correction module is used to perform distortion correction on other images captured by the camera based on the constructed NURBS surface model.

8. A NURBS-based camera distortion calibration device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, or a non-volatile computer-readable storage medium carrying a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images