High-precision camera calibration method and system based on multi-modal feature fusion, and medium

Abstract

The application provides a high-precision camera calibration method and system based on multi-modal feature fusion and a medium. The method comprises the following steps: selecting a calibration board with known three-dimensional coordinates, placing the calibration board at different positions and angles of a camera, shooting multiple groups of images, and pre-processing the multiple groups of shot images to obtain image data and point cloud data; extracting two-dimensional coordinates of circular mark points of the calibration board and phase information of each pixel point based on the image data, and obtaining three-dimensional coordinates of the circular mark points based on the point cloud data; generating geometric features based on the two-dimensional coordinates of the circular mark points, generating phase features based on the phase information, constructing a multi-parameter projection model, projecting the three-dimensional coordinates based on the multi-parameter projection model, and synchronously adjusting internal and external parameters of the camera by using a bundle adjustment optimization algorithm; and obtaining the geometric features and the phase features in the images through the calibration board, so that the projection distortion information is accurately analyzed, and the camera calibration precision is improved.

Description

Technical Field

[0001] This application relates to the field of camera calibration technology, and more specifically, to a high-precision camera calibration method, system, and medium based on multimodal feature fusion. Background Technology

[0002] In the fields of computer vision and optical measurement, camera calibration is a fundamental step in applications such as 3D reconstruction and vision-guided processing. Traditional calibration methods (such as the Zhang Zhengyou calibration method) mainly rely on checkerboard patterns or circular markers for geometric feature extraction, but they suffer from the following key problems in practical industrial applications:

[0003] Traditional methods rely on corner or center point detection, which are prone to failure under low light, strong reflection, or motion blur conditions. For example, surface reflection or complex curvature of metal parts can cause deviations in marker point detection, affecting calibration accuracy.

[0004] Mainstream calibration methods (such as the Brown-Conrady model) only consider radial and tangential distortions, ignoring higher-order nonlinear distortions (such as eccentric distortion and projection nonlinearity). Under large field of view (>300mm) or wide-angle lenses, existing models cannot accurately correct edge distortions.

[0005] When calibrating a multi-camera system, traditional methods optimize the parameters of each camera independently, failing to fully utilize the geometric constraints between cameras (such as epipolar constraints), leading to the accumulation of stitching errors. In multi-view measurement of large-sized parts, the accumulated error may exceed 0.05mm, which cannot meet the requirements of high-precision machining.

[0006] Industrial environments are subject to interference such as vibration and temperature changes, which can cause traditional calibration results to drift and require frequent recalibration. Summary of the Invention

[0007] The purpose of this application is to provide a high-precision camera calibration method, system, and medium based on multimodal feature fusion. By acquiring geometric and phase features in the image through a calibration plate, the projection distortion information can be accurately analyzed, thereby improving the camera calibration accuracy.

[0008] This application also provides a high-precision camera calibration method based on multimodal feature fusion, including:

[0009] Select a calibration board with known three-dimensional coordinates. The calibration board is a multi-modal calibration board that includes a circular marker array and a sinusoidal fringe phase code.

[0010] The calibration board was placed at different positions and angles of the camera to capture multiple sets of images. The captured images were then preprocessed to obtain image data and point cloud data.

[0011] The two-dimensional coordinates of the circular markers on the calibration board and the phase information of each pixel are extracted from the image data. The phase information of the pixels is converted into position coordinates, and the three-dimensional coordinates of the circular markers are obtained based on the point cloud data.

[0012] Geometric features are generated based on the two-dimensional coordinates of circular marker points, phase features are generated based on phase information, constraints are established based on geometric features and phase features, a multi-parameter projection model is constructed, the three-dimensional coordinates are projected based on the multi-parameter projection model, and the projection distortion information is analyzed by comparing with the two-dimensional coordinates.

[0013] Based on projection distortion information, a bundle adjustment optimization algorithm is used to synchronously adjust the camera's intrinsic and extrinsic parameters.

[0014] Optionally, in the high-precision camera calibration method based on multimodal feature fusion described in the embodiments of this application, the calibration plate is placed at different positions and angles of the camera, multiple sets of images are captured, and the captured multiple sets of images are preprocessed to obtain image data and point cloud data, specifically including:

[0015] Obtain the initial position of the camera, and set the placement position of the calibration plate based on the initial position of the camera;

[0016] Based on the initial position of the camera, the horizontal and vertical adjustment distances are set to obtain several adjustment positions;

[0017] Multiple sets of images were obtained by taking pictures of the calibration board at several adjustment positions.

[0018] Multiple sets of captured images are converted to grayscale to obtain multiple sets of grayscale images;

[0019] Noise removal is performed on multiple grayscale images based on median filtering and Gaussian filtering to remove salt-and-pepper noise and Gaussian noise from the grayscale images.

[0020] Enhancement processing is performed on the noise-removed grayscale image, and image data and point cloud data are obtained.

[0021] Optionally, in the high-precision camera calibration method based on multimodal feature fusion described in this application embodiment, the two-dimensional coordinates of the circular marker points on the calibration board and the phase information of each pixel are extracted based on the image data, and the phase information of the pixels is converted into position coordinates, specifically including:

[0022] Edge extraction is performed on the preprocessed image based on the Canny edge detection algorithm to obtain the edge information of the circular marker points;

[0023] The position of the center point of the circular marker is calculated based on the edge information of the circular marker, and the two-dimensional coordinates of the center point are obtained.

[0024] Phase information of pixels is obtained by performing phase calculation on the captured image based on a combination of phase shifting method and multi-frequency heterodyne method.

[0025] The mapping relationship between phase information and actual spatial position is analyzed based on the actual spatial position of the circular marker.

[0026] The pixel information is transformed based on the mapping relationship to obtain the position coordinates of the pixel.

[0027] Optionally, in the high-precision camera calibration method based on multimodal feature fusion described in the embodiments of this application, the multi-parameter projection model includes radial distortion terms, tangential distortion terms, and eccentric distortion terms;

[0028] The formula for the multi-parameter projection model is as follows:

[0029]

[0030] The formula for the radial distortion term is as follows:

[0031] Δx radial =x'(k1r 2 +k2r 4 +k3r 6 )

[0032] Δy radial =y'(k1r 2 +k2r 4 +k3r 6 )

[0033] The formula for tangential distortion is as follows:

[0034] Δx tangent =2p1x'y'+p2(r 2 +2x ,2 )

[0035] Δy tangent =p1(r 2 +2y' 2 )+2p2x'y'

[0036] The formula for the eccentric distortion term is as follows:

[0037] Δx decentering =s1(r 2 +2x' 2 )+2s2x'y'

[0038] Δy decentering =2s1x'y'+s2(r 2 +2y ,2 )

[0039] In the formula, x and y represent the coordinates of the image pixels; x′ and y′ represent the normalized image coordinates after distortion correction; Δx radial with Δy radial The radial distortion correction amount is represented by k1, k2, and k3, which represent the radial distortion coefficients, and r represents the image plane radius. Δx tangent With Δx tangent p1 and p2 represent the tangential distortion correction amount, and p1 and p2 represent the tangential distortion coefficients, Δx decenterign with Δy decentering s1 and s2 represent the eccentricity correction amount and the eccentricity coefficients, respectively.

[0040] Optionally, in the high-precision camera calibration method based on multimodal feature fusion described in the embodiments of this application, the projection distortion information analysis method includes:

[0041] Initialize the parameters of the multi-parameter projection model. The parameters of the multi-parameter projection model include the camera's intrinsic parameters, extrinsic parameters, and distortion parameters. The camera's intrinsic parameters include focal length and principal point coordinates. The camera's extrinsic parameters include rotation matrix and translation vector. The distortion parameters include radial distortion, tangential distortion, and eccentric distortion.

[0042] The theoretical projection position of the circular marker in the image is calculated based on the parameters of the initialized multi-parameter projection model.

[0043] Obtain the two-dimensional coordinates of the actually detected circular marker points, and calculate the error between the theoretical projection position and the actual detected two-dimensional coordinates;

[0044] Radial distortion, tangential distortion, and eccentric distortion information are calculated based on the error between the theoretical projection position and the actual detected two-dimensional coordinates.

[0045] Projection distortion information is generated based on radial distortion information, tangential distortion information, and eccentric distortion information.

[0046] Optionally, in the high-precision camera calibration method based on multimodal feature fusion described in the embodiments of this application, the camera's intrinsic and extrinsic parameters are simultaneously adjusted using a bundle adjustment optimization algorithm based on projection distortion information, specifically including:

[0047] The error between the theoretical projection position and the actual detected two-dimensional coordinates is obtained based on projection distortion information;

[0048] Calculate the sum of squared errors for all circular marker points, and use the sum of squared errors for all circular marker points as the target error function;

[0049] The gradient information of the target error function is analyzed based on the bundle adjustment optimization algorithm, and the target error value is analyzed based on the gradient information.

[0050] Compare the target error value with the set error threshold;

[0051] If the target error value is less than the set error threshold, the calibrated camera's internal and external parameters are obtained.

[0052] If the target error value is greater than or equal to the set error threshold, the number of iterations is adjusted, and the target error function is adjusted accordingly.

[0053] Secondly, embodiments of this application provide a high-precision camera calibration system based on multimodal feature fusion. The system includes a memory and a processor. The memory includes a program for a high-precision camera calibration method based on multimodal feature fusion. When the program for the high-precision camera calibration method based on multimodal feature fusion is executed by the processor, it performs the following steps:

[0054] Select a calibration board with known three-dimensional coordinates. The calibration board is a multi-modal calibration board that includes a circular marker array and a sinusoidal fringe phase code.

[0055] The calibration board was placed at different positions and angles of the camera to capture multiple sets of images. The captured images were then preprocessed to obtain image data and point cloud data.

[0056] The two-dimensional coordinates of the circular markers on the calibration board and the phase information of each pixel are extracted from the image data. The phase information of the pixels is converted into position coordinates, and the three-dimensional coordinates of the circular markers are obtained based on the point cloud data.

[0057] Geometric features are generated based on the two-dimensional coordinates of circular marker points, phase features are generated based on phase information, constraints are established based on geometric features and phase features, a multi-parameter projection model is constructed, the three-dimensional coordinates are projected based on the multi-parameter projection model, and the projection distortion information is analyzed by comparing with the two-dimensional coordinates.

[0058] Based on projection distortion information, a bundle adjustment optimization algorithm is used to synchronously adjust the camera's intrinsic and extrinsic parameters.

[0059] Optionally, in the high-precision camera calibration system based on multimodal feature fusion described in this application embodiment, the calibration plate is placed at different positions and angles of the camera to capture multiple sets of images, and the captured multiple sets of images are preprocessed to obtain image data and point cloud data, specifically including:

[0060] Obtain the initial position of the camera, and set the placement position of the calibration plate based on the initial position of the camera;

[0061] Based on the initial position of the camera, the horizontal and vertical adjustment distances are set to obtain several adjustment positions;

[0062] Multiple sets of images were obtained by taking pictures of the calibration board at several adjustment positions.

[0063] Multiple sets of captured images are converted to grayscale to obtain multiple sets of grayscale images;

[0064] Noise removal is performed on multiple grayscale images based on median filtering and Gaussian filtering to remove salt-and-pepper noise and Gaussian noise from the grayscale images.

[0065] Enhancement processing is performed on the noise-removed grayscale image, and image data and point cloud data are obtained.

[0066] Optionally, in the high-precision camera calibration system based on multimodal feature fusion described in this application embodiment, the two-dimensional coordinates of the circular marker points on the calibration board and the phase information of each pixel are extracted based on the image data, and the phase information of the pixels is converted into position coordinates, specifically including:

[0067] Edge extraction is performed on the preprocessed image based on the Canny edge detection algorithm to obtain the edge information of the circular marker points;

[0068] The position of the center point of the circular marker is calculated based on the edge information of the circular marker, and the two-dimensional coordinates of the center point are obtained.

[0069] Phase information of pixels is obtained by performing phase calculation on the captured image based on a combination of phase shifting method and multi-frequency heterodyne method.

[0070] The mapping relationship between phase information and actual spatial position is analyzed based on the actual spatial position of the circular marker.

[0071] The pixel information is transformed based on the mapping relationship to obtain the position coordinates of the pixel.

[0072] Thirdly, embodiments of this application also provide a computer-readable storage medium, which includes a high-precision camera calibration method program based on multimodal feature fusion. When the high-precision camera calibration method program based on multimodal feature fusion is executed by a processor, it implements the steps of the high-precision camera calibration method based on multimodal feature fusion as described in any of the above claims.

[0073] As described above, the high-precision camera calibration method, system, and medium based on multimodal feature fusion provided in this application involves selecting a calibration board with known three-dimensional coordinates. The calibration board is a multimodal calibration board containing an array of circular marker points and sinusoidal fringe phase codes. The calibration board is placed at different positions and angles of the camera, and multiple sets of images are captured. The captured images are preprocessed to obtain image data and point cloud data. Based on the image data, the two-dimensional coordinates of the circular marker points on the calibration board and the phase information of each pixel are extracted. The phase information of the pixels is converted into position coordinates, and the three-dimensional coordinates of the circular marker points are obtained based on the point cloud data. Geometric features are generated based on the two-dimensional coordinates of the circular marker points, and phase features are generated based on the phase information. Constraints are established based on the geometric features and phase features to construct a multi-parameter projection model. The three-dimensional coordinates are projected based on the multi-parameter projection model, and the projection distortion information is analyzed by comparing it with the two-dimensional coordinates. Based on the projection distortion information, the camera's intrinsic and extrinsic parameters are adjusted synchronously using a bundle adjustment optimization algorithm. By acquiring the geometric and phase features in the image through the calibration board, the projection distortion information can be accurately analyzed, thereby improving the camera calibration accuracy. Attached Figure Description

[0074] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1 A flowchart illustrating a high-precision camera calibration method based on multimodal feature fusion provided in this application embodiment;

[0076] Figure 2 This is a flowchart of an image preprocessing method for a high-precision camera calibration method based on multimodal feature fusion provided in an embodiment of this application.

[0077] Figure 3 A flowchart illustrating the position coordinate acquisition method of the high-precision camera calibration method based on multimodal feature fusion provided in this application embodiment;

[0078] Figure 4 A schematic diagram of a multimodal calibration board for a high-precision camera calibration system based on multimodal feature fusion provided in this application embodiment;

[0079] Figure 5 This is a schematic diagram of dual-camera calibration for a high-precision camera calibration system based on multimodal feature fusion, provided in an embodiment of this application. Detailed Implementation

[0080] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0081] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0082] Please refer to Figure 1 , Figure 1 This is a flowchart of a high-precision camera calibration method based on multimodal feature fusion, as described in some embodiments of this application. This high-precision camera calibration method based on multimodal feature fusion is used in a terminal device and includes the following steps:

[0083] S101, Select a calibration plate with known three-dimensional coordinates. The calibration plate is a multi-modal calibration plate containing a circular marker array and a sinusoidal fringe phase code.

[0084] S102, place the calibration board at different positions and angles of the camera, take multiple sets of images, and preprocess the multiple sets of images to obtain image data and point cloud data;

[0085] S103, based on image data, extract the two-dimensional coordinates of the circular marker points on the calibration board and the phase information of each pixel point, convert the phase information of the pixel points into position coordinates, and obtain the three-dimensional coordinates of the circular marker points based on point cloud data;

[0086] S104: Generate geometric features based on the two-dimensional coordinates of the circular marker points, generate phase features based on phase information, establish constraints based on the geometric features and phase features, construct a multi-parameter projection model, project the three-dimensional coordinates based on the multi-parameter projection model, and compare and analyze the projection distortion information with the two-dimensional coordinates.

[0087] S105 uses a bundle adjustment optimization algorithm based on projection distortion information to synchronously adjust the camera's intrinsic and extrinsic parameters.

[0088] It should be noted that the calibration board should be placed at different positions and angles of the camera to capture multiple sets of images. Generally, at least 10-20 images from different perspectives are required to cover the entire field of view of the camera and different shooting angles. During the shooting process, it is important to maintain stable lighting conditions to avoid image quality fluctuations caused by changes in lighting, which could affect the accuracy of subsequent feature extraction and calibration. If a multimodal calibration board is used, the point cloud data of the LiDAR should be acquired simultaneously to ensure the temporal and spatial consistency between the point cloud data and the image data.

[0089] The core of sinusoidal fringe phase encoding is phase calculation. In practical applications, a combination of multi-frequency heterodyne and phase-shifting methods is typically used for phase calculation. The phase-shifting method projects sinusoidal fringe patterns with different phase shifts (usually three-step or four-step phase shifts) onto a calibration board, acquires the corresponding images, and calculates the phase value of each pixel based on trigonometric relationships. The multi-frequency heterodyne method, on the other hand, utilizes sinusoidal fringe patterns of different frequencies, expanding the phase value over a wider range through frequency combinations, thus resolving ambiguity issues in phase calculation and achieving sub-pixel level accuracy. When combined with circular markers, these markers can serve as reference points for phase encoding. By interpolating and calibrating the phase values ​​in the area surrounding the markers, the accuracy and stability of phase encoding are further improved. Furthermore, to adapt to different lighting conditions and scene requirements, the projection intensity and frequency of the sinusoidal fringes can be adaptively adjusted according to the ambient light intensity and the sensor's dynamic range.

[0090] Please refer to Figure 2 , Figure 2 This is a flowchart of an image preprocessing method for a high-precision camera calibration method based on multimodal feature fusion, as described in some embodiments of this application. According to embodiments of the present invention, a calibration plate is placed at different positions and angles of the camera to capture multiple sets of images, and the captured images are preprocessed to obtain image data and point cloud data. Specifically, this includes:

[0091] S201, Obtain the initial position of the camera, and set the placement position of the calibration plate based on the initial position of the camera;

[0092] S202, based on the initial position of the camera, set the horizontal and vertical adjustment distances to obtain several adjustment positions, and take pictures of the calibration board based on the several adjustment positions to obtain multiple sets of captured images;

[0093] S203 converts multiple sets of captured images to grayscale to obtain multiple sets of grayscale images;

[0094] S204, based on median filtering and Gaussian filtering, performs noise removal on multiple grayscale images, eliminating salt-and-pepper noise and Gaussian noise in the grayscale images;

[0095] S205 performs enhancement processing on the noise-removed grayscale image and acquires image data and point cloud data.

[0096] It should be noted that median filtering is used for salt-and-pepper noise. A 3×3 or 5×5 window is selected centered on each pixel. The gray values ​​of all pixels within the window are sorted, and the median value is taken as the gray value of the current pixel, effectively removing isolated noise points.

[0097] Gaussian filtering: If Gaussian noise exists in the image, Gaussian filtering is used. A suitable Gaussian kernel size is selected based on the standard deviation of the noise, such as 3×3 or 5×5. A weighted average of the pixels within the window is applied using a Gaussian function to reduce noise while preserving as much edge and detail information as possible.

[0098] Please refer to Figure 3 , Figure 3 This is a flowchart illustrating a method for obtaining position coordinates in a high-precision camera calibration method based on multimodal feature fusion, as described in some embodiments of this application. According to embodiments of the present invention, the method involves extracting the two-dimensional coordinates of circular markers on a calibration board and the phase information of each pixel from image data, and converting the phase information of the pixels into position coordinates. Specifically, this includes:

[0099] S301, Based on the Canny edge detection algorithm, the edge of the preprocessed image is extracted to obtain the edge information of the circular marker points;

[0100] S302, Calculate the position of the center point of the circular marker based on the edge information of the circular marker, and obtain the two-dimensional coordinates of the center point;

[0101] S303 performs phase calculation on captured images based on a combination of phase-shifting and multi-frequency heterodyne methods to obtain the phase information of pixels.

[0102] S304, Analyze the mapping relationship between phase information and actual spatial position based on the actual spatial position of the circular marker;

[0103] S305 performs coordinate transformation on pixel information based on the mapping relationship to obtain the position coordinates of the pixel.

[0104] It should be noted that the Canny edge detection algorithm is used to extract edges from the preprocessed image. The Canny algorithm is implemented through the following steps: First, a Gaussian filter is used to smooth the image to reduce noise interference with edge detection; then, the magnitude and direction of the image gradient are calculated to determine the intensity and direction of the edges; next, non-maximum suppression is applied to retain the points with the largest local gradients, thus refining the edges; finally, a double threshold algorithm is used to determine the true edge points, identifying points with gradient magnitudes greater than the high threshold as edge points, excluding points with gradient magnitudes less than the low threshold, and deciding whether to retain points in between based on their connectivity with the identified edge points, thereby obtaining a clear circular marker edge image.

[0105] Before projecting the sinusoidal fringe pattern, the system needs to be pre-calibrated to establish a mapping relationship between the phase value and the actual spatial position. Using the known three-dimensional coordinate information of the calibration plate and the corresponding phase value, a mathematical model (such as polynomial fitting, least squares method, etc.) is used to fit the functional relationship between the phase value and the spatial position coordinates.

[0106] According to an embodiment of the present invention, the multi-parameter projection model includes radial distortion terms, tangential distortion terms, and eccentric distortion terms;

[0107] The formula for the multi-parameter projection model is as follows:

[0108]

[0109] The formula for the radial distortion term is as follows:

[0110] Δx radial =x'(k1r 2 +k2r 4 +k3r 6 )

[0111] Δy radial =y'(k1r 2 +k2r 4 +k3r 6 )

[0112] The formula for tangential distortion is as follows:

[0113] Δx tangent =2p1x'y'+p2(r 2 +2x' 2 )

[0114] Δy tangent =p1(r 2 +2y ,2 )+2p2x'y'

[0115] The formula for the eccentric distortion term is as follows:

[0116] Δxdecentering =s1(r 2 +2x' 2 )+2s2x'y'

[0117] Δy decentering =2s1x'y'+s2(r 2 +2y' 2 )

[0118] In the formula, x and y represent the coordinates of the image pixels; x′ and y′ represent the normalized image coordinates after distortion correction; Δx radial with Δy radial The radial distortion correction amount is represented by k1, k2, and k3, which represent the radial distortion coefficients, and r represents the image plane radius. Δx tangent With Δx tangent p1 and p2 represent the tangential distortion correction amount, and p1 and p2 represent the tangential distortion coefficients, Δx decenterign with Δy decentering s1 and s2 represent the eccentricity correction amount and the eccentricity coefficients, respectively.

[0119] According to an embodiment of the present invention, the projection distortion information analysis method includes:

[0120] Initialize the parameters of the multi-parameter projection model. The parameters of the multi-parameter projection model include the camera's intrinsic parameters, extrinsic parameters, and distortion parameters. The camera's intrinsic parameters include focal length and principal point coordinates. The camera's extrinsic parameters include rotation matrix and translation vector. The distortion parameters include radial distortion, tangential distortion, and eccentric distortion.

[0121] The theoretical projection position of the circular marker in the image is calculated based on the parameters of the initialized multi-parameter projection model.

[0122] Obtain the two-dimensional coordinates of the actually detected circular marker points, and calculate the error between the theoretical projection position and the actual detected two-dimensional coordinates;

[0123] Radial distortion, tangential distortion, and eccentric distortion information are calculated based on the error between the theoretical projection position and the actual detected two-dimensional coordinates.

[0124] Projection distortion information is generated based on radial distortion information, tangential distortion information, and eccentric distortion information.

[0125] It should be noted that by continuously adjusting the camera's intrinsic and extrinsic parameters, the error between the theoretical projection position and the actual detection position is minimized, thus achieving high-precision camera calibration. Compared to a simple linear model, the multi-parameter projection model can capture more subtle projection differences, thereby significantly improving the accuracy of camera calibration and meeting the needs of applications such as high-precision visual measurement and 3D reconstruction.

[0126] According to an embodiment of the present invention, the intrinsic and extrinsic parameters of the camera are synchronously adjusted based on projection distortion information using a bundle adjustment optimization algorithm, specifically including:

[0127] The error between the theoretical projection position and the actual detected two-dimensional coordinates is obtained based on projection distortion information;

[0128] Calculate the sum of squared errors for all circular marker points, and use the sum of squared errors for all circular marker points as the target error function;

[0129] The gradient information of the target error function is analyzed based on the bundle adjustment optimization algorithm, and the target error value is analyzed based on the gradient information.

[0130] Compare the target error value with the set error threshold;

[0131] If the target error value is less than the set error threshold, the calibrated camera's internal and external parameters are obtained.

[0132] If the target error value is greater than or equal to the set error threshold, the number of iterations is adjusted, and the target error function is adjusted accordingly.

[0133] It should be noted that, based on the multi-parameter projection model, the theoretical projection positions of 3D spatial points in the image are calculated using the initialized parameters. For each feature point with established correspondence, the error between its theoretical projection position and the actual detected 2D coordinates is calculated; the commonly used error metric is Euclidean distance.

[0134] According to an embodiment of the present invention, the formula for calculating the weighted sum of reprojection error and phase consistency error by minimizing the target error function is as follows:

[0135]

[0136] N: The total number of feature points, i.e., the number of all circular marker points (or phase encoding points) on the calibration plate.

[0137] M: Number of cameras (in a multi-camera system): For example, when M cameras are calibrated together, the error function may be expanded to be a superposition of observations from all cameras.

[0138] Where the weight w i Dynamic calculation based on local contrast of feature points:

[0139]

[0140] In the formula, E represents the total error of the objective error function, and w i The dynamic weight of the i-th feature point is represented by λ, where λ represents the weighting coefficient of the phase error term, and u i Represents the observed coordinates of the i-th feature point. φ represents the projected coordinates of the i-th feature point. j This represents the measurement value of the j-th feature point. The theoretical value of the j-th feature point, σ local σ represents the standard deviation of a local region of a feature point. noise This represents the standard deviation of global noise in the image.

[0141] like Figures 4-5 As shown, in a second aspect, embodiments of this application provide a high-precision camera calibration system based on multimodal feature fusion. The system includes a memory and a processor. The memory includes a program for a high-precision camera calibration method based on multimodal feature fusion. When the program for the high-precision camera calibration method based on multimodal feature fusion is executed by the processor, it implements the following steps:

[0142] Select a calibration board with known three-dimensional coordinates. The calibration board is a multi-modal calibration board that includes a circular marker array and a sinusoidal fringe phase code.

[0143] The calibration board was placed at different positions and angles of the camera to capture multiple sets of images. The captured images were then preprocessed to obtain image data and point cloud data.

[0144] The two-dimensional coordinates of the circular markers on the calibration board and the phase information of each pixel are extracted from the image data. The phase information of the pixels is converted into position coordinates, and the three-dimensional coordinates of the circular markers are obtained based on the point cloud data.

[0145] Geometric features are generated based on the two-dimensional coordinates of circular marker points, phase features are generated based on phase information, constraints are established based on geometric features and phase features, a multi-parameter projection model is constructed, the three-dimensional coordinates are projected based on the multi-parameter projection model, and the projection distortion information is analyzed by comparing with the two-dimensional coordinates.

[0146] Based on projection distortion information, a bundle adjustment optimization algorithm is used to synchronously adjust the camera's intrinsic and extrinsic parameters.

[0147] It should be noted that the calibration board should be placed at different positions and angles of the camera to capture multiple sets of images. Generally, at least 10-20 images from different perspectives are required to cover the entire field of view of the camera and different shooting angles. During the shooting process, it is important to maintain stable lighting conditions to avoid image quality fluctuations caused by changes in lighting, which could affect the accuracy of subsequent feature extraction and calibration. If a multimodal calibration board is used, the point cloud data of the LiDAR should be acquired simultaneously to ensure the temporal and spatial consistency between the point cloud data and the image data.

[0148] The core of sinusoidal fringe phase encoding is phase calculation. In practical applications, a combination of multi-frequency heterodyne and phase-shifting methods is typically used for phase calculation. The phase-shifting method projects sinusoidal fringe patterns with different phase shifts (usually three-step or four-step phase shifts) onto a calibration board, acquires the corresponding images, and calculates the phase value of each pixel based on trigonometric relationships. The multi-frequency heterodyne method, on the other hand, utilizes sinusoidal fringe patterns of different frequencies, expanding the phase value over a wider range through frequency combinations, thus resolving ambiguity issues in phase calculation and achieving sub-pixel level accuracy. When combined with circular markers, these markers can serve as reference points for phase encoding. By interpolating and calibrating the phase values ​​in the area surrounding the markers, the accuracy and stability of phase encoding are further improved. Furthermore, to adapt to different lighting conditions and scene requirements, the projection intensity and frequency of the sinusoidal fringes can be adaptively adjusted according to the ambient light intensity and the sensor's dynamic range.

[0149] According to an embodiment of the present invention, a calibration board is placed at different positions and angles of the camera to capture multiple sets of images, and the captured multiple sets of images are preprocessed to obtain image data and point cloud data, specifically including:

[0150] Obtain the initial position of the camera, and set the placement position of the calibration plate based on the initial position of the camera;

[0151] Based on the initial position of the camera, the horizontal and vertical adjustment distances are set to obtain several adjustment positions;

[0152] Multiple sets of images were obtained by taking pictures of the calibration board at several adjustment positions.

[0153] Multiple sets of captured images are converted to grayscale to obtain multiple sets of grayscale images;

[0154] Noise removal is performed on multiple grayscale images based on median filtering and Gaussian filtering to remove salt-and-pepper noise and Gaussian noise from the grayscale images.

[0155] Enhancement processing is performed on the noise-removed grayscale image, and image data and point cloud data are obtained.

[0156] It should be noted that median filtering is used for salt-and-pepper noise. A 3×3 or 5×5 window is selected centered on each pixel. The gray values ​​of all pixels within the window are sorted, and the median value is taken as the gray value of the current pixel, effectively removing isolated noise points.

[0157] Gaussian filtering: If Gaussian noise exists in the image, Gaussian filtering is used. A suitable Gaussian kernel size is selected based on the standard deviation of the noise, such as 3×3 or 5×5. A weighted average of the pixels within the window is applied using a Gaussian function to reduce noise while preserving as much edge and detail information as possible.

[0158] According to an embodiment of the present invention, the two-dimensional coordinates of the circular markers on the calibration board and the phase information of each pixel are extracted based on image data, and the phase information of the pixels is converted into position coordinates, specifically including:

[0159] Edge extraction is performed on the preprocessed image based on the Canny edge detection algorithm to obtain the edge information of the circular marker points;

[0160] The position of the center point of the circular marker is calculated based on the edge information of the circular marker, and the two-dimensional coordinates of the center point are obtained.

[0161] Phase information of pixels is obtained by performing phase calculation on the captured image based on a combination of phase shifting method and multi-frequency heterodyne method.

[0162] The mapping relationship between phase information and actual spatial position is analyzed based on the actual spatial position of the circular marker.

[0163] The pixel information is transformed based on the mapping relationship to obtain the position coordinates of the pixel.

[0164] It should be noted that the Canny edge detection algorithm is used to extract edges from the preprocessed image. The Canny algorithm is implemented through the following steps: First, a Gaussian filter is used to smooth the image to reduce noise interference with edge detection; then, the magnitude and direction of the image gradient are calculated to determine the intensity and direction of the edges; next, non-maximum suppression is applied to retain the points with the largest local gradients, thus refining the edges; finally, a double threshold algorithm is used to determine the true edge points, identifying points with gradient magnitudes greater than the high threshold as edge points, excluding points with gradient magnitudes less than the low threshold, and deciding whether to retain points in between based on their connectivity with the identified edge points, thereby obtaining a clear circular marker edge image.

[0165] Before projecting the sinusoidal fringe pattern, the system needs to be pre-calibrated to establish a mapping relationship between the phase value and the actual spatial position. Using the known three-dimensional coordinate information of the calibration plate and the corresponding phase value, a mathematical model (such as polynomial fitting, least squares method, etc.) is used to fit the functional relationship between the phase value and the spatial position coordinates.

[0166] According to an embodiment of the present invention, the multi-parameter projection model includes radial distortion terms, tangential distortion terms, and eccentric distortion terms;

[0167] The formula for the multi-parameter projection model is as follows:

[0168]

[0169] The formula for the radial distortion term is as follows:

[0170] Δx radial=x'(k1r 2 +k2r 4 +k3r 6 )

[0171] Δy radial =y'(k1r 2 +k2r 4 +k3r 6 )

[0172] The formula for tangential distortion is as follows:

[0173] Δx tangent =2p1x'y'+p2(r 2 +2x' 2 )

[0174] Δy tangent =p1(r 2 +2y' 2 )+2p2x'y'

[0175] The formula for the eccentric distortion term is as follows:

[0176] Δx decentering =s1(r 2 +2x' 2 )+2s2x'y'

[0177] Δy decentering =2s1x'y'+s2(r 2 +2y' 2 )

[0178] In the formula, x and y represent the coordinates of the image pixels; x′ and y′ represent the normalized image coordinates after distortion correction; Δx radial with Δy radial The radial distortion correction amount is represented by k1, k2, and k3, which represent the radial distortion coefficients, and r represents the image plane radius. Δx tangent With Δx tangent p1 and p2 represent the tangential distortion correction amount, and p1 and p2 represent the tangential distortion coefficients, Δx decenterign with Δy decentering s1 and s2 represent the eccentricity correction amount and the eccentricity coefficients, respectively.

[0179] According to an embodiment of the present invention, the projection distortion information analysis method includes:

[0180] Initialize the parameters of the multi-parameter projection model. The parameters of the multi-parameter projection model include the camera's intrinsic parameters, extrinsic parameters, and distortion parameters. The camera's intrinsic parameters include focal length and principal point coordinates. The camera's extrinsic parameters include rotation matrix and translation vector. The distortion parameters include radial distortion, tangential distortion, and eccentric distortion.

[0181] The theoretical projection position of the circular marker in the image is calculated based on the parameters of the initialized multi-parameter projection model.

[0182] Obtain the two-dimensional coordinates of the actually detected circular marker points, and calculate the error between the theoretical projection position and the actual detected two-dimensional coordinates;

[0183] Radial distortion, tangential distortion, and eccentric distortion information are calculated based on the error between the theoretical projection position and the actual detected two-dimensional coordinates.

[0184] Projection distortion information is generated based on radial distortion information, tangential distortion information, and eccentric distortion information.

[0185] It should be noted that by continuously adjusting the camera's intrinsic and extrinsic parameters, the error between the theoretical projection position and the actual detection position is minimized, thus achieving high-precision camera calibration. Compared to a simple linear model, the multi-parameter projection model can capture more subtle projection differences, thereby significantly improving the accuracy of camera calibration and meeting the needs of applications such as high-precision visual measurement and 3D reconstruction.

[0186] According to an embodiment of the present invention, the intrinsic and extrinsic parameters of the camera are synchronously adjusted based on projection distortion information using a bundle adjustment optimization algorithm, specifically including:

[0187] The error between the theoretical projection position and the actual detected two-dimensional coordinates is obtained based on projection distortion information;

[0188] Calculate the sum of squared errors for all circular marker points, and use the sum of squared errors for all circular marker points as the target error function;

[0189] The gradient information of the target error function is analyzed based on the bundle adjustment optimization algorithm, and the target error value is analyzed based on the gradient information.

[0190] Compare the target error value with the set error threshold;

[0191] If the target error value is less than the set error threshold, the calibrated camera's internal and external parameters are obtained.

[0192] If the target error value is greater than or equal to the set error threshold, the number of iterations is adjusted, and the target error function is adjusted accordingly.

[0193] It should be noted that, based on the multi-parameter projection model, the theoretical projection positions of 3D spatial points in the image are calculated using the initialized parameters. For each feature point with established correspondence, the error between its theoretical projection position and the actual detected 2D coordinates is calculated; the commonly used error metric is Euclidean distance.

[0194] According to an embodiment of the present invention, the formula for calculating the weighted sum of reprojection error and phase consistency error by minimizing the target error function is as follows:

[0195]

[0196] Where the weight w i Dynamic calculation based on local contrast of feature points:

[0197]

[0198] In the formula, E represents the total error of the objective error function, and w i The dynamic weight of the i-th feature point is represented by λ, where λ represents the weighting coefficient of the phase error term, and u i Represents the observed coordinates of the i-th feature point. φ represents the projected coordinates of the i-th feature point. j This represents the measurement value of the j-th feature point. The theoretical value of the j-th feature point, σ local σ represents the standard deviation of a local region of a feature point. noise This represents the standard deviation of global noise in the image.

[0199] A third aspect of the present invention provides a computer-readable storage medium including a high-precision camera calibration method program based on multimodal feature fusion. When the high-precision camera calibration method program based on multimodal feature fusion is executed by a processor, it implements the steps of the high-precision camera calibration method based on multimodal feature fusion as described above.

[0200] This invention discloses a high-precision camera calibration method, system, and medium based on multimodal feature fusion. The method involves selecting a calibration board with known three-dimensional coordinates. The calibration board is a multimodal calibration board containing an array of circular marker points and sinusoidal fringe phase codes. The calibration board is placed at different positions and angles of the camera, and multiple sets of images are captured. These images are preprocessed to obtain image data and point cloud data. Based on the image data, the two-dimensional coordinates of the circular marker points on the calibration board and the phase information of each pixel are extracted. The phase information of the pixels is converted into position coordinates, and the three-dimensional coordinates of the circular marker points are obtained based on the point cloud data. Geometric features are generated based on the two-dimensional coordinates of the circular marker points, and phase features are generated based on the phase information. Constraints are established based on the geometric and phase features to construct a multi-parameter projection model. The three-dimensional coordinates are projected using the multi-parameter projection model, and the projection distortion information is analyzed by comparing it with the two-dimensional coordinates. Based on the projection distortion information, the camera's intrinsic and extrinsic parameters are simultaneously adjusted using a bundle adjustment optimization algorithm. By acquiring geometric and phase features from the images through the calibration board, the projection distortion information can be accurately analyzed, improving the camera calibration accuracy.

[0201] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0202] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0203] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0204] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0205] Alternatively, if the integrated units of the present invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

Claims

1. A high-precision camera calibration method based on multimodal feature fusion, characterized in that, include: Select a calibration board with known three-dimensional coordinates. The calibration board is a multi-modal calibration board that includes a circular marker array and a sinusoidal fringe phase code. The calibration board was placed at different positions and angles of the camera to capture multiple sets of images. The captured images were then preprocessed to obtain image data and point cloud data. The two-dimensional coordinates of the circular markers on the calibration board and the phase information of each pixel are extracted from the image data. The phase information of the pixels is converted into position coordinates, and the three-dimensional coordinates of the circular markers are obtained based on the point cloud data. Geometric features are generated based on the two-dimensional coordinates of circular markers, phase features are generated based on phase information, constraints are established based on geometric features and phase features, a multi-parameter projection model is constructed, three-dimensional coordinates are projected based on the multi-parameter projection model, and projection distortion information is analyzed by comparing with two-dimensional coordinates. Based on projection distortion information, the camera's intrinsic and extrinsic parameters are simultaneously adjusted using a bundle adjustment optimization algorithm. Based on image data, the two-dimensional coordinates of the circular markers on the calibration board and the phase information of each pixel are extracted. The phase information of the pixels is then converted into position coordinates, specifically including: The edge information of the circular marker points is obtained by extracting the edge of the preprocessed image based on the Canny edge detection algorithm. The position of the center point of the circular marker is calculated based on the edge information of the circular marker, and the two-dimensional coordinates of the center point are obtained. Phase information of pixels is obtained by performing phase calculation on the captured image based on a combination of phase shifting method and multi-frequency heterodyne method. The mapping relationship between phase information and actual spatial position is analyzed based on the actual spatial position of the circular marker. Based on the mapping relationship, coordinate transformation is performed on pixel information to obtain the position coordinates of the pixel; The multi-parameter projection model includes radial distortion terms, tangential distortion terms, and eccentric distortion terms; The formula for the multi-parameter projection model is as follows: ； The formula for the radial distortion term is as follows: ； The formula for tangential distortion is as follows: ； The formula for the eccentric distortion term is as follows: ； In the formula, The coordinates of image pixels; Represents the normalized image coordinates after distortion correction; and This indicates the radial distortion correction amount. Represents the radial distortion coefficient. Represents the radius of the image plane. and This indicates the tangential distortion correction amount. Indicates the tangential distortion coefficient. and This indicates the amount of eccentric distortion correction. This represents the eccentricity distortion coefficient.

2. The high-precision camera calibration method based on multimodal feature fusion according to claim 1, characterized in that, The calibration board was placed at different positions and angles of the camera to capture multiple sets of images. These images were then preprocessed to obtain image data and point cloud data, specifically including: Obtain the initial position of the camera, and set the placement position of the calibration plate based on the initial position of the camera; Based on the initial position of the camera, the horizontal and vertical adjustment distances are set to obtain several adjustment positions; Multiple sets of images were obtained by taking pictures of the calibration board at several adjustment positions. Multiple sets of captured images are converted to grayscale to obtain multiple sets of grayscale images; Noise removal is performed on multiple grayscale images based on median filtering and Gaussian filtering to remove salt-and-pepper noise and Gaussian noise from the grayscale images. Enhancement processing is performed on the noise-removed grayscale image, and image data and point cloud data are obtained.

3. The high-precision camera calibration method based on multimodal feature fusion according to claim 2, characterized in that, Methods for analyzing projection distortion information include: Initialize the parameters of the multi-parameter projection model. The parameters of the multi-parameter projection model include the camera's intrinsic parameters, extrinsic parameters, and distortion parameters. The camera's intrinsic parameters include focal length and principal point coordinates. The camera's extrinsic parameters include rotation matrix and translation vector. The distortion parameters include radial distortion, tangential distortion, and eccentric distortion. The theoretical projection position of the circular marker in the image is calculated based on the parameters of the initialized multi-parameter projection model. Obtain the two-dimensional coordinates of the actually detected circular marker points, and calculate the error between the theoretical projection position and the actual detected two-dimensional coordinates; Radial distortion, tangential distortion, and eccentric distortion information are calculated based on the error between the theoretical projection position and the actual detected two-dimensional coordinates. Projection distortion information is generated based on radial distortion information, tangential distortion information, and eccentric distortion information.

4. The high-precision camera calibration method based on multimodal feature fusion according to claim 3, characterized in that, Based on projection distortion information, a bundle adjustment optimization algorithm is used to synchronously adjust the camera's intrinsic and extrinsic parameters, specifically including: The error between the theoretical projection position and the actual detected two-dimensional coordinates is obtained based on projection distortion information; Calculate the sum of squared errors for all circular marker points, and use the sum of squared errors for all circular marker points as the target error function; The gradient information of the target error function is analyzed based on the bundle adjustment optimization algorithm, and the target error value is analyzed based on the gradient information. Compare the target error value with the set error threshold; If the target error value is less than the set error threshold, the calibrated camera's internal and external parameters are obtained. If the target error value is greater than or equal to the set error threshold, the number of iterations is adjusted, and the target error function is adjusted accordingly.

5. A high-precision camera calibration system based on multimodal feature fusion, characterized in that, The system includes a memory and a processor. The memory includes a program for the high-precision camera calibration method based on multimodal feature fusion as described in claim 1. When the program for the high-precision camera calibration method based on multimodal feature fusion is executed by the processor, it performs the following steps: Select a calibration board with known three-dimensional coordinates. The calibration board is a multi-modal calibration board that includes a circular marker array and a sinusoidal fringe phase code. The calibration board was placed at different positions and angles of the camera to capture multiple sets of images. The captured images were then preprocessed to obtain image data and point cloud data. The two-dimensional coordinates of the circular markers on the calibration board and the phase information of each pixel are extracted from the image data. The phase information of the pixels is converted into position coordinates, and the three-dimensional coordinates of the circular markers are obtained based on the point cloud data. Geometric features are generated based on the two-dimensional coordinates of circular markers, phase features are generated based on phase information, constraints are established based on geometric features and phase features, a multi-parameter projection model is constructed, three-dimensional coordinates are projected based on the multi-parameter projection model, and projection distortion information is analyzed by comparing with two-dimensional coordinates. Based on projection distortion information, a bundle adjustment optimization algorithm is used to synchronously adjust the camera's intrinsic and extrinsic parameters.

6. The high-precision camera calibration system based on multimodal feature fusion according to claim 5, characterized in that, The calibration board was placed at different positions and angles of the camera to capture multiple sets of images. These images were then preprocessed to obtain image data and point cloud data, specifically including: Obtain the initial position of the camera, and set the placement position of the calibration plate based on the initial position of the camera; Based on the initial position of the camera, the horizontal and vertical adjustment distances are set to obtain several adjustment positions; Multiple sets of images were obtained by taking pictures of the calibration board at several adjustment positions. Multiple sets of captured images are converted to grayscale to obtain multiple sets of grayscale images; Noise removal is performed on multiple grayscale images based on median filtering and Gaussian filtering to remove salt-and-pepper noise and Gaussian noise from the grayscale images. Enhancement processing is performed on the noise-removed grayscale image, and image data and point cloud data are obtained.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a high-precision camera calibration method program based on multimodal feature fusion. When the high-precision camera calibration method program based on multimodal feature fusion is executed by a processor, it implements the steps of the high-precision camera calibration method based on multimodal feature fusion as described in any one of claims 1 to 4.

Images