Computer vision based dms camera calibration method

Abstract

The present application relates to the technical field of vehicle vision calibration, and discloses a DMS camera calibration method based on computer vision. The method comprises the following steps: collecting a multi-view original image sequence in the target vehicle cockpit, and synchronously acquiring the camera installation position, the visual angle inclination angle and the environmental light intensity parameter; after distortion correction is performed on the original image sequence, the edge features of the calibration reference in each frame of image are extracted through image recognition to generate an initial calibration matrix containing the mapping relationship between the pixel coordinates and the physical coordinates; a three-dimensional space projection model is constructed based on the initial calibration matrix, an iterative optimization algorithm is used to adjust the model rotation and translation parameters, so that the error between the virtual imaging position and the actual edge features is minimized; the intrinsic matrix and the extrinsic matrix of the optimized model are extracted and fused into the final calibration parameters, which are written into the DMS camera firmware storage area. The method improves the precision and environmental adaptability of the DMS camera calibration, and provides stable technical support for the driver state monitoring.

Description

Technical Field

[0001] This invention relates to the field of vehicle vision calibration technology, specifically a DMS camera calibration method based on computer vision. Background Technology

[0002] Driver monitoring systems (DMS), as an important component of vehicle active safety technology, use cameras to collect information such as driver facial features and body movements to achieve real-time monitoring of dangerous states such as fatigued driving and distracted driving, thereby reducing the probability of traffic accidents. The calibration quality of the DMS camera directly determines the system's monitoring accuracy. Its core lies in establishing an accurate mapping relationship between image pixel coordinates and real three-dimensional spatial coordinates, providing a basic coordinate reference for subsequent image analysis and state recognition.

[0003] Current DMS camera calibration methods on the market have many limitations. Some traditional methods rely on manually setting up calibration scenarios, requiring the fixing of calibration boards of specific specifications in the cockpit. The operation process is cumbersome and time-consuming, making it difficult to adapt to the high-efficiency assembly requirements of vehicle production lines. These methods also require a high level of professional skills from operators, and differences in operation by different personnel can easily lead to deviations in calibration results, affecting the consistency of calibration during mass production.

[0004] Environmental factors also significantly interfere with existing calibration methods. The lighting conditions inside a vehicle's cockpit are complex and variable; scenarios such as backlighting at dawn and dusk, and alternating light and shadow in tunnels make it difficult to extract image features from calibration reference objects. Traditional methods lack specific parameter adaptation mechanisms, making them prone to feature recognition errors and thus reducing calibration accuracy. Simultaneously, the installation position of the DMS camera on the vehicle may shift due to differences in vehicle model and assembly errors, causing changes in the viewing angle. Existing methods often struggle to fully capture these installation parameters, resulting in calibration models that cannot accurately match the actual installation state, ultimately affecting the accuracy of driver condition monitoring.

[0005] The optimization mechanisms of existing calibration methods are not perfect. Some methods use fixed projection model parameters and do not dynamically adjust for the different cockpit structures of different vehicles, making it difficult to offset the effects of changes in seat position and obstruction by interior components. Other methods have iterative optimization algorithms with slow convergence speeds, failing to minimize errors quickly, resulting in low calibration efficiency. Furthermore, the optimized parameters lack stability and are prone to calibration drift due to vibrations and other factors during vehicle operation, requiring frequent recalibration and increasing usage costs and operational complexity. These problems hinder the further promotion and application of DMS technology in the field of vehicle safety, necessitating a more efficient, accurate, and adaptable calibration method to address these challenges. Summary of the Invention

[0006] The purpose of this invention is to provide a computer vision-based DMS camera calibration method to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides a computer vision-based DMS camera calibration method, the method comprising:

[0008] Collect raw image sequences from multiple perspectives inside the cockpit of the target vehicle. The raw image sequences include parameters such as camera installation position, viewing angle, and ambient light intensity.

[0009] The original image sequence is subjected to distortion correction processing. The edge features of the calibration reference object in each frame image are extracted by image recognition technology to generate an initial calibration matrix containing the mapping relationship between pixel coordinates and physical coordinates.

[0010] A three-dimensional spatial projection model is constructed based on the initial calibration matrix. The rotation and translation parameters in the projection model are adjusted through an iterative optimization algorithm to minimize the error between the virtual imaging position of the calibration reference object in the projection model and the edge feature in the actual image.

[0011] The intrinsic and extrinsic parameter matrices in the optimized projection model are extracted, fused into the final calibration parameters, and written into the firmware storage area of ​​the DMS camera.

[0012] Preferably, the sequence of original images acquired from multiple perspectives within the cockpit of the target vehicle includes:

[0013] A checkerboard calibration board with known geometric dimensions is arranged in the cockpit, and the DMS camera is controlled to capture images of the calibration board at different pitch angles;

[0014] Simultaneously record the exposure time, ambient light sensor data, and camera mechanical posture data for each frame of image;

[0015] Timestamp alignment is performed on consecutive frames of images to form an original image sequence containing spatiotemporal correlation.

[0016] Preferably, the distortion correction processing of the original image sequence includes:

[0017] Detect the sub-pixel coordinates of the calibration board corner points in each frame of the image, and establish a distortion correction mapping table based on the radial distortion coefficient and tangential distortion coefficient of the camera lens;

[0018] A bilinear interpolation algorithm is used to perform pixel-by-pixel correction on the original image sequence to generate a distortion-free image sequence.

[0019] Preferably, the step of extracting the edge features of the calibration reference object in each frame of the image using image recognition technology includes:

[0020] Adaptive threshold segmentation is performed on the distortion-free image sequence to extract the binarized contour of the calibration plate region;

[0021] Morphological gradient operations are used to enhance the contour edges, and the curvature extrema of the contour chain code are calculated as feature points.

[0022] The pixel coordinates of the feature points are matched one by one with the actual physical coordinates of the calibration board to form an initial feature correspondence table.

[0023] Preferably, generating the initial calibration matrix containing the mapping relationship between pixel coordinates and physical coordinates includes:

[0024] Calculate the initial solution of the homography transformation matrix based on the matching point pairs in the initial feature correspondence table;

[0025] The accuracy of the homography transformation matrix is ​​optimized by eliminating mismatched point pairs through a random sampling consensus algorithm.

[0026] The optimized homography transformation matrix is ​​multiplied by the distortion correction mapping table to obtain the initial calibration matrix.

[0027] Preferably, the construction of the three-dimensional spatial projection model includes:

[0028] The initial calibration matrix is ​​decomposed into an intrinsic parameter matrix for camera focal length and principal point offset, and an extrinsic parameter matrix for coordinate system transformation.

[0029] The projection equations between the cockpit world coordinate system and the camera imaging plane are established based on the external parameter matrix;

[0030] By adding a lens distortion compensation term to the projection equation, a complete three-dimensional spatial projection model is formed.

[0031] Preferably, adjusting the rotation and translation parameters in the projection model through an iterative optimization algorithm includes:

[0032] The reprojection error of the feature points of the calibration plate in the projection model is calculated, and the extrinsic parameter matrix is ​​iteratively optimized using the Levenberg-Marquardt algorithm.

[0033] The optimization is terminated when the change in reprojection error is less than the threshold after three consecutive iterations, and the final rotation and translation parameters are locked.

[0034] Preferably, the extracted intrinsic and extrinsic parameter matrices in the optimized projection model include:

[0035] The focal length parameter and principal point coordinates of the intrinsic parameter matrix are separated from the converged 3D spatial projection model.

[0036] The optimized rotation matrix and translation vector are combined into an extrinsic parameter matrix;

[0037] Verify the consistency of the projection of the intrinsic parameter matrix and the extrinsic parameter matrix at different positions on the calibration plate.

[0038] Preferably, writing the final calibration parameters into the firmware storage area of ​​the DMS camera includes:

[0039] Serialize the intrinsic and extrinsic parameter matrices into a binary data stream;

[0040] The binary data stream is transmitted to the DMS camera controller via the vehicle CAN bus protocol.

[0041] A dedicated storage block for calibration parameters is allocated in the firmware storage area to achieve persistent storage even when power is off.

[0042] Preferably, the method further includes:

[0043] The vibration amplitude of the DMS camera mounting bracket is monitored in real time, and the calibration parameter verification process is triggered when the vibration amplitude exceeds the threshold.

[0044] Read the calibration parameters from the firmware storage area and load them into the three-dimensional spatial projection model to calculate the current imaging quality degradation index;

[0045] The offset compensation value of the extrinsic parameter matrix in the calibration parameters is dynamically adjusted according to the degradation index.

[0046] Compared with the prior art, the beneficial effects of the present invention are:

[0047] In the data acquisition phase, this method specifically acquires multi-view raw image sequences and simultaneously obtains key parameters such as camera installation position, viewing angle, and ambient light intensity. This overcomes the limitations of traditional calibration methods that rely solely on single-view images or ignore environmental and installation parameters. Multi-view images can comprehensively cover the spatial information within the cockpit, reducing feature omissions caused by blind spots. Meanwhile, the simultaneously acquired installation and lighting parameters provide rich foundational data for subsequent image processing and model adjustments, enabling the calibration process to fully adapt to installation differences across different vehicle models and complex cockpit lighting environments, significantly improving the method's scene adaptability.

[0048] In the image processing and initial calibration matrix generation stages, distortion correction effectively offsets image distortion caused by the camera's optical system, ensuring the accuracy of subsequent feature extraction. By leveraging image recognition technology to accurately extract edge features of the calibration reference object and establish a mapping relationship between pixel coordinates and physical coordinates, the generated initial calibration matrix has a solid data foundation. This avoids initial parameter deviations caused by fuzzy feature extraction or chaotic coordinate mapping in traditional methods, laying a reliable foundation for subsequent model construction and optimization.

[0049] The construction of a 3D spatial projection model and the application of an iterative optimization algorithm have significantly improved calibration accuracy. This model can accurately simulate the imaging process of the DMS camera, continuously reducing the error between the virtual imaging position and the actual image features by dynamically adjusting rotation and translation parameters, ensuring that the calibration parameters closely match the actual working state of the camera. Compared to the fixed-parameter projection model in traditional methods, this dynamic optimization mechanism can effectively cope with complex factors such as structural differences and installation errors within the cockpit. Even with adjustments to seat positions or slight obstruction from interior components, iterative optimization can offset interference, guaranteeing the accuracy and stability of the calibration results.

[0050] The fusion of final calibration parameters and the firmware writing design further enhance the practicality and engineering value of the method. Merging the intrinsic and extrinsic parameter matrices into the final calibration parameters achieves a high degree of parameter integration, avoiding the retrieval chaos that might result from storing multiple parameters independently. Writing the parameters directly into the firmware storage area of ​​the DMS camera allows the calibration results to be accessed by the camera in real time, eliminating the need for recalibration each time the system starts, significantly improving the startup speed and operating efficiency of the DMS system. Simultaneously, this parameter storage method effectively resists factors such as vibration and electromagnetic interference during vehicle operation, reducing calibration drift, extending the calibration cycle, and lowering maintenance costs during use.

[0051] This method eliminates the need for complex manual operations and specialized calibration equipment, simplifying the calibration process. It is adaptable to the installation requirements of DMS cameras on different vehicle models, efficiently completing calibration work in both batch calibration scenarios on the vehicle production line and individual calibration scenarios during later maintenance. Its stable calibration accuracy and good environmental adaptability provide precise coordinate references for driver condition monitoring, enabling the DMS system to more accurately identify dangerous states such as fatigued driving and distracted driving, further enhancing the vehicle's active safety performance and providing stronger protection for the lives of drivers and passengers. It also creates favorable conditions for the widespread application of DMS technology in various types of vehicles. Attached Figure Description

[0052] Figure 1 Analysis diagram for calibration and optimization process;

[0053] Figure 2 A flowchart for acquiring raw image sequences from multiple perspectives;

[0054] Figure 3 A flowchart for extracting edge features of the calibration reference object;

[0055] Figure 4 This is a verification diagram for the 3D projection model. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Please see Figure 1 This invention provides a computer vision-based DMS camera calibration method. The method includes: acquiring a sequence of original images from multiple perspectives within the cockpit of a target vehicle, the sequence containing key parameters such as camera installation position, viewing angle, and ambient light intensity; performing distortion correction on the original image sequence and extracting edge features of the calibration reference object in each frame using image recognition technology, thereby generating an initial calibration matrix containing the mapping relationship between pixel coordinates and physical coordinates; constructing a three-dimensional spatial projection model based on this matrix, and adjusting the rotation and translation parameters in the model through an iterative optimization algorithm to minimize the error between the virtual imaging position of the calibration reference object in the model and the edge features in the actual image; extracting intrinsic and extrinsic parameter matrices from the optimized projection model, fusing them into final calibration parameters, and writing them into the firmware storage area of ​​the DMS camera to complete the entire calibration process.

[0058] Example 1: See Figure 2 When implementing this calibration method, comprehensive and systematic data acquisition is paramount, requiring the deployment of a structurally known calibration reference object within the target vehicle's cockpit. A high-precision checkerboard-patterned flat plate is typically chosen as the calibration board, with the physical dimensions of each square measured and recorded using precision instruments to establish a stable world coordinate system. The DMS camera is controlled to perform a series of preset actions, causing it to pitch around its axis at different angles, thereby capturing images of the calibration board from multiple perspectives. This process simulates potential head posture changes by the driver and the field of view that the camera might observe after actual installation. The capture of each perspective is not isolated; simultaneously recorded are the exposure time parameters of the image sensor at the moment of exposure, the real-time illuminance data read by the ambient light sensor installed in the cockpit, and the instantaneous attitude angle information fed back by the miniature inertial measurement unit integrated into the camera module. All these data streams are stamped and aligned through a unified high-precision time generator, ensuring that each image frame corresponds strictly to its corresponding auxiliary data on the timeline, ultimately forming a tightly linked original image sequence in the spatiotemporal dimension. This sequence provides rich and relevant background information for subsequent precise analysis.

[0059] After image acquisition, the next step is preprocessing the image sequence to eliminate geometric distortions introduced by the lens itself. Lens distortion is mainly divided into two categories: radial distortion and tangential distortion, which cause straight lines in the image to bend after imaging. The correction process begins with the precise detection of the calibration plate corner positions in each frame of the image. The positioning accuracy of the corner points needs to be improved from the pixel level to the sub-pixel level, which is usually achieved by analyzing the gray-level gradient distribution of pixels near the corner points. Based on the radial and tangential distortion coefficients calibrated at the camera lens factory or obtained through preliminary calculations, a distortion correction mapping table covering the entire imaging plane can be constructed. This mapping table is essentially a two-dimensional array that indicates where each pixel in the original distorted image should obtain its brightness value in the corrected image. The actual pixel value correction process uses a bilinear interpolation algorithm. For each integer coordinate pixel in the corrected image, its corresponding floating-point coordinate position in the original distorted image is found by looking up the mapping table. This position usually does not fall exactly at the pixel center of the original image. Next, the gray values ​​of the four nearest neighbors around this floating-point coordinate are calculated, and a weighted average is taken based on their relative distances to obtain a smoother and more accurate gray value for the current pixel. Through this pixel-by-pixel mapping and interpolation operation, the geometry of the entire image is corrected, generating a distortion-free image sequence with a more realistic visual effect and restored straight lines.

[0060] After obtaining the distortion-free image sequence, the next step is to stably and accurately extract the edge and feature point information of the calibration board, which serves as the calibration reference. Since the lighting conditions inside the cockpit can be complex and variable, directly using a fixed threshold for binarization often yields poor results; therefore, an adaptive thresholding segmentation algorithm is required. This algorithm does not use a single threshold for the entire image but dynamically calculates the local threshold based on the grayscale distribution within the neighborhood of each pixel in the image. This allows for better separation of the calibration board area from the background, resulting in a connected binary contour region. To further enhance the edge information of the contour, morphological gradient operations can be used. Morphological gradients, by calculating the difference between the image after dilation and the image after erosion, effectively highlight the edges of the target, making the boundaries of the calibration board clearer and sharper. After obtaining the enhanced contour, it is necessary to locate its key feature points, typically the corner points of the calibration board. One method is to calculate the chain code representation of the contour and then analyze the changes in the chain code direction to find extreme points where curvature abruptly changes; these points often correspond to the various corner points of the calibration board. Matching the corner pixel coordinates extracted through image processing techniques with the known corner world coordinates (measured in physical units, such as millimeters) of the calibration board is a crucial step in establishing the connection between pixel space and physical space. During matching, the checkerboard pattern of the calibration board must be followed to ensure that the order of corner points in the image matches the order of corner points in the world coordinate system, thus forming an accurate initial feature point correspondence table.

[0061] Based on the established correspondence between pixel coordinates and physical coordinates, a preliminary mapping transformation model can be calculated. The homography transformation matrix describes the projection mapping relationship between two planes, which in this case is the relationship between the world coordinate plane where the calibration board is located and the camera imaging plane. Using methods such as direct linear transformation, the initial solution of the homography transformation matrix can be calculated from multiple sets of matching point pairs. However, during the feature point matching process, some erroneous matching pairs may inevitably be introduced, which will seriously affect the accuracy of the initial solution. To remove these erroneous data, a random sample consensus algorithm is used for optimization. This algorithm is a robust model parameter estimation method. It randomly selects the minimum required number of points from the matching point pairs to calculate a homography matrix model, and then verifies the degree of conformity of all matching points with respect to the model, counting the number of interior points within the error tolerance. This process is repeated iteratively multiple times, and finally the model with the most interior points is selected as the optimized result. The accuracy and reliability of the homography transformation matrix calculated from the matching point pairs after being purified by the random sample consensus algorithm are significantly improved. Finally, the optimized homography matrix is ​​multiplied and fused with the distortion correction mapping table used for geometric correction in the first step to obtain a comprehensive initial calibration matrix. This initial matrix not only contains the planar projection relationship but also implicitly includes lens distortion correction information, providing important initial parameters for constructing more complex 3D projection models.

[0062] The entire implementation process begins with meticulous data acquisition and preparation, followed by detailed image preprocessing to eliminate hardware errors, robust image feature extraction and matching, and finally, the establishment of a preliminary coordinate mapping relationship through robust model calculation. Each step is interconnected. The multi-view, multi-parameter synchronous data acquired during the acquisition phase provides rich material for processing; the distortion correction stage ensures the geometric authenticity of the input data; the adaptive processing strategy addresses the challenges of environmental changes; and the introduction of the random sample consensus algorithm effectively guarantees the accuracy of model parameter estimation. This process fully considers various non-ideal situations that may be encountered in real-world applications, aiming to lay a solid foundation for obtaining accurate and reliable calibration parameters for the DMS camera through systematic operation steps, enabling it to accurately understand the correspondence between two-dimensional image pixels and the three-dimensional cockpit space.

[0063] Example 2: See Figure 3For the distortion-free image sequence after distortion correction, in-depth analysis is needed to extract the precise features of the calibration board. Adaptive thresholding segmentation is first applied to separate the calibration board region from the background. Since the lighting conditions inside the cockpit may be uneven, with shadows cast by the windshield or localized illumination from the dashboard lights, global thresholding methods often fail. The adaptive thresholding method calculates the threshold individually for each small neighborhood of each pixel in the image, dynamically adapting to local lighting changes, thus producing a binary image with balanced contrast. In this binary image, the calibration board region should appear as a continuous white block, while the background is black. However, the edges of the image may appear blurred or discontinuous due to noise or lighting transitions. To enhance edge information, morphological gradient operations are introduced into the processing flow. This operation calculates the difference between the result after dilation and the result after erosion. Dilation expands the white area and bridges small breaks, while erosion erodes the boundaries of the white area. Subtracting the two leaves only the area where the object boundary is significantly enhanced, making the outline of each square on the calibration board exceptionally clear and sharp.

[0064] After obtaining the edge-enhanced binary contour, the next crucial step is to locate its feature points, typically the corner points where each black and white square on the calibration board intersects. Chain code representation is used to describe the contour's orientation; chain code depicts the object's shape by recording the direction sequence of adjacent pixels on the contour. By analyzing the rate of change of this direction sequence, the curvature of each point on the contour can be calculated. The extreme points of curvature usually correspond to corner locations because the orientation changes abruptly at these points. The method for calculating curvature extreme points is robust enough to find these key locations, even with minor irregularities in the contour. The pixel coordinates of all detected corner points are recorded, forming an image feature point set. Meanwhile, the calibration board itself is a physically known grid structure, and each corner point has precise physical coordinates in a world coordinate system based on the calibration board itself. These physical coordinates are pre-measured and known quantities. The next task is to correctly match the extracted corner pixel coordinates from the image with the corner physical coordinates in the world coordinate system. This process requires consideration of the calibration board's grid topology. Typically, we start with a specific corner point that is easily identifiable in the image (such as a corner of a checkerboard), and gradually establish the correspondence of the entire grid based on the relative positional relationship of its adjacent corner points, eventually forming a detailed initial feature correspondence table. Each row in the table contains a pair of matching points: image pixel coordinates and corresponding world physical coordinates.

[0065] Based on the established initial feature correspondence table, we can begin calculating the mathematical model describing the projection relationship between the two planes: the homography transformation matrix. The homography transformation matrix is ​​a 3x3 matrix that projects points on the calibration plane in the world coordinate system to the pixel coordinate system of the image. Using the direct linear transformation algorithm, the initial values ​​of this matrix can be solved based on multiple pairs of matching points. The algorithm obtains the elements of the matrix by constructing a homogeneous linear equation system and finding its least-squares solution. However, due to image noise, corner detection errors, or local pattern similarity, the initial feature correspondence table inevitably contains some erroneous matching point pairs. These outliers will severely distort the calculation results of the homography matrix. To eliminate these erroneous matches and improve the robustness and accuracy of the model, a random sample consensus algorithm is required. This algorithm is an iterative robust parameter estimation method. Its core idea is to randomly select the minimum required number of point pairs (at least four pairs are needed for homography matrix calculation) from the set of matched point pairs to compute a temporary model. This temporary model is then used to test all point pairs in the set, calculating the projection error for each pair. Points with errors less than a preset threshold are marked as inliers. This process is repeated numerous times, with each random sampling generating a candidate model and its corresponding set of inliers. The candidate model with the largest set of inliers is selected as the optimal model, and all incorrectly matched pairs marked as outliers are removed.

[0066] After purifying the set of matching point pairs using a random sampling consensus algorithm, the final homography transformation matrix is ​​recalculated using all retained interior points. This calculation typically employs the least squares method, and since the interference from exterior points has been removed, the accuracy of the matrix obtained in this calculation is significantly higher than the initial solution. This optimized homography transformation matrix describes the ideal perspective projection relationship between the calibration plate plane and the distortion-free image plane. However, in practical applications, the original image acquired by the camera is distorted, which we have previously corrected to a distortion-free image using a distortion correction mapping table. Therefore, to establish a complete mapping from the physical world to the original distorted image pixels, the optimized homography transformation matrix needs to be combined with the distortion correction mapping table generated in the first step. This combination is achieved through matrix multiplication, combining the homography transformation with the inverse distortion correction transformation. The result of the multiplication is the initial calibration matrix, which establishes a direct mapping relationship from the physical coordinates of the calibration plate to the pixel coordinates of the original image from the camera. It implicitly contains the lens distortion parameters and the projection geometry information of the plane.

[0067] The generation of the initial calibration matrix is ​​a crucial transitional stage, marking the shift from image feature extraction to mathematical model building. Adaptive thresholding and morphological processing ensure the stability of feature extraction, chain code curvature analysis provides accurate corner location, and the random sampling consensus algorithm ensures the reliability of the mathematical model. The entire process design fully considers issues such as noise, mismatches, and model uncertainties that may be encountered in actual image processing. Through a series of progressively refined operations, the raw image data is transformed into a preliminary but vital set of calibration parameters. Although this initial calibration matrix does not yet distinguish between the camera's internal and external parameters, it provides indispensable initial conditions and constraints for subsequently constructing more complex 3D spatial projection models and optimizing their rotation and translation parameters, enabling the entire calibration process to iterate towards higher accuracy.

[0068] Example 3: After obtaining the initial calibration matrix describing the projection relationship between the calibration plate plane and the image plane, it needs to be decomposed into more physically meaningful camera parameters. The initial calibration matrix is ​​a mathematical representation that integrates multiple transformations. Using matrix decomposition techniques, this composite matrix can be broken down into two independent parts: an intrinsic parameter matrix and an extrinsic parameter matrix. The intrinsic parameter matrix describes the inherent internal optical characteristics of the camera, mainly including the focal length parameter and principal point coordinates. The focal length determines the magnification of the image, and the principal point coordinates usually refer to the intersection of the optical axis and the image plane. The extrinsic parameter matrix describes the position and orientation of the camera in three-dimensional space. It consists of a rotation matrix and a translation vector. The rotation matrix defines the rotational transformation of the world coordinate system relative to the camera coordinate system, and the translation vector indicates the positional offset between the origins of the two coordinate systems. After successfully decomposing the initial calibration matrix, the initial estimates of the intrinsic and extrinsic parameters are obtained. Based on the initial values ​​of the intrinsic and extrinsic parameter matrices obtained from the decomposition, a complete three-dimensional spatial projection model can be constructed. This model aims to describe the complete transformation process from any point in the three-dimensional world coordinate system to the pixel coordinates of the two-dimensional image. The core of the projection model is the perspective projection equation, which projects a 3D point onto a 2D image plane through the optical center of a camera. However, the basic perspective projection equation assumes an ideal pinhole model and does not consider the distortion problems inherent in actual lenses. Therefore, a lens distortion compensation term must be added to the projection equation to more realistically simulate the imaging process of a camera. The distortion model typically includes radial distortion and tangential distortion. Radial distortion causes image points to shift radially, manifesting as barrel or pincushion distortion; tangential distortion is caused by lens manufacturing and installation errors, causing image points to shift tangentially. The complete projection process can be described by a mathematical expression that clearly shows the mapping relationship from the physical world to the pixel image:

[0069]

[0070] Where: symbol It represents two-dimensional coordinates on the pixel plane of an image and is a dimensionless quantity; It represents the coordinates of a three-dimensional point in the world coordinate system, and its dimension is length unit; It is a set of intrinsic parameters that characterize the internal geometry of the camera, including parameters such as focal length; It is a set of extrinsic parameters describing the spatial attitude of the camera, including rotation and translation. It is a set of parameters characterizing lens distortion; mapping function This defines a complete projection operation from a 3D world point to a 2D pixel. The significance of this equation is that both sides represent positions in the 2D image space, and therefore are dimensionless pixel positions.

[0071] After constructing a complete 3D spatial projection model, the next task is to adjust the parameters in the model through optimization algorithms to minimize the difference between the predicted and actual observations. The optimization process mainly targets the rotation and translation components in the extrinsic parameter matrix, as the intrinsic parameter matrix usually does not change much after the camera hardware is fixed and can be considered a relatively stable parameter. The optimization objective is to minimize the reprojection error, which is the Euclidean distance between the image pixel coordinates calculated by the current projection model for corner points with known physical coordinates on the calibration board and the corresponding corner point pixel coordinates detected from the image. For all corner points involved in the calibration, the sum of the squares of all reprojection errors constitutes the objective function of the optimization. Since the projection model is a complex nonlinear function, a nonlinear least squares optimization algorithm is required to solve it. The Levenberg-Marquardt algorithm is very suitable for handling this type of problem. This algorithm is a combination of gradient descent and Gauss-Newton's method. When the parameters are far from the optimal solution, it exhibits gradient descent to ensure stable convergence; when it is close to the optimal solution, it transitions to Gauss-Newton's method to achieve fast convergence.

[0072] The Levenberg-Marquardt algorithm's iterative process begins with initial parameters, which are the extrinsic parameter matrices obtained from the decomposition of the initial calibration matrix. In each iteration, the algorithm calculates the reprojection error under the current parameters and the Jacobian matrix of the error function with respect to each parameter to be optimized. This Jacobian matrix describes the sensitivity of the error to parameter changes. Then, the algorithm solves a system of linear equations to obtain the parameter update step size. The Levenberg-Marquardt algorithm adjusts the update step size by introducing a damping factor. When the error decreases in the current iteration, the damping factor is decreased, the step size is increased, and convergence is accelerated; if the error increases, the damping factor is increased, the step size is decreased, tending towards a more conservative gradient descent. Through this adaptive mechanism, the algorithm can efficiently explore the parameter space. After each iteration, the reprojection error of all corner points is recalculated using the updated parameters, and it is determined whether the convergence condition is met. The convergence condition is typically set as the change in reprojection error being less than a preset minimum threshold over a number of consecutive iterations, or reaching the maximum number of iterations.

[0073] When the optimization process meets the convergence condition, the iteration stops, and the corresponding rotation matrix and translation vector are considered to be the optimal extrinsic parameter matrices. Locking these final parameters means that the 3D spatial projection model has reached the best fit state under the current data, and the model can predict the position of the calibration board corner points in the image with minimal deviation. This optimization process is the core of camera calibration. It tightly combines the information extracted from the 2D image with the geometric constraints of 3D space, and uses mathematical optimization methods to deduce the precise spatial pose of the camera. The optimized projection model is not only effective for the current calibration data, but more importantly, it establishes a universal mapping relationship, enabling the DMS system to map any subsequently acquired 2D image points back to the 3D space of the cockpit for understanding and analysis, such as estimating the driver's head position and gaze direction. The entire optimization process relies on the stability of numerical calculations and the robustness of the algorithm, and can still obtain accurate and reliable calibration results even with certain image noise and measurement errors. The accuracy of the projection model parameters directly determines the accuracy of all subsequent vision-based perception tasks; therefore, the implementation of this step requires rigorous mathematical derivation and precise numerical calculations as support.

[0074] See Figure 4This visualization demonstrates the correspondence between the actual detected positions of the calibration board corner points on the image plane and their theoretical positions calculated using the projection model. The mesh structure shows the projection shape of the calibration board in the image, and the connecting lines represent the projection error vectors of each corner point. This visualization analysis intuitively demonstrates the accuracy of the intrinsic and extrinsic parameter matrices in describing the geometric relationships of camera imaging, while also reflecting the effectiveness of lens distortion correction. By comparing the positional differences between the actual observation points and the theoretical projection points, the quality of the calibration parameters can be evaluated, ensuring that the established projection model can accurately map points in the three-dimensional space of the cockpit to the two-dimensional image coordinate system, laying a solid foundation for subsequent visual perception tasks such as driver status monitoring.

[0075] Example 4: After the 3D spatial projection model is optimized using the Levenberg-Marquardt algorithm and reaches convergence, the parameter matrix in the projection model is stable. The final camera intrinsic and extrinsic parameter matrices need to be extracted from it. Separating the intrinsic parameter matrix from the converged model involves extracting the focal length parameter and principal point coordinates. For a common form of camera intrinsic parameter matrix, the two diagonal elements in the upper left corner typically represent the equivalent focal length values ​​in pixels along the x and y axes of the image. These parameters describe the camera's magnification and field of view. The principal point coordinates are usually represented by the first two elements of the third column of the matrix. They mark the intersection of the camera's optical axis and the image plane, theoretically located at the image center, but may be slightly offset in practice due to sensor installation deviations. Extracting the extrinsic parameter matrix involves combining the rotation matrix and translation vector locked during the optimization process. The rotation matrix is ​​a 3x3 orthogonal matrix that defines the rotation transformation from the world coordinate system to the camera coordinate system, containing complete information about the camera's orientation. The translation vector is a 3x1 vector that indicates the exact position of the world coordinate system origin in the camera coordinate system, usually in millimeters.

[0076] After extracting the initial intrinsic and extrinsic parameters, rigorous verification is necessary to ensure consistent projection across different spatial locations on the calibration board. This verification process requires selecting a series of checkpoints not involved in model optimization calculations. These checkpoints can be other corner points on the same calibration board that were not used for optimization, or corner points from another set of calibration board images taken at different positions and orientations. Using the extracted intrinsic and extrinsic parameter matrices, the 3D world coordinates of these checkpoints are reprojected onto the 2D image pixel coordinate system, resulting in a set of theoretically calculated pixel coordinates. Simultaneously, image processing algorithms are used to accurately detect the actual pixel coordinates corresponding to these checkpoints in the actual captured images. The Euclidean distance between each pair of theoretical projected coordinates and actual detected coordinates is calculated; this distance represents the reprojection error of that point. Statistical analysis of the reprojection errors of all checkpoints allows for the evaluation of the accuracy and robustness of the calibration parameters. Table 1 shows a simplified verification result, listing the coordinate comparison and error calculation for five checkpoints.

[0077] Table 1: Analysis of Reprojection Error at Calibration Parameter Verification Points

[0078] Verification point number World coordinates (mm) Actual detected pixel coordinates Model projection pixel coordinates Reprojection error (pixels) V01 (125.0,85.5,0.0) (512.3,387.6) (512.8,387.9) 0.54 V02 (-63.2,101.7,0.0) (288.1,365.2) (287.7,365.0) 0.45 V03 (40.5,-55.8,0.0) (402.5,452.8) (402.0,453.4) 0.67 V04 (-97.3,-72.1,0.0) (265.4,470.1) (264.8,469.5) 0.85 V05 (150.0,0.0,0.0) (535.6,400.5) (535.9,400.1) 0.45

[0079] When the reprojection error of all verification points is within an acceptable threshold range, it indicates that the extracted intrinsic and extrinsic parameter matrices are accurate and reliable. The next step is to persistently store these parameters in hardware. The first step is to serialize the verified intrinsic and extrinsic parameter matrices into a binary data stream. The serialization process requires converting each floating-point parameter in the matrix into a byte sequence according to a predefined, compact format. A typical serialization format includes a short header, which may contain metadata such as data version, matrix dimensions, and data type to facilitate correct parsing later. Following the header are the element values ​​of the intrinsic and extrinsic parameter matrices, arranged sequentially, for example, in row-major order. This binary format avoids the parsing complexity and storage waste associated with text formats, making it well-suited for embedded systems.

[0080] The generated binary data stream is transmitted to the DMS camera controller via the vehicle-mounted CAN bus protocol. CAN bus is a reliable and real-time communication network widely used in the automotive industry. Data transmission must adhere to vehicle network communication specifications, such as configuring message identifiers using predefined parameters and segmenting the binary data into multiple small data packets that conform to the CAN bus data frame length limit (typically a maximum of 8 bytes of payload). The communication protocol needs to include mechanisms such as frame sequence numbers and checksums to ensure the integrity and correctness of the data during transmission. At the receiving end, the controller needs to reassemble the data packets according to the protocol and verify their correctness. In the firmware storage area of ​​the DMS camera, a dedicated non-volatile storage area needs to be pre-allocated to store calibration parameters. This area may be located in a specific sector of the Flash memory. After receiving the complete binary data stream, the controller writes the data to this dedicated storage block through the programming interface. The non-volatile nature of the storage ensures that even if the vehicle is powered off, the calibration parameters will not be lost and can be read and loaded again when the system is powered on next time, thus ensuring that the DMS camera can operate based on accurate calibration parameters throughout its entire lifespan. The entire process, from parameter extraction and verification to serialization, transmission, and final storage, forms a complete closed loop, ensuring that the calibration results can be reliably applied to actual products.

[0081] Example 5: Focusing on the parameter stability maintenance of the DMS system during long-term use, its core lies in establishing a proactive and responsive calibration parameter verification and compensation mechanism. This mechanism begins with continuous monitoring of the physical state of the camera mounting bracket. By integrating a microelectromechanical system (MEMS) accelerometer sensor onto the bracket or camera body, vibration acceleration data in three axes can be collected at millisecond intervals. The raw data collected by the sensor is low-pass filtered to eliminate high-frequency noise, and then converted into vibration displacement amplitude through numerical integration. The system continuously calculates the root mean square (RMS) value of the vibration displacement within a time window, using it as a quantitative indicator of the current vibration intensity. When the RMS value of the vibration amplitude continuously exceeds a preset threshold, for example, reaching above 50 micrometers and remaining there for more than 100 milliseconds, the system determines that the camera may have experienced pose shift due to bumpy roads, vehicle collisions, or component aging, and automatically triggers the calibration parameter verification process. This process requires no manual intervention, ensuring the system's rapid response to potential accuracy degradation.

[0082] After the calibration process begins, the system first reads the previously calibrated intrinsic and extrinsic parameter matrices from the firmware non-volatile storage area of ​​the DMS camera via the vehicle network, and loads these parameters into the 3D spatial projection model running on the processor. Next, the system controls the camera to acquire images of a known fixed reference object within the cockpit. This reference object can be a small, high-contrast pattern intentionally kept within the field of view, whose precise 3D position was determined during initial calibration. Using the loaded calibration parameters, the projection model calculates the theoretical projected pixel position of the reference object's 3D coordinates. Simultaneously, the image processing algorithm detects the actual pixel center position of the reference object from the currently acquired real-time image. The Euclidean distance between the theoretical projected position and the actual detected position is calculated; this distance value is defined as the image quality degradation index for the current frame. To improve reliability, the system typically acquires multiple frames continuously and calculates an average degradation index to eliminate potential noise interference in single-frame images.

[0083] Based on the calculated average degradation index, the system enters a dynamic adjustment phase. If the degradation index is within a low tolerance range, such as less than 2 pixels, the system considers the current calibration parameters still valid, records only the vibration event and verification results, and does not modify the parameters. If the degradation index significantly exceeds the tolerance, it indicates that the actual extrinsic parameters of the camera have undergone observable deviation. The adjustment logic does not directly replace the original extrinsic parameter matrix, but calculates a compensation value. This compensation value mainly acts on the translation vector component in the extrinsic parameter matrix. The system generates a small translation compensation vector based on the direction and magnitude of the degradation index. For example, if the actual detected reference position is consistently biased to the right of the theoretical position, the compensation algorithm will calculate a leftward translation, attempting to pull the projection model back to a state consistent with the actual situation. This compensation value is used as an increment and temporarily superimposed on the original extrinsic parameter matrix loaded from memory, forming a set of effective extrinsic parameters applicable to the current physical state.

[0084] This dynamic adjustment is adaptive, with the compensation value exhibiting a non-linear relationship with the degradation index. Initial changes are rapid for quick convergence, while changes slow down as the target is approached to prevent overshoot. The adjusted parameters are applied to all subsequent visual perception algorithms, such as driver gaze estimation and head posture tracking, to maintain the accuracy of these functions. Notably, this adjustment is typically temporary or marked as requiring manual review. The system records vibration events, degradation indices, and applied compensation amounts; these logs allow maintenance personnel to analyze the long-term stability of the camera mount. The entire implementation process constructs a closed-loop system from sensing physical vibration to quantifying image quality degradation and then to intelligent parameter compensation. This allows the DMS system to move beyond one-off factory calibration and maintain measurement accuracy in real, dynamic in-vehicle environments, thereby improving the system's robustness and reliability throughout its product lifecycle.

[0085] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0086] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A computer vision-based DMS camera calibration method, characterized in that, The method comprises the following steps: Collecting a multi-view original image sequence in the target vehicle cabin, which contains camera installation position, view angle inclination and ambient light intensity parameters; Performing distortion correction on the original image sequence, extracting the edge features of the calibration reference in each frame of image through image recognition technology, and generating an initial calibration matrix containing the mapping relationship between pixel coordinates and physical coordinates; Constructing a three-dimensional space projection model according to the initial calibration matrix, adjusting the rotation and translation parameters in the projection model through an iterative optimization algorithm, so that the virtual imaging position of the calibration reference in the projection model and the edge features in the actual image have minimized error; Extracting the intrinsic matrix and extrinsic matrix in the optimized projection model, fusing them into the final calibration parameters, and writing the final calibration parameters into the firmware storage area of the DMS camera; The construction of the three-dimensional space projection model comprises: Decomposing the initial calibration matrix into an intrinsic matrix of camera focal length and principal point offset and an extrinsic matrix of coordinate system conversion; Establishing a projection equation of the cabin world coordinate system and the camera imaging plane according to the extrinsic matrix; Adding a lens distortion compensation term in the projection equation to form a complete three-dimensional space projection model.

2. The computer vision based DMS camera calibration method of claim 1, wherein, The collection of the multi-view original image sequence in the target vehicle cabin comprises: Arranging a checkerboard calibration board with known geometric dimensions in the cabin, and controlling the DMS camera to shoot calibration board images at different pitch angles; Synchronously recording the exposure time, ambient light sensor data and camera mechanical posture data of each frame of image; Performing timestamp alignment processing on the continuous frame images to form an original image sequence containing spatio-temporal correlation.

3. The computer vision based DMS camera calibration method of claim 2, wherein, The distortion correction processing on the original image sequence comprises: Detecting the sub-pixel coordinates of the calibration board corner points in each frame of image, and establishing a distortion correction mapping table according to the radial distortion coefficient and tangential distortion coefficient of the camera lens; Using a bilinear interpolation algorithm to correct the original image sequence pixel by pixel to generate a non-distorted image sequence.

4. The computer vision based DMS camera calibration method of claim 3, wherein, The extraction of the edge features of the calibration reference in each frame of image through image recognition technology comprises: Performing adaptive threshold segmentation on the non-distorted image sequence to extract the binary profile of the calibration board region; Using morphological gradient operation to enhance the profile edge, and calculating the curvature extreme points of the profile chain code as feature points; Matching the pixel coordinates of the feature points with the actual physical coordinates of the calibration board one by one to form an initial feature correspondence table.

5. The computer vision based DMS camera calibration method of claim 4, wherein, The generation of the initial calibration matrix containing the mapping relationship between pixel coordinates and physical coordinates comprises: Calculating the initial solution of the homography transformation matrix according to the matching point pairs in the initial feature correspondence table; Optimizing the accuracy of the homography transformation matrix by eliminating the mis-matching point pairs through a random sample consensus algorithm; Multiplying the optimized homography transformation matrix with the distortion correction mapping table to obtain the initial calibration matrix.

6. The computer vision based DMS camera calibration method of claim 1, wherein, The adjustment of the rotation and translation parameters in the projection model through the iterative optimization algorithm comprises: Calculating the re-projection error of the calibration board feature points in the projection model, and iteratively optimizing the extrinsic matrix through the Levenberg-Marquardt algorithm; Terminating the optimization when the change amount of the re-projection error of the continuous three iterations is less than a threshold value, and locking the final rotation and translation parameters.

7. The computer vision-based DMS camera calibration method of claim 6, wherein, The extraction of the intrinsic matrix and extrinsic matrix in the optimized projection model comprises: Separate the focal length parameter and principal point coordinates of the intrinsic matrix from the converged three-dimensional space projection model; Combine the optimized rotation matrix and translation vector into the extrinsic matrix; Verify the projection consistency of the intrinsic matrix and extrinsic matrix at different positions of the calibration board.

8. The computer vision-based DMS camera calibration method of claim 7, wherein, The writing of the final calibration parameters into the firmware storage area of the DMS camera includes: Serializing the intrinsic matrix and extrinsic matrix into a binary data stream; Transmitting the binary data stream to the DMS camera controller through the vehicle-mounted CAN bus protocol; Dividing the calibration parameter dedicated storage block in the firmware storage area to achieve power-off persistent storage.

9. The computer vision based DMS camera calibration method of claim 8, wherein, Also includes: Real-time monitoring of the vibration amplitude of the DMS camera mounting bracket, triggering the calibration parameter verification process when the vibration amplitude exceeds the threshold; Read the calibration parameters in the firmware storage area and load them into the three-dimensional space projection model to calculate the current imaging quality degradation index; According to the degradation index, dynamically adjust the extrinsic matrix offset compensation value in the calibration parameters.

Images