Automatic Calibration Method and System for Ultra-High Definition 8K Dual-Light Fusion System

Abstract

The present invention provides an automatic calibration method and system for an ultra-high-definition 8K dual-light fusion system, which relates to the technical field of image processing and is applied to a dual-light fusion system including a visible light camera and an infrared camera. The method includes periodically reading the real-time ambient light intensity and dynamically adjusting the exposure parameters of the visible light camera and the infrared camera in combination with a preset parameter table; obtaining a visible light image and an infrared image based on the exposure parameters and performing feature point matching to obtain a feature point matching result; allocating image weights according to the real-time ambient light intensity and performing multi-modal fusion calibration in combination with the feature point matching result and the initial parameters based on single-camera calibration to obtain preliminary calibration parameters; using a non-linear optimization algorithm to globally optimize the obtained preliminary calibration parameters, outputting the final calibration parameters and completing the calibration. The present invention realizes the calibration of the dual-light fusion system under different lighting conditions.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to an automatic calibration method and system for an ultra-high-definition 8K dual-light fusion system. Background Art

[0002] With the rapid development of ultra-high-definition video technology, video content with 8K resolution is gradually becoming mainstream, showing great application potential in many fields such as security monitoring, autonomous driving, intelligent healthcare, and aerospace. Under complex and changing environmental conditions, a single-modal image sensor often fails to meet the visual perception requirements for all-weather and full-scene scenarios. For example, in low-light or completely dark environments, visible-light cameras are difficult to capture effective information, while infrared cameras can provide key image data by virtue of their thermal imaging characteristics; conversely, in strong-light or uniformly illuminated environments, infrared images may be noisy due to thermal radiation interference, and at this time, visible-light images are clearer and more reliable. Therefore, a dual-light fusion system that combines a visible-light camera and an infrared camera has emerged, aiming to enhance the environmental adaptability and target recognition ability of the system by integrating the advantages of the two modal images.

[0003] However, the performance of the dual-light fusion system highly depends on the precise calibration between the two cameras, including the accurate determination of internal parameters (such as focal length, principal point coordinates) and external parameters (such as rotation matrix, translation vector). Traditional calibration methods mostly adopt an offline method and are carried out under specific lighting conditions, making it difficult to adapt to the dynamic changes of the lighting environment in practical applications. In addition, under different lighting conditions, the imaging characteristics of the camera (such as noise level, contrast) will change significantly. If fixed calibration parameters are still used, it will lead to quality problems such as ghosting and blurring in the fused image, seriously affecting the reliability and stability of the system.

[0004] Therefore, it is necessary to provide an automatic calibration method and system for an ultra-high-definition 8K dual-light fusion system to solve the above technical problems. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides an automatic calibration method and system for an ultra-high-definition 8K dual-light fusion system, which combines the imaging characteristics of a visible-light camera and an infrared camera, and realizes the calibration of the dual-light fusion system under different lighting conditions through dynamically adjusting exposure parameters, feature point matching, multi-modal fusion calibration, and non-linear optimization.

[0006] The present invention provides an automatic calibration method for an ultra-high-definition 8K dual-light fusion system, which is applied to a dual-light fusion system including a visible-light camera and an infrared camera. The method includes the following steps:

[0007] Periodically read the real-time ambient light intensity, and dynamically adjust the exposure parameters of the visible light camera and the infrared camera in combination with a preset parameter table, where the preset parameter table represents the optimal exposure parameter combinations of the visible light camera and the infrared camera in different light intensity intervals;

[0008] Obtain a visible light image and an infrared image based on the exposure parameters, and perform feature point matching on the visible light image and the infrared image to obtain a feature point matching result;

[0009] Allocate image weights according to the real-time ambient light intensity, and perform multi-modal fusion calibration in combination with the feature point matching result and the initial parameters based on single camera calibration to obtain preliminary calibration parameters;

[0010] Define an error correction factor based on the real-time ambient light intensity, aim to minimize the weighted reprojection error, use a non-linear optimization algorithm to globally optimize the preliminary calibration parameters, output the final calibration parameters and complete the calibration.

[0011] Preferably, the exposure parameters include exposure duration and gain value.

[0012] Preferably, the dynamic adjustment of the exposure parameters includes the following steps:

[0013] Determine the light intensity interval to which it belongs according to the currently read real-time ambient light intensity;

[0014] Match the optimal exposure parameter combinations of the visible light camera and the infrared camera corresponding to the light intensity interval to which it belongs from the preset parameter table;

[0015] Synchronously adjust the exposure parameters of the visible light camera and the infrared camera to the optimal exposure parameter combination.

[0016] Preferably, the dynamic adjustment of the exposure parameters further includes:

[0017] If the real-time ambient light intensity is in the transition region between two adjacent light intensity intervals, use the linear interpolation method to calculate the optimal exposure parameter combination according to the data in the preset parameter table.

[0018] Preferably, the obtaining of the visible light image and the infrared image based on the exposure parameters, and the performing of feature point matching on the visible light image and the infrared image to obtain a feature point matching result includes the following steps:

[0019] Obtain the visible light image and the infrared image captured by the visible light camera and the infrared camera respectively according to the dynamically adjusted optimal exposure parameter combination, and perform preprocessing;

[0020] Extract robust feature points from visible light images and infrared images using a pre-trained deep learning model, and based on the extracted robust feature points, where the robust feature points include visible light feature points and infrared feature points;

[0021] Perform bidirectional matching on the extracted visible light feature points and infrared feature points, and filter out feature point pairs with matching consistency higher than a preset threshold through a geometric constraint algorithm to obtain the feature point matching result.

[0022] Preferably, the process of obtaining the preliminary calibration parameters includes the following steps:

[0023] Define a weight function based on the deviation degree between the real-time ambient light intensity and the preset ideal illumination intensity, where the deviation degree is inversely proportional to the image weight;

[0024] Assign weight values to each pair of visible light images and infrared images according to the weight function;

[0025] Calculate the initial internal parameter matrix and the initial external parameter matrix of the visible light camera and the infrared camera respectively based on the single camera calibration algorithm;

[0026] Calculate the relative pose relationship between the visible light camera and the infrared camera according to the feature point matching result, including the rotation matrix and the translation vector;

[0027] Using the weight value as the optimization weight, combining the initial internal parameter matrix, the initial external parameter matrix and the relative pose relationship, perform weighted iterative optimization on the multi-modal fusion calibration parameters through non-linear least squares method to generate preliminary calibration parameters.

[0028] Preferably, the expression of the weight function is:

[0029]

[0030] Where, is the real-time ambient light intensity, is the preset ideal illumination intensity, is the adjustment coefficient for controlling the smoothness of the weight function, and , is the real-time ambient light intensity under the weight value, is the base of the exponential function.

[0031] Preferably, the process of outputting the final calibration parameters includes the following steps:

[0032] Define an error correction factor based on the deviation degree between the real-time ambient light intensity and the preset ideal illumination intensity;

[0033] Multiply the error correction factor by the assigned weight value to obtain the comprehensive weight;

[0034] Weight the reprojection error of each pair of visible light images and infrared images based on the comprehensive weight to construct a global optimization objective function;

[0035] Based on the global optimization objective function, use the LM algorithm to iteratively optimize the preliminary calibration parameters. When the weighted reprojection error meets the preset termination condition, terminate the optimization and output the final calibration parameters, where the final calibration parameters include the optimized internal parameters, external parameters, and relative pose parameters of the visible light camera and the infrared camera.

[0036] The present invention also provides an automatic calibration system for an ultra-high-definition 8K dual-light fusion system, which is applied to a dual-light fusion system including a visible light camera and an infrared camera and is used to execute the automatic calibration method for an ultra-high-definition 8K dual-light fusion system. The system includes:

[0037] An exposure parameter adjustment module for periodically reading the real-time ambient light intensity and dynamically adjusting the exposure parameters of the visible light camera and the infrared camera in combination with a preset parameter table, where the preset parameter table represents the optimal exposure parameter combinations of the visible light camera and the infrared camera in different light intensity intervals;

[0038] A feature point matching module for obtaining visible light images and infrared images based on the exposure parameters and performing feature point matching on the visible light images and the infrared images to obtain a feature point matching result;

[0039] A preliminary calibration module for allocating image weights according to the real-time ambient light intensity and performing multi-modal fusion calibration in combination with the feature point matching result and the initial parameters based on single-camera calibration to obtain preliminary calibration parameters;

[0040] A global optimization module for defining an error correction factor based on the real-time ambient light intensity, aiming to minimize the weighted reprojection error, and using a non-linear optimization algorithm to globally optimize the preliminary calibration parameters, output the final calibration parameters, and complete the calibration.

[0041] Compared with the related technology, the automatic calibration method and system for an ultra-high-definition 8K dual-light fusion system provided by the present invention have the following beneficial effects:

[0042] The automatic calibration method for the ultra-high-definition 8K dual-light fusion system proposed by the present invention can accurately and dynamically adjust the exposure parameters of the visible light camera and the infrared camera according to the real-time ambient light intensity, ensuring high-quality images are obtained under various lighting conditions. By innovatively combining multi-modal fusion calibration and non-linear optimization algorithms, not only the calibration accuracy is greatly improved, but also the system's excellent adaptability and stability to complex and changing lighting environments are enhanced. Description of the Drawings

[0043] Figure 1 Flow chart of an automatic calibration method for an ultra-high definition 8K dual-light fusion system provided by the present invention;

[0044] Figure 2 Module structure diagram of an automatic calibration system for an ultra-high definition 8K dual-light fusion system provided by the present invention. Detailed implementation manners

[0045] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present invention, rather than limiting the present invention. Additionally, it should be noted that for the sake of description, only parts related to the present invention are shown in the accompanying drawings rather than all structures. Furthermore, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0046] It should also be noted that for the sake of description, only parts related to the present invention are shown in the accompanying drawings rather than all content. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the operations can be rearranged. When the operations are completed, the process can be terminated, but there can also be additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0047] Embodiment 1

[0048] The present invention provides an automatic calibration method for an ultra-high definition 8K dual-light fusion system, which is applied to a dual-light fusion system including a visible light camera and an infrared camera. Referring to Figure 1 as shown, the method includes the following steps:

[0049] S1: Periodically read the real-time ambient light intensity, and dynamically adjust the exposure parameters of the visible light camera and the infrared camera in combination with a preset parameter table, where the preset parameter table represents the optimal exposure parameter combinations of the visible light camera and the infrared camera in different light intensity intervals.

[0050] Before step S1, it is necessary to construct a preset parameter table. Specifically: in a controllable lighting laboratory, use a standard light source to simulate different lighting intensity environments, and record the ambient light intensity through a high-precision light sensor. For each lighting intensity interval, collect visible light images and infrared images, determine the optimal exposure parameter combination according to the image quality evaluation index, and store the lighting intensity interval and the corresponding exposure parameter combination as a preset parameter table, where the exposure parameter combination includes the exposure duration and the gain value.

[0051] Specifically, the dynamic adjustment of the exposure parameters includes the following:

[0052] When the ambient light intensity is read in real time, data is collected through the light sensor at a preset frequency, and the data is subjected to moving average filtering to suppress instantaneous noise; according to the filtered light intensity value, match the corresponding interval in the preset parameter table: if the light intensity completely falls into a certain interval, directly adopt the exposure parameters corresponding to this interval as the optimal exposure parameters; if it is in the transition area between adjacent intervals, calculate the smoothly transitioning exposure parameters through linear interpolation.

[0053] When adjusting the exposure parameters, send an exposure parameter update command including the exposure duration and the gain value to the visible light and infrared cameras through the camera control interface, and adopt a progressive transition strategy for parameter switching to avoid screen flickering.

[0054] S2: Obtain visible light images and infrared images based on the exposure parameters, and perform feature point matching on the visible light images and infrared images to obtain the feature point matching result.

[0055] Specifically, step S2 specifically includes the following steps:

[0056] S21: Obtain the visible light images and infrared images captured by the visible light camera and the infrared camera respectively according to the dynamically adjusted optimal exposure parameter combination, and perform preprocessing.

[0057] In this embodiment, the visible light camera and the infrared camera capture images synchronously according to the dynamically adjusted optimal exposure parameters. After the visible light image is acquired, contrast-limited adaptive histogram equalization (CLAHE) is first performed, with the block size set to 8×8 and the contrast limit to 2.0, to enhance the details in the dark areas and suppress the overexposed areas. Subsequently, adaptive gamma correction is adopted to dynamically adjust the gamma value (range 0.5~1.5) according to the local brightness of the image to compensate for uneven illumination. For the infrared image, non-uniformity correction is performed: based on the two-point correction method, a blackbody radiation source (temperatures 25°C and 60°C) is used to calibrate the gain and offset parameters of each pixel to eliminate the sensor response differences. The corrected infrared image is smoothed of noise by Gaussian filtering (kernel size 5×5, standard deviation 1.5), and histogram stretching is adopted to compress the pixel value range from the original 12 bits to 8 bits to improve visibility. Finally, geometric alignment is performed on the visible light and infrared images: based on the corner coordinates of the calibration board, affine transformation is applied to register the infrared image to the visible light image coordinate system to ensure that the fields of view of the two are consistent.

[0058] S22: Use a pre-trained deep learning model to extract robust feature points from the visible light image and the infrared image, and based on the extracted robust feature points, where the robust feature points include visible light feature points and infrared feature points.

[0059] In this embodiment, a pre-trained two-stream feature extraction network, that is, a deep learning model, has the following network structure:

[0060] Input layer: The size of the visible light image is 3840×2160, and the infrared image is 640×512, both normalized to the range [-1, 1].

[0061] Backbone network: An improved ResNet-50, with a cross-modal attention module inserted after Stage3, generating spatial attention weights through the visible light feature map to guide the infrared branch to focus on the texture-rich areas.

[0062] Output layer: Each branch outputs a 256-dimensional feature descriptor and the coordinates of the feature points (2000 points are extracted from the visible light image, and 500 points are extracted from the infrared image).

[0063] Training details:

[0064] Dataset: FLIR ADAS dataset (including 100,000 pairs of visible light-infrared aligned image pairs).

[0065] Loss function: Triplet Loss, with the margin set to 0.5 and the hard example mining ratio 30%.

[0066] Data augmentation: Random illumination change (±20% brightness), Gaussian noise ( = 0.05) and simulated lens blur (kernel size 3×3).

[0067] During the inference stage, for visible light images, non-maximum suppression (NMS) is applied at the feature point coordinates with a neighborhood radius of 5 pixels, and the top 1000 points with the highest confidence are retained; for infrared images, the feature response in low-texture regions is enhanced based on the thermal radiation gradient, and regions with a gradient value greater than 10 °C / pixel are screened for dense sampling.

[0068] S23: Bidirectionally match the extracted visible light feature points and infrared feature points, and filter out the feature point pairs with a matching consistency higher than a preset threshold through a geometric constraint algorithm to obtain the feature point matching result.

[0069] In this embodiment, during the initial matching, calculate the cosine similarity matrix of the visible light feature point and infrared feature point descriptors, with a dimension of 1000×500; perform bidirectional nearest neighbor matching, for each visible light feature point, find the infrared feature point with the highest similarity; perform the same operation on the infrared feature points in reverse; retain the pairs that match bidirectionally (i.e., A→B and B→A) to obtain the initial matching pairs.

[0070] During the geometric consistency verification: Use the RANSAC algorithm to estimate the fundamental matrix, set the maximum number of iterations to 1000, and the reprojection error threshold to 1.0 pixel; calculate the Sampson distance of all matching pairs, retain the inliers with a distance less than 0.8 pixel, and count the inlier ratio. If it is lower than 80%, it is determined as an invalid matching set and re-matching is triggered; calculate the final matching confidence based on the feature point descriptor similarity (normalized to 0 to 1) and the inlier status; retain the matching pairs with a confidence ≥0.8 and output them as the feature point matching result, including the image coordinates, descriptors, and confidence scores of the matching point pairs.

[0071] S3: Allocate image weights according to the real-time ambient light intensity, and perform multi-modal fusion calibration in combination with the feature point matching result and the initial parameters based on single camera calibration to obtain the preliminary calibration parameters.

[0072] Specifically, the process of obtaining the preliminary calibration parameters includes the following steps:

[0073] Define a weight function based on the deviation degree between the real-time ambient light intensity and the preset ideal illumination intensity, where the deviation degree is inversely proportional to the image weight;

[0074] Among them, the expression of the weight function is:

[0075]

[0076] Among them, is the real-time ambient light intensity, is the preset ideal illumination intensity, is the adjustment coefficient for controlling the smoothness of the weight function, and , is the real-time ambient light intensity at which the weight value is calculated, is the base of the exponential function.

[0077] In this embodiment, the method for calibrating parameters in the laboratory is as follows:

[0078] Collect multiple sets of dual-light image data at different real-time ambient light intensities in a controllable lighting environment.

[0079] Determine the optimal weight value at each light intensity through manual evaluation or automated metrics (such as the correct rate of feature point matching, image signal-to-noise ratio).

[0080] Use the non-linear least squares method to fit the values so that the function curve matches the experimental data (for example = 100).

[0081] Technical effect: When is close to , is approximately equal to 0.5, assigning a high weight; when deviates from , exponentially decays to suppress the influence of low-quality data.

[0082] Assign weight values to each pair of visible light images and infrared images according to the weight function.

[0083] In this embodiment, according to the current ambient light intensity , substitute it into the weight function to calculate the weight value of each pair of visible light-infrared images.

[0084] If multiple sets of image data (such as consecutive frames) are processed simultaneously, normalize the weight values to the interval [0.1, 1.0] to avoid interference from extreme values in the optimization process.

[0085] Assign an initial weight to each feature point pair and multiply it by the matching confidence (output by step S23) to form the weight value.

[0086] Based on the single-camera calibration algorithm, calculate the initial internal parameter matrix and initial external parameter matrix of the visible light camera and the infrared camera respectively.

[0087] In the single-camera calibration stage, first, a standard checkerboard calibration board needs to be prepared, and the physical sizes of its black and white squares should be precisely set. Place the calibration board within the field of view of both the visible-light camera and the infrared camera, and position it in multiple different angles and positions (at least 15 different poses) to ensure that the calibration board covers the central and edge regions of the image. Take the corresponding visible-light images and infrared images for each pose for subsequent calculations.

[0088] Independently calibrate each camera using the Zhang Zhengyou calibration method. In the specific process, utilize the corner coordinates of the calibration board in the image to calculate the internal parameters of the camera, including the focal length parameter (describing the light-focusing ability of the lens), the principal point offset parameter (indicating the position of the lens optical axis on the image plane), and the distortion coefficients for correcting image deformation. During the calibration process, it is necessary to ensure that the calculated reprojection error (i.e., the average deviation between the actual position and the theoretical projection position of the calibration board corners in the image) does not exceed 0.5 pixels to guarantee the calibration accuracy.

[0089] In addition, it is also necessary to calculate the external parameters of the camera, that is, the rotation angles and translation distances of the camera relative to the calibration board in different poses. By solving the relationship between the three-dimensional spatial position of the calibration board and its two-dimensional projection in the image, the spatial pose of the camera at each shooting is obtained, and these parameters provide an initial reference for subsequent multi-modal fusion calibration.

[0090] According to the feature point matching results, calculate the relative pose relationship between the visible-light camera and the infrared camera, including the rotation matrix and the translation vector.

[0091] In this embodiment, when calculating the relative pose relationship between the visible-light and infrared cameras, first, based on the feature point matching results obtained in step S2, at least 50 pairs of high-confidence matching points are selected. Exclude the mismatched points through robust estimation algorithms (including but not limited to RANSAC), and estimate the fundamental geometric relationship matrix between the two cameras. This matrix can describe the projection constraint conditions between the perspectives of the two cameras.

[0092] Subsequently, convert the fundamental geometric relationship matrix into an essential matrix describing the actual pose relationship between the cameras. Through mathematical decomposition of the essential matrix, possible rotation angle combinations and translation directions are deduced, and these combinations need to be verified by three-dimensional space points. Specifically, use the two-dimensional coordinates of the matching point pairs to reverse-infer their three-dimensional spatial positions, calculate the projection errors under different rotation and translation combinations, and finally select the combination with the minimum error as the precise relative pose parameters between the cameras.

[0093] During this process, if it is found that the projection error exceeds the preset threshold (for example, more than 1 pixel), it is necessary to re-screen the matching points or adjust the parameter decomposition method until a relative pose relationship that meets the accuracy requirements is obtained. These parameters will be used for subsequent fusion calibration to ensure the precise alignment of visible light and infrared images in space.

[0094] Using the weight value as the optimization weight, combining the initial internal parameter matrix, the initial external parameter matrix, and the relative pose relationship, the multi-modal fusion calibration parameters are weighted and iteratively optimized through the non-linear least squares method to generate preliminary calibration parameters.

[0095] In this embodiment, in the weighted iterative optimization stage, the initial parameters obtained from single-camera calibration, the relative pose relationship, and the weight value dynamically allocated based on the lighting conditions are combined, and preliminary calibration parameters are generated through a non-linear optimization algorithm. During the optimization process, the goal is to minimize the weighted projection error of all feature point pairs, that is, to adjust the camera parameters so that the actual position of the feature points in the image is as close as possible to the theoretical projection position, while considering the difference in data credibility under different lighting conditions.

[0096] First, define the set of parameters to be optimized, including the internal parameters (such as focal length, principal point offset, distortion coefficient) of visible light and infrared cameras, the external parameters (such as rotation angle, translation vector), and the relative pose relationship between the two. These parameters jointly affect the projection result of three-dimensional space points to two-dimensional images through a mathematical model.

[0097] When constructing the objective function, calculate the projection error of each feature point matching pair in the visible light and infrared images, and multiply the error by the corresponding comprehensive weight value (determined jointly by the lighting weight and the feature point confidence). For example, high-quality image data collected under ideal lighting conditions will be given a higher weight, while the influence of data with lighting deviating from the ideal value or low matching confidence on the optimization will be weakened. By accumulating all weighted error values to form a global optimization objective, ensure that the optimization process pays more attention to high-reliability data.

[0098] The non-linear least squares optimization algorithm (Levenberg-Marquardt algorithm) is used for iterative solution. In each iteration, the algorithm calculates the objective function value based on the current parameters, and adjusts the parameters in the gradient descent direction to reduce the error. Set the convergence condition during the optimization process: terminate the optimization when the change rate of the weighted error between two adjacent iterations is less than 1% or reach the maximum number of iterations (such as 50 times). If parameter oscillation or error non-convergence is detected during the optimization process, trigger an exception handling mechanism, such as reverting to historical parameters or re-initializing the optimization variables.

[0099] After the optimization is completed, output the preliminary calibration parameters, including the optimized internal parameters, external parameters of the camera, and the relative pose relationship.

[0100] S4: Define an error correction factor based on the real-time ambient light intensity. With the goal of minimizing the weighted reprojection error, use a non-linear optimization algorithm to globally optimize the preliminary calibration parameters, output the final calibration parameters, and complete the calibration.

[0101] Specifically, the output process of the final calibration parameters includes the following steps:

[0102] Define an error correction factor based on the deviation degree between the real-time ambient light intensity and the preset ideal light intensity.

[0103] In this embodiment, the error correction factor is defined as:

[0104]

[0105] represents the real-time ambient light intensity, is the preset ideal light intensity, and are positive coefficients calibrated through experiments.

[0106] Define an error correction factor based on the deviation degree between the real-time ambient light intensity and the preset ideal light intensity. The error correction factor adopts the form of an exponential function, and its parameters and are determined through experimental calibration: In a controllable lighting laboratory, simulate scenarios with different deviation degrees by adjusting the light source intensity, collect multiple sets of dual-light image data, calculate the attenuation trend of the calibration error in each scenario, and use non-linear least squares to fit and values so that the error correction factor can effectively suppress the error contribution of data under non-ideal lighting conditions.

[0107] Multiply the error correction factor by the assigned weight value to obtain the comprehensive weight.

[0108] In this embodiment, the comprehensive weight is defined as:

[0109]

[0110] The role of the comprehensive weight is to dynamically adjust the credibility weight of data under different lighting conditions. For example, when the lighting deviates significantly from the ideal value, the comprehensive weight is significantly reduced to reduce the negative impact of low-quality data on global optimization. The specific calculation method of the comprehensive weight is: for each pair of visible-infrared images, calculate the weighted value according to its real-time light intensity and the confidence of feature point matching, ensuring that data with high confidence and lighting conditions close to the ideal dominate in the optimization.

[0111] Weight the reprojection error of each pair of visible light images and infrared images based on the comprehensive weight, and construct a global optimization objective function.

[0112] In this embodiment, the global optimization objective function is defined as:

[0113]

[0114] Where is the projection function, and are the rotation matrix and translation vector to be optimized, is a three-dimensional space point, is an image feature point.

[0115] When constructing the global optimization objective function, the goal is to minimize the weighted reprojection error. Specifically, calculate the Euclidean distance between the projected position and the actual detected position of each feature point pair in the visible light and infrared images, multiply this distance by the corresponding comprehensive weight, and accumulate all the weighted distances as the optimization objective. During the projection process, consider the influence of the internal parameters (such as focal length, distortion coefficient) and external parameters (such as rotation matrix, translation vector) of the camera and the relative pose parameters on the projection result. For example, after the three-dimensional space point is transformed to the camera coordinate system through rotation and translation, it is projected onto the image plane in combination with the focal length and principal point offset parameters, and the distortion coefficient is used to correct the non-linear deformation error.

[0116] Based on the global optimization objective function, use the LM algorithm to iteratively optimize the preliminary calibration parameters. When the weighted reprojection error meets the preset termination condition, terminate the optimization and output the final calibration parameters. Among them, the final calibration parameters include the optimized internal parameters, external parameters and relative pose parameters of the visible light camera and the infrared camera.

[0117] In this embodiment, the Levenberg-Marquardt (LM) algorithm is used to perform iterative optimization. The initial parameters are the preliminary calibration parameters output in step S3, including the internal parameters, external parameters and relative pose of the visible light and infrared cameras. During the optimization process, the LM algorithm adjusts the parameters according to the gradient direction of the objective function, gradually reducing the weighted reprojection error. The algorithm configuration is as follows: the maximum number of iterations is 100 times, and the convergence condition is that the change rate of the weighted error between adjacent iterations is less than 0.5% or the absolute error is less than 0.2 pixels. If the change rate of the iterative error is lower than 0.1% for 5 consecutive times, the optimization is terminated in advance.

[0118] After the optimization is completed, output the final calibration parameters, including the optimized internal parameters (focal length, principal point offset, distortion coefficient), external parameters (rotation matrix, translation vector) of the visible light and infrared cameras, and the relative pose parameters (rotation matrix, translation vector) between the two.

[0119] Embodiment 2

[0120] The present invention also provides an automatic calibration system for an ultra-high definition 8K dual-light fusion system, which is applied to a dual-light fusion system including a visible light camera and an infrared camera, and is used to execute an automatic calibration method for an ultra-high definition 8K dual-light fusion system. Refer to Figure 2 As shown, the system includes:

[0121] An exposure parameter adjustment module 100, configured to periodically read the real-time ambient light intensity, and dynamically adjust the exposure parameters of the visible light camera and the infrared camera in combination with a preset parameter table, where the preset parameter table represents the optimal exposure parameter combinations of the visible light camera and the infrared camera in different light intensity intervals.

[0122] A feature point matching module 200, configured to obtain a visible light image and an infrared image based on the exposure parameters, and perform feature point matching on the visible light image and the infrared image to obtain a feature point matching result.

[0123] A preliminary calibration module 300, configured to allocate image weights according to the real-time ambient light intensity, and perform multi-modal fusion calibration in combination with the feature point matching result and the initial parameters based on single-camera calibration to obtain preliminary calibration parameters.

[0124] A global optimization module 400, configured to define an error correction factor based on the real-time ambient light intensity, target at minimizing the weighted reprojection error, and globally optimize the preliminary calibration parameters using a non-linear optimization algorithm, output the final calibration parameters and complete the calibration.

[0125] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowcharts and / or block diagrams can be implemented by computer program instructions, and the combination of the processes and / or blocks in the flowcharts and / or block diagrams can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for realizing the functions specified in one process Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0126] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. The storage medium includes read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc memories, magnetic disc memories, tape memories, or any other medium that can be used to carry or store data and is computer-readable.

[0127] It should also be noted that the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such a process, method, commodity or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, commodity or device including the element.

Claims

1. An automatic calibration method for an ultra-high definition 8K dual-light fusion system, which is applied to a dual-light fusion system including a visible light camera and an infrared camera, characterized in that, The method includes the following steps: Periodically read the real-time ambient light intensity, and dynamically adjust the exposure parameters of the visible light camera and the infrared camera in combination with a preset parameter table, where the preset parameter table represents the optimal exposure parameter combinations of the visible light camera and the infrared camera in different light intensity intervals; Obtain a visible light image and an infrared image based on the exposure parameters, and perform feature point matching on the visible light image and the infrared image to obtain a feature point matching result; Allocate image weights according to the real-time ambient light intensity, and perform multi-modal fusion calibration in combination with the feature point matching result and the initial parameters based on single camera calibration to obtain preliminary calibration parameters; Define an error correction factor based on the real-time ambient light intensity, and use a non-linear optimization algorithm to globally optimize the preliminary calibration parameters with the goal of minimizing the weighted reprojection error, output the final calibration parameters and complete the calibration.

2. The automatic calibration method for an ultra-high-definition 8K dual-light fusion system according to claim 1, wherein The exposure parameters include exposure duration and gain value.

3. The automatic calibration method for an ultra-high-definition 8K dual-light fusion system according to claim 2, wherein The dynamic adjustment of the exposure parameters includes the following steps: Determine the light intensity interval to which it belongs according to the currently read real-time ambient light intensity; Match the optimal exposure parameter combinations of the visible light camera and the infrared camera corresponding to the light intensity interval to which it belongs from the preset parameter table; Synchronously adjust the exposure parameters of the visible light camera and the infrared camera to the optimal exposure parameter combination.

4. The automatic calibration method for an ultra-high-definition 8K dual-light fusion system according to claim 3, characterized in that, The dynamic adjustment of the exposure parameters further includes: If the real-time ambient light intensity is in the transition region between two adjacent light intensity intervals, use the linear interpolation method to calculate the optimal exposure parameter combination according to the data in the preset parameter table.

5. The automatic calibration method for an ultra-high-definition 8K dual-light fusion system according to claim 4, wherein, The obtaining of the visible light image and the infrared image based on the exposure parameters, and the performing of feature point matching on the visible light image and the infrared image to obtain a feature point matching result includes the following steps: Obtain the visible light image and the infrared image captured by the visible light camera and the infrared camera respectively according to the dynamically adjusted optimal exposure parameter combination, and perform preprocessing; Use a pre-trained deep learning model to extract robust feature points from the visible light image and the infrared image, and based on the extracted robust feature points, where the robust feature points include visible light feature points and infrared feature points; Perform bidirectional matching on the extracted visible light feature points and infrared feature points, and screen out the feature point pairs with a matching consistency higher than a preset threshold through a geometric constraint algorithm to obtain the feature point matching result.

6. The automatic calibration method for an ultra-high-definition 8K dual-light fusion system according to claim 5, characterized in that: The process of obtaining the preliminary calibration parameters includes the following steps: Define a weight function based on the deviation degree between the real-time ambient light intensity and the preset ideal light intensity, where the deviation degree is inversely proportional to the image weight; Allocate weight values to each pair of visible light images and infrared images according to the weight function; Calculate the initial internal parameter matrix and the initial external parameter matrix of the visible light camera and the infrared camera respectively based on the single camera calibration algorithm; Calculate the relative pose relationship between the visible light camera and the infrared camera according to the feature point matching result, including a rotation matrix and a translation vector; Using the weight value as the optimization weight, combining the initial intrinsic matrix, the initial extrinsic matrix and the relative pose relationship, the multi-modal fusion calibration parameters are weighted and iteratively optimized by the non-linear least squares method to generate preliminary calibration parameters.

7. An automatic calibration method for an ultra-high-definition 8K dual-light fusion system according to claim 6, characterized in that, The output process of the final calibration parameters includes the following steps: Based on the deviation degree between the real-time ambient light intensity and the preset ideal light intensity, an error correction factor is defined; Multiply the error correction factor by the assigned weight value to obtain a comprehensive weight; Based on the comprehensive weight, the reprojection errors of each pair of visible light images and infrared images are weighted to construct a global optimization objective function; Based on the global optimization objective function, the LM algorithm is used to iteratively optimize the preliminary calibration parameters. When the weighted reprojection error meets the preset termination condition, the optimization is terminated and the final calibration parameters are output, where the final calibration parameters include the optimized intrinsic and extrinsic parameters and the relative pose parameters of the visible light camera and the infrared camera.

8. An automatic calibration system for an ultra-high definition 8K dual-light fusion system, which is applied to a dual-light fusion system including a visible light camera and an infrared camera, and is used to execute an automatic calibration method for an ultra-high definition 8K dual-light fusion system according to any one of claims 1 to 7, characterized in that The system includes: An exposure parameter adjustment module for periodically reading the real-time ambient light intensity and dynamically adjusting the exposure parameters of the visible light camera and the infrared camera in combination with a preset parameter table, where the preset parameter table represents the optimal exposure parameter combinations of the visible light camera and the infrared camera in different light intensity intervals; A feature point matching module for obtaining visible light images and infrared images based on the exposure parameters and performing feature point matching on the visible light images and the infrared images to obtain a feature point matching result; A preliminary calibration module for assigning image weights according to the real-time ambient light intensity and performing multi-modal fusion calibration in combination with the feature point matching result and the initial parameters based on single camera calibration to obtain preliminary calibration parameters; A global optimization module for defining an error correction factor based on the real-time ambient light intensity, aiming to minimize the weighted reprojection error, and using a non-linear optimization algorithm to globally optimize the preliminary calibration parameters, output the final calibration parameters and complete the calibration.

Images