Automatic calibration method and system for ultra-high-definition 8K dual-light fusion system

Abstract

The invention provides an automatic calibration method and system for an ultra-high-definition 8K dual-light fusion system, relates to the technical field of image processing, is applied to a dual-light fusion system comprising a visible light camera and an infrared camera, and comprises the following steps: periodically reading real-time ambient light intensity; the exposure parameters of the visible light camera and the infrared camera are dynamically adjusted in combination with a preset parameter table; obtaining a visible light image and an infrared image based on the exposure parameters, and carrying out feature point matching to obtain a feature point matching result; image weights are distributed according to the real-time ambient light intensity, and multi-modal fusion calibration is executed in combination with a feature point matching result and initial parameters calibrated based on a single camera to obtain initial calibration parameters; and performing global optimization on the obtained preliminary calibration parameters by using a nonlinear optimization algorithm, outputting final calibration parameters and completing calibration, thereby realizing calibration of the dual-light fusion system under different illumination conditions.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to an automatic calibration method and system for an ultra-high definition 8K dual-light fusion system. Background Art

[0002] With the rapid development of ultra-high definition video technology, video content with 8K resolution is gradually becoming the mainstream, showing great application potential in many fields such as security monitoring, autonomous driving, intelligent healthcare, and aerospace. Under complex and variable environmental conditions, a single-modal image sensor often fails to meet the visual perception requirements of all-weather and full-scene scenarios. For example, in low-light or completely dark environments, visible light cameras are difficult to capture effective information, while infrared cameras can provide key image data by virtue of their thermal imaging characteristics; conversely, in strong light or uniformly illuminated environments, infrared images may be noisy due to thermal radiation interference, and at this time, visible light images are clearer and more reliable. Therefore, a dual-light fusion system that combines a visible light camera and an infrared camera has emerged, aiming to improve the environmental adaptability and target recognition ability of the system by integrating the advantages of the two modal images.

[0003] However, the performance of the dual-light fusion system highly depends on the accurate calibration between the two cameras, including the accurate determination of internal parameters (such as focal length, principal point coordinates) and external parameters (such as rotation matrix, translation vector). Traditional calibration methods mostly adopt an offline method and are carried out under specific lighting conditions, making it difficult to adapt to the dynamic changes of the lighting environment in practical applications. In addition, under different lighting conditions, the imaging characteristics of the camera (such as noise level, contrast) will change significantly. If fixed calibration parameters are still used, it will lead to quality problems such as ghosting and blurring in the fused image, seriously affecting the reliability and stability of the system.

[0004] Therefore, it is necessary to provide an automatic calibration method and system for an ultra-high definition 8K dual-light fusion system to solve the above technical problems. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides an automatic calibration method and system for an ultra-high definition 8K dual-light fusion system, which combines the imaging characteristics of a visible light camera and an infrared camera, and realizes the calibration of the dual-light fusion system under different lighting conditions through dynamically adjusting exposure parameters, feature point matching, multi-modal fusion calibration, and non-linear optimization. The present invention provides an automatic calibration method for an ultra-high definition 8K dual-light fusion system, which is applied to a dual-light fusion system including a visible light camera and an infrared camera. The method includes the following steps: Periodically read the real-time ambient light intensity, and dynamically adjust the exposure parameters of the visible light camera and the infrared camera in combination with a preset parameter table, where the preset parameter table represents the optimal exposure parameter combinations of the visible light camera and the infrared camera in different light intensity intervals; Obtain a visible light image and an infrared image based on the exposure parameters, and perform feature point matching on the visible light image and the infrared image to obtain a feature point matching result; Allocate image weights according to the real-time ambient light intensity, and perform multi-modal fusion calibration in combination with the feature point matching result and the initial parameters based on single camera calibration to obtain preliminary calibration parameters; Define an error correction factor based on the real-time ambient light intensity, target to minimize the weighted reprojection error, and use a non-linear optimization algorithm to globally optimize the preliminary calibration parameters, output the final calibration parameters and complete the calibration.

[0006] Preferably, the exposure parameters include exposure duration and gain value.

[0007] Preferably, the dynamic adjustment of the exposure parameters includes the following steps: Determine the light intensity interval to which it belongs according to the currently read real-time ambient light intensity; Match the optimal exposure parameter combinations of the visible light camera and the infrared camera corresponding to the light intensity interval to which it belongs from the preset parameter table; Synchronously adjust the exposure parameters of the visible light camera and the infrared camera to the optimal exposure parameter combination.

[0008] Preferably, the dynamic adjustment of the exposure parameters further includes: If the real-time ambient light intensity is in the transition region between two adjacent light intensity intervals, use the linear interpolation method to calculate the optimal exposure parameter combination according to the data in the preset parameter table.

[0009] Preferably, the obtaining of the visible light image and the infrared image based on the exposure parameters, and the performing of feature point matching on the visible light image and the infrared image to obtain a feature point matching result includes the following steps: Obtain the visible light image and the infrared image captured by the visible light camera and the infrared camera respectively according to the dynamically adjusted optimal exposure parameter combination, and perform preprocessing; Use a pre-trained deep learning model to extract robust feature points from the visible light image and the infrared image, and based on the extracted robust feature points, where the robust feature points include visible light feature points and infrared feature points; Perform two-way matching on the extracted visible light feature points and infrared feature points, and filter out the feature point pairs with matching consistency higher than a preset threshold through a geometric constraint algorithm to obtain the feature point matching result.

[0010] Preferably, the process of obtaining the preliminary calibration parameters includes the following steps: Define a weight function based on the deviation degree between the real-time ambient light intensity and the preset ideal illumination intensity, where the deviation degree is inversely proportional to the image weight; Assign weight values to each pair of visible light images and infrared images according to the weight function; Calculate the initial intrinsic matrix and the initial extrinsic matrix of the visible light camera and the infrared camera respectively based on the single camera calibration algorithm; According to the feature point matching result, calculate the relative pose relationship between the visible light camera and the infrared camera, including the rotation matrix and the translation vector; Taking the weight value as the optimization weight, combining the initial intrinsic matrix, the initial extrinsic matrix and the relative pose relationship, and performing weighted iterative optimization on the multi-modal fusion calibration parameters through the nonlinear least squares method to generate preliminary calibration parameters.

[0011] Preferably, the expression of the weight function is: Wherein, is the real-time ambient light intensity, is the preset ideal illumination intensity, is the adjustment coefficient for controlling the smoothness of the weight function, and , is the real-time ambient light intensity under the weight value, is the base of the exponential function.

[0012] Preferably, the process of outputting the final calibration parameters includes the following steps: Define an error correction factor based on the deviation degree between the real-time ambient light intensity and the preset ideal illumination intensity; Multiply the error correction factor by the assigned weight value to obtain the comprehensive weight; Based on the comprehensive weight, weight the reprojection error of each pair of visible light images and infrared images to construct a global optimization objective function; Based on the global optimization objective function, use the LM algorithm to perform iterative optimization on the preliminary calibration parameters, and terminate the optimization when the weighted reprojection error meets the preset termination condition, and output the final calibration parameters, where the final calibration parameters include the optimized intrinsic and extrinsic parameters and the relative pose parameters of the visible light camera and the infrared camera.

[0013] The present invention also provides an automatic calibration system for an ultra-high-definition 8K dual-light fusion system, which is applied to a dual-light fusion system including a visible light camera and an infrared camera, and is used to execute the automatic calibration method for an ultra-high-definition 8K dual-light fusion system. The system includes: An exposure parameter adjustment module, configured to periodically read the real-time ambient light intensity, and dynamically adjust the exposure parameters of the visible light camera and the infrared camera in combination with a preset parameter table, where the preset parameter table represents the optimal exposure parameter combinations of the visible light camera and the infrared camera in different light intensity intervals; A feature point matching module, configured to obtain a visible light image and an infrared image based on the exposure parameters, and perform feature point matching on the visible light image and the infrared image to obtain a feature point matching result; A preliminary calibration module, configured to allocate image weights according to the real-time ambient light intensity, and perform multi-modal fusion calibration in combination with the feature point matching result and the initial parameters based on single-camera calibration to obtain preliminary calibration parameters; A global optimization module, configured to define an error correction factor based on the real-time ambient light intensity, and use a non-linear optimization algorithm to globally optimize the preliminary calibration parameters with the goal of minimizing the weighted reprojection error, output the final calibration parameters, and complete the calibration.

[0014] Compared with the related technologies, the automatic calibration method and system for an ultra-high-definition 8K dual-light fusion system provided by the present invention have the following beneficial effects: The automatic calibration method for the ultra-high-definition 8K dual-light fusion system proposed by the present invention can accurately and dynamically adjust the exposure parameters of the visible light camera and the infrared camera according to the real-time ambient light intensity, ensuring high-quality images are obtained under various lighting conditions. By innovatively combining multi-modal fusion calibration and non-linear optimization algorithms, not only the calibration accuracy is greatly improved, but also the excellent adaptability and stability of the system to complex and variable lighting environments are enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flowchart of an automatic calibration method for an ultra-high-definition 8K dual-light fusion system provided by the present invention; Figure 2 is a module structure diagram of an automatic calibration system for an ultra-high-definition 8K dual-light fusion system provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0016] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present invention, rather than limiting the present invention. Additionally, it should be noted that for the sake of description, only parts related to the present invention rather than all structures are shown in the drawings. Furthermore, the embodiments in the present invention and the features in the embodiments can be combined with each other without conflict.

[0017] It should also be noted that for the sake of description, only parts related to the present invention rather than all content are shown in the drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the operations can be rearranged. When the operations are completed, the process can be terminated, but there can also be additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0018] Embodiment 1 The present invention provides an automatic calibration method for an ultra-high-definition 8K dual-light fusion system, which is applied to a dual-light fusion system including a visible light camera and an infrared camera. Referring to Figure 1 as shown, the method includes the following steps: S1: Periodically read the real-time ambient light intensity, and dynamically adjust the exposure parameters of the visible light camera and the infrared camera in combination with a preset parameter table, where the preset parameter table represents the optimal exposure parameter combinations of the visible light camera and the infrared camera in different light intensity intervals.

[0019] Before step S1, it is necessary to construct a preset parameter table. Specifically: in a controllable lighting laboratory, use a standard light source to simulate different light intensity environments, and record the ambient light intensity through a high-precision light sensor. For each light intensity interval, collect visible light images and infrared images, determine the optimal exposure parameter combinations according to the image quality evaluation index, and store the light intensity intervals and the corresponding exposure parameter combinations as a preset parameter table, where the exposure parameter combinations include exposure duration and gain value.

[0020] Specifically, the dynamic adjustment of the exposure parameters includes the following: When reading the real-time ambient light intensity, collect data through a light sensor at a preset frequency, and perform moving average filtering on the data to suppress instantaneous noise; match the corresponding interval in the preset parameter table according to the filtered light intensity value: if the light intensity completely falls within a certain interval, directly adopt the exposure parameters corresponding to this interval as the optimal exposure parameters; if it is in the transition region between adjacent intervals, calculate the smoothly transitioning exposure parameters through linear interpolation.

[0021] When adjusting the exposure parameters, an exposure parameter update instruction including the exposure duration and gain value is sent to the visible light and infrared cameras through the camera control interface. The parameter switching adopts a progressive transition strategy to avoid screen flickering.

[0022] S2: Obtain the visible light image and the infrared image based on the exposure parameters, and perform feature point matching on the visible light image and the infrared image to obtain the feature point matching result.

[0023] Specifically, step S2 specifically includes the following steps: S21: Obtain the visible light image and the infrared image captured by the visible light camera and the infrared camera respectively according to the dynamically adjusted optimal exposure parameter combination, and perform preprocessing.

[0024] In this embodiment, the visible light camera and the infrared camera capture images synchronously according to the dynamically adjusted optimal exposure parameters. After the visible light image is collected, contrast-limited adaptive histogram equalization (CLAHE) is first performed, with the block size set to 8×8 and the contrast limit to 2.0, to enhance the details of the dark areas and suppress the overexposed areas. Subsequently, adaptive gamma correction is adopted to dynamically adjust the gamma value (range 0.5~1.5) according to the local brightness of the image to compensate for uneven illumination. For the infrared image, non-uniformity correction is performed: based on the two-point correction method, a blackbody radiation source (temperatures 25°C and 60°C) is used to calibrate the gain and offset parameters of each pixel to eliminate the sensor response differences. The corrected infrared image is smoothed by Gaussian filtering (kernel size 5×5, standard deviation 1.5), and the pixel value range is compressed from the original 12 bits to 8 bits by histogram stretching to improve visibility. Finally, geometric alignment is performed on the visible light and infrared images: based on the corner coordinates of the calibration board, affine transformation is applied to register the infrared image to the visible light image coordinate system to ensure that their fields of view are consistent.

[0025] S22: Use a pre-trained deep learning model to extract robust feature points from the visible light image and the infrared image, and based on the extracted robust feature points, where the robust feature points include visible light feature points and infrared feature points.

[0026] In this embodiment, a pre-trained two-stream feature extraction network, that is, a deep learning model, has the following network structure: Input layer: The size of the visible light image is 3840×2160, and the infrared image is 640×512, both normalized to the range [-1, 1].

[0027] Backbone network: An improved ResNet-50, with a cross-modal attention module inserted after Stage3, generating spatial attention weights through the visible light feature map to guide the infrared branch to focus on the texture-rich areas.

[0028] Output layer: Each branch outputs a 256-dimensional feature descriptor and the coordinates of feature points (2000 points are extracted from visible light images and 500 points are extracted from infrared images).

[0029] Training details: Dataset: FLIR ADAS dataset (including 100,000 pairs of visible light-infrared aligned images).

[0030] Loss function: Triplet Loss, with the margin set to 0.5 and the hard example mining ratio of 30%.

[0031] Data augmentation: Random illumination change (±20% brightness), Gaussian noise ( = 0.05), and simulated lens blur (kernel size 3×3).

[0032] During the inference stage, for visible light images, non-maximum suppression (NMS) is applied at the coordinates of feature points, with a neighborhood radius of 5 pixels, and the top 1000 points with the highest confidence are retained; for infrared images, the feature response in low-texture regions is enhanced based on the thermal radiation gradient, and regions with a gradient value greater than 10 °C / pixel are selected for dense sampling.

[0033] S23: Bidirectionally match the extracted visible light feature points and infrared feature points, and filter out the feature point pairs with a matching consistency higher than a preset threshold through a geometric constraint algorithm to obtain the feature point matching result.

[0034] In this embodiment, during the initial matching, calculate the cosine similarity matrix of the descriptors of visible light feature points and infrared feature points, with a dimension of 1000×500; perform bidirectional nearest neighbor matching, for each visible light feature point, find the infrared feature point with the highest similarity; perform the same operation on infrared feature points in reverse; retain the pairs that match bidirectionally (i.e., A→B and B→A) to obtain the initial matching pairs.

[0035] During the geometric consistency verification: Use the RANSAC algorithm to estimate the fundamental matrix, set the maximum number of iterations to 1000, and the reprojection error threshold to 1.0 pixel; calculate the Sampson distance of all matching pairs, retain the inliers with a distance less than 0.8 pixels, count the inlier ratio, if it is lower than 80%, it is determined as an invalid matching set and re-matching is triggered; calculate the final matching confidence according to the similarity of feature point descriptors (normalized to 0 to 1) and the inlier status; retain the matching pairs with a confidence ≥0.8, and output them as the feature point matching result, including the image coordinates, descriptors, and confidence scores of the matching point pairs.

[0036] S3: Allocate image weights according to the real-time ambient light intensity, and perform multi-modal fusion calibration in combination with the feature point matching result and the initial parameters based on single camera calibration to obtain the preliminary calibration parameters.

[0037] Specifically, the process of obtaining the preliminary calibration parameters includes the following steps: Define a weight function based on the deviation degree between the real-time ambient light intensity and the preset ideal illumination intensity, where the deviation degree is inversely proportional to the image weight; Among them, the expression of the weight function is: Among them, is the real-time ambient light intensity, is the preset ideal illumination intensity, is the adjustment coefficient for controlling the smoothness of the weight function, and , is the real-time ambient light intensity under the weight value, is the base of the exponential function.

[0038] In this embodiment, the method for calibrating parameters in the laboratory: Collect multiple groups of dual-light image data with different real-time ambient light intensities in a controllable illumination environment.

[0039] Determine the optimal weight value under each illumination intensity through manual evaluation or automated metrics (such as the correct rate of feature point matching, image signal-to-noise ratio).

[0040] Use the non-linear least squares method to fit values to make the function curve match the experimental data (for example = 100).

[0041] Technical effect: When is close to , is approximately equal to 1, giving a high weight; when deviates from , exponentially decays to suppress the influence of low-quality data.

[0042] Assign weight values to each pair of visible light images and infrared images according to the weight function.

[0043] In this embodiment, according to the current ambient light intensity , substitute it into the weight function to calculate the weight value of each pair of visible light-infrared images.

[0044] If multiple groups of image data (such as consecutive frames) are processed simultaneously, normalize the weight values to the interval [0.1, 1.0] to avoid interference from extreme values in the optimization process.

[0045] Assign an initial weight to each feature point pair and multiply it by the matching confidence (output by step S23) to form a weight value.

[0046] Based on the single-camera calibration algorithm, calculate the initial intrinsic matrix and the initial extrinsic matrix of the visible-light camera and the infrared camera respectively.

[0047] In the single-camera calibration stage, first, a standard checkerboard calibration board needs to be prepared, and the physical sizes of its black and white squares need to be precisely set. Place the calibration board within the field of view of the visible-light camera and the infrared camera respectively, and place it in multiple different angles and positions (at least 15 different poses) to ensure that the calibration board covers the central area and the edge area of the image. Take the corresponding visible-light images and infrared images for each pose for subsequent calculations.

[0048] Independently calibrate each camera through the Zhang-Zhengyou calibration method. In the specific process, use the corner coordinates of the calibration board in the image to calculate the internal parameters of the camera, including the focal length parameter (describing the focusing ability of the lens on light), the principal point offset parameter (indicating the position of the lens optical axis on the image plane), and the distortion coefficient used to correct image deformation. During the calibration process, it is necessary to ensure that the calculated reprojection error (i.e., the average deviation between the actual position and the theoretical projection position of the calibration board corners in the image) does not exceed 0.5 pixels to ensure the calibration accuracy.

[0049] In addition, it is also necessary to calculate the external parameters of the camera, that is, the rotation angle and the translation distance of the camera relative to the calibration board in different poses. By solving the three-dimensional spatial position of the calibration board and the two-dimensional projection relationship in the image, the spatial pose of the camera at each shooting is obtained, and these parameters provide an initial reference for subsequent multi-modal fusion calibration.

[0050] According to the feature point matching results, calculate the relative pose relationship between the visible-light camera and the infrared camera, including the rotation matrix and the translation vector.

[0051] In this embodiment, when calculating the relative pose relationship between the visible-light camera and the infrared camera, first, based on the feature point matching results obtained in step S2, at least 50 pairs of high-confidence matching points are selected. Exclude the mismatched points through a robust estimation algorithm (including but not limited to RANSAC), and estimate the fundamental geometric relationship matrix between the two cameras. This matrix can describe the projection constraint conditions between the perspectives of the two cameras.

[0052] Subsequently, convert the fundamental geometric relationship matrix into an essential matrix describing the actual pose relationship between the cameras. Through mathematical decomposition of the essential matrix, possible rotation angle combinations and translation directions are deduced, and these combinations need to be verified by three-dimensional space points. Specifically, use the two-dimensional coordinates of the matching point pairs to reverse-infer their three-dimensional spatial positions, and calculate the projection errors under different rotation and translation combinations. Finally, select the combination with the smallest error as the precise relative pose parameters between the cameras.

[0053] During this process, if it is found that the projection error exceeds the preset threshold (for example, more than 1 pixel), it is necessary to re-screen the matching points or adjust the parameter decomposition method until the relative pose relationship that meets the accuracy requirements is obtained. These parameters will be used for subsequent fusion calibration to ensure the precise alignment of visible light and infrared images in space.

[0054] Using the weight value as the optimization weight, combining the initial internal parameter matrix, the initial external parameter matrix, and the relative pose relationship, the multi-modal fusion calibration parameters are weighted and iteratively optimized through the non-linear least squares method to generate preliminary calibration parameters.

[0055] In this embodiment, in the weighted iterative optimization stage, the initial parameters obtained from single-camera calibration, the relative pose relationship, and the weight value dynamically allocated based on the lighting conditions are combined, and preliminary calibration parameters are generated through a non-linear optimization algorithm. During the optimization process, the goal is to minimize the weighted projection error of all feature point pairs, that is, to adjust the camera parameters so that the actual position of the feature points in the image is as close as possible to the theoretical projection position, while considering the difference in data credibility under different lighting conditions.

[0056] First, define the set of parameters to be optimized, including the internal parameters (such as focal length, principal point offset, distortion coefficient) of visible light and infrared cameras, the external parameters (such as rotation angle, translation vector), and the relative pose relationship between the two. These parameters jointly affect the projection result of three-dimensional space points to two-dimensional images through a mathematical model.

[0057] When constructing the objective function, calculate the projection error of each feature point matching pair in the visible light and infrared images, and multiply this error by the corresponding comprehensive weight value (determined jointly by the lighting weight and the feature point confidence). For example, high-quality image data collected under ideal lighting conditions will be given a higher weight, while the influence of data with lighting deviating from the ideal value or low matching confidence on the optimization will be weakened. By accumulating all weighted error values to form a global optimization objective, ensure that the optimization process pays more attention to high-reliability data.

[0058] The non-linear least squares optimization algorithm (Levenberg-Marquardt algorithm) is used for iterative solution. In each iteration, the algorithm calculates the objective function value according to the current parameters, and adjusts the parameters in the gradient descent direction to reduce the error. Set the convergence condition during the optimization process: terminate the optimization when the change rate of the weighted error between two adjacent iterations is less than 1% or reach the maximum number of iterations (such as 50 times). If parameter oscillation or error non-convergence is detected during the optimization process, trigger an exception handling mechanism, such as reverting to historical parameters or re-initializing the optimization variables.

[0059] After the optimization is completed, output the preliminary calibration parameters, including the optimized internal parameters, external parameters, and relative pose relationship of the camera.

[0060] S4: Define an error correction factor based on the real-time ambient light intensity. Aiming to minimize the weighted reprojection error, use a non-linear optimization algorithm to globally optimize the preliminary calibration parameters, output the final calibration parameters, and complete the calibration.

[0061] Specifically, the output process of the final calibration parameters includes the following steps: Define an error correction factor based on the deviation degree between the real-time ambient light intensity and the preset ideal light intensity.

[0062] In this embodiment, the error correction factor is defined as: represents the real-time ambient light intensity, is the preset ideal light intensity, and are positive coefficients calibrated through experiments.

[0063] Define an error correction factor based on the deviation degree between the real-time ambient light intensity and the preset ideal light intensity. The error correction factor adopts the form of an exponential function, and its parameters and are determined through experimental calibration: In a controllable lighting laboratory, simulate scenarios with different deviation degrees by adjusting the light source intensity, collect multiple sets of dual-light image data, calculate the attenuation trend of the calibration error in each scenario, and use non-linear least squares to fit and values so that the error correction factor can effectively suppress the error contribution of data under non-ideal lighting conditions.

[0064] Multiply the error correction factor by the assigned weight value to obtain the comprehensive weight.

[0065] In this embodiment, the comprehensive weight is defined as: .

[0066] The role of the comprehensive weight is to dynamically adjust the credibility weight of data under different lighting conditions. For example, when the lighting seriously deviates from the ideal value, the comprehensive weight is significantly reduced to reduce the negative impact of low-quality data on global optimization. The specific calculation method of the comprehensive weight is: for each pair of visible-infrared images, calculate the weighted value according to its real-time light intensity and the confidence of feature point matching, ensuring that data with high confidence and lighting conditions close to the ideal play a dominant role in the optimization.

[0067] Based on the comprehensive weight, weight the reprojection error of each pair of visible light images and infrared images to construct a global optimization objective function.

[0068] In this embodiment, the global optimization objective function is defined as: where is the projection function, and are the rotation matrix and translation vector to be optimized, is a three-dimensional space point, is an image feature point.

[0069] When constructing the global optimization objective function, the goal is to minimize the weighted reprojection error. Specifically, calculate the Euclidean distance between the projected positions and the actual detected positions of each feature point pair in visible light and infrared images, multiply this distance by the corresponding comprehensive weight, and accumulate all weighted distances as the optimization objective. The influence of the internal parameters (such as focal length, distortion coefficient) and external parameters (such as rotation matrix, translation vector) of the camera and the relative pose parameters on the projection result is considered during the projection process. For example, after the three-dimensional space point is transformed to the camera coordinate system through rotation and translation, it is projected onto the image plane in combination with the focal length and principal point offset parameters, and the distortion coefficient is used to correct the non-linear deformation error.

[0070] Based on the global optimization objective function, the LM algorithm is used to iteratively optimize the preliminary calibration parameters. When the weighted reprojection error meets the preset termination condition, the optimization is terminated, and the final calibration parameters are output. Among them, the final calibration parameters include the optimized internal parameters, external parameters, and relative pose parameters of the visible light camera and the infrared camera.

[0071] In this embodiment, the Levenberg-Marquardt (LM) algorithm is used to perform iterative optimization. The initial parameters are the preliminary calibration parameters output in step S3, including the internal parameters, external parameters, and relative pose of the visible light and infrared cameras. During the optimization process, the LM algorithm adjusts the parameters according to the gradient direction of the objective function, gradually reducing the weighted reprojection error. The algorithm configuration is as follows: the maximum number of iterations is 100 times, and the convergence condition is that the change rate of the weighted error between adjacent iterations is less than 0.5% or the absolute error is less than 0.2 pixels. If the change rate of the iteration error is lower than 0.1% for 5 consecutive times, the optimization is terminated in advance.

[0072] After the optimization is completed, the final calibration parameters are output, including the optimized internal parameters (focal length, principal point offset, distortion coefficient), external parameters (rotation matrix, translation vector) of the visible light and infrared cameras, and the relative pose parameters (rotation matrix, translation vector) between the two.

[0073] Embodiment 2 The present invention also provides an automatic calibration system for an ultra-high-definition 8K dual-light fusion system, which is applied to a dual-light fusion system including a visible light camera and an infrared camera, and is used to execute an automatic calibration method for the ultra-high-definition 8K dual-light fusion system, with reference to Figure 2 As shown, the system includes: An exposure parameter adjustment module 100, configured to periodically read the real-time ambient light intensity and dynamically adjust the exposure parameters of the visible light camera and the infrared camera in combination with a preset parameter table, where the preset parameter table represents the optimal exposure parameter combinations of the visible light camera and the infrared camera in different light intensity intervals.

[0074] A feature point matching module 200, configured to obtain a visible light image and an infrared image based on the exposure parameters, and perform feature point matching on the visible light image and the infrared image to obtain a feature point matching result.

[0075] A preliminary calibration module 300, configured to allocate image weights according to the real-time ambient light intensity, and perform multi-modal fusion calibration in combination with the feature point matching result and the initial parameters based on single-camera calibration to obtain preliminary calibration parameters.

[0076] A global optimization module 400, configured to define an error correction factor based on the real-time ambient light intensity, target at minimizing the weighted reprojection error, and use a non-linear optimization algorithm to globally optimize the preliminary calibration parameters, output the final calibration parameters and complete the calibration.

[0077] This application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of the flows and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0078] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. The storage medium includes read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc memories, magnetic disk memories, tape memories, or any other medium that can be used to carry or store data and is computer-readable.

[0079] It should also be noted that the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, commodity or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, commodity or device including the element.

Claims

1. An automatic calibration method for an ultra-high-definition 8K dual-light fusion system, applied to a dual-light fusion system including a visible light camera and an infrared camera, characterized in that: The method comprises the following steps: Periodically read the real-time ambient light intensity, and dynamically adjust the exposure parameters of the visible light camera and the infrared camera in combination with a preset parameter table, wherein the preset parameter table represents the optimal exposure parameter combination of the visible light camera and the infrared camera under different light intensity ranges; Acquire a visible light image and an infrared image based on the exposure parameters, and perform feature point matching on the visible light image and the infrared image to obtain a feature point matching result; Allocating image weights according to the real-time ambient light intensity, and performing multimodal fusion calibration in combination with the feature point matching results and initial parameters based on single camera calibration to obtain preliminary calibration parameters; An error correction factor based on the real-time ambient light intensity is defined, and the preliminary calibration parameters are globally optimized using a nonlinear optimization algorithm with the goal of minimizing the weighted reprojection error, and the final calibration parameters are output and the calibration is completed.

2. According to claim 1, an automatic calibration method for an ultra-high-definition 8K dual-light fusion system is characterized in that: The exposure parameters include exposure duration and gain value.

3. According to claim 2, an automatic calibration method for an ultra-high-definition 8K dual-light fusion system is characterized in that: The dynamic adjustment of the exposure parameters comprises the following steps: According to the currently read real-time ambient light intensity, determine the light intensity range; Matching the optimal exposure parameter combination of the visible light camera and the infrared camera corresponding to the corresponding light intensity interval from the preset parameter table; The exposure parameters of the visible light camera and the infrared camera are synchronously adjusted to the optimal exposure parameter combination.

4. According to claim 3, an automatic calibration method for an ultra-high-definition 8K dual-light fusion system is characterized in that: The dynamic adjustment of the exposure parameters also includes: If the real-time ambient light intensity is in a transition region between two adjacent illumination intensity intervals, a linear interpolation method is used to calculate an optimal exposure parameter combination according to the data in the preset parameter table.

5. According to claim 4, an automatic calibration method for an ultra-high-definition 8K dual-light fusion system is characterized in that: The step of acquiring a visible light image and an infrared image based on the exposure parameters, and performing feature point matching on the visible light image and the infrared image to obtain a feature point matching result comprises the following steps: Obtaining the visible light image and the infrared image taken by the visible light camera and the infrared camera respectively according to the optimal exposure parameter combination after dynamic adjustment, and performing preprocessing; Extracting robust feature points from the visible light image and the infrared image using a pre-trained deep learning model, and based on the extracted robust feature points, wherein the robust feature points include visible light feature points and infrared feature points; The extracted visible light feature points and infrared feature points are bidirectionally matched, and feature point pairs whose matching consistency is higher than a preset threshold are screened out through a geometric constraint algorithm to obtain the feature point matching result.

6. The automatic calibration method for an ultra-high-definition 8K dual-light fusion system according to claim 5, characterized in that: The process of obtaining the preliminary calibration parameters includes the following steps: Defining a weight function based on the degree of deviation between the real-time ambient light intensity and the preset ideal light intensity, wherein the degree of deviation is inversely proportional to the image weight; assigning a weight value to each pair of visible light image and infrared image according to the weight function; Based on the single camera calibration algorithm, the initial intrinsic parameter matrix and initial extrinsic parameter matrix of the visible light camera and infrared camera are calculated respectively; According to the feature point matching results, the relative position relationship between the visible light camera and the infrared camera is calculated, including the rotation matrix and the translation vector; Taking the weight value as the optimization weight, combining the initial intrinsic parameter matrix, the initial extrinsic parameter matrix and the relative posture relationship, the multimodal fusion calibration parameters are weighted iteratively optimized through the nonlinear least squares method to generate preliminary calibration parameters.

7. The automatic calibration method for an ultra-high-definition 8K dual-light fusion system according to claim 6, characterized in that: The expression of the weight function is: in, is the real-time ambient light intensity, To preset the ideal light intensity, is the adjustment coefficient for controlling the smoothness of the weight function, and , is the real-time ambient light intensity The weight value under is the base of the exponential function.

8. The automatic calibration method for an ultra-high-definition 8K dual-light fusion system according to claim 7, characterized in that: The output process of the final calibration parameters includes the following steps: Based on the deviation between the real-time ambient light intensity and the preset ideal light intensity, an error correction factor is defined; Multiplying the error correction factor by the assigned weight value to obtain a comprehensive weight; Based on the comprehensive weight, the reprojection error of each pair of visible light images and infrared images is weighted to construct a global optimization objective function; Based on the global optimization objective function, the preliminary calibration parameters are iteratively optimized using the LM algorithm. When the weighted projection error meets the preset termination condition, the optimization is terminated and the final calibration parameters are output, wherein the final calibration parameters include the optimized intrinsic parameters, extrinsic parameters and relative posture parameters of the visible light camera and the infrared camera.

9. An automatic calibration system for an ultra-high-definition 8K dual-light fusion system, applied to a dual-light fusion system including a visible light camera and an infrared camera, for executing an automatic calibration method for an ultra-high-definition 8K dual-light fusion system as claimed in any one of claims 1 to 8, characterized in that: The system comprises: An exposure parameter adjustment module, which is used to periodically read the real-time ambient light intensity and dynamically adjust the exposure parameters of the visible light camera and the infrared camera in combination with a preset parameter table, wherein the preset parameter table represents the optimal exposure parameter combination of the visible light camera and the infrared camera under different light intensity ranges; A feature point matching module, used for acquiring a visible light image and an infrared image based on the exposure parameters, and performing feature point matching on the visible light image and the infrared image to obtain a feature point matching result; A preliminary calibration module, used to assign image weights according to the real-time ambient light intensity, and perform multimodal fusion calibration based on the feature point matching results and initial parameters based on single camera calibration to obtain preliminary calibration parameters; The global optimization module is used to define an error correction factor based on the real-time ambient light intensity, with the goal of minimizing the weighted reprojection error, use a nonlinear optimization algorithm to globally optimize the preliminary calibration parameters, output final calibration parameters and complete the calibration.

Images