Three-order dynamic luminosity calibration method adopting estimation-optimization and double-layer processing architecture

Abstract

The invention relates to a three-order dynamic luminosity calibration method and device adopting an estimation-optimization and double-layer processing architecture and computer equipment. The method comprises the following steps: selecting at least one target key frame from a target video sequence, and determining a video frame sample set; establishing an estimation-optimization network structure based on optical flow and physical constraints according to the video frame sample set, preliminarily estimating a luminosity parameter set, setting constraint conditions, and performing luminosity calibration on the video frame by adopting the obtained result of the luminosity parameter set to obtain a luminosity characteristic calibration value and obtain an estimation-optimization framework; and establishing a luminosity dynamic adjustment double-layer processing framework comprising a front end and a rear end for processing the received video frame in real time and outputting a luminosity characteristic calibration value, and iterating the luminosity parameter set and transmitting the luminosity parameter set to the front end. The method provides a continuous and self-adaptive luminosity calibration strategy, and is more suitable for real-time application scenes.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision and VSLAM (Visual SLAM, Simultaneous Localization and Mapping), and in particular to a third-order dynamic photometric calibration method, device and computer equipment using an estimation-optimization and double-layer processing architecture. Background Art

[0002] In the development of direct visual positioning and SLAM algorithms, these technologies require online photometric calibration to achieve efficient photometric invariance processing. Traditionally, many visual systems rely on the assumption of constant image brightness, but in automatic exposure video capture, this assumption often does not hold, resulting in the need for real-time adjustment of photometric parameters to adapt to the changing visual environment. In past studies, although a variety of photometric calibration methods have been proposed, such as multi-image approximation and linear optimization, these methods are usually only applicable to offline applications such as panoramic stitching and photo synthesis, and cannot effectively cope with dynamic changes in video sequences. The limitation of these methods is that they often require image alignment or assume no vignetting effects, which is often unrealistic in practical applications.

[0003] Therefore, new solutions are needed in this field that can automatically process and correct the camera's video data for exposure in real time without relying on static image alignment and complex preprocessing, thereby improving the accuracy and robustness of SLAM and visual positioning systems. Summary of the invention

[0004] Based on this, it is necessary to provide a third-order dynamic photometric calibration method, device and computer equipment that adopts estimation-optimization and two-layer processing architecture, which can quickly adapt to lighting changes and automatically adjust photometric parameters, and is suitable for high-precision visual navigation and spatial reconstruction in dynamic lighting or uneven lighting environments.

[0005] In a first aspect, the present application provides a third-order dynamic photometric calibration method using an estimation-optimization and dual-layer processing architecture, which is used to implement photometric calibration of a target video sequence, including:

[0006] Step S1, selecting at least one target key frame in the target video sequence, using KLT feature tracking technology to capture the pixel intensity of the field light in the target key frame, solving the optical flow vector by estimating the movement direction and speed of each point in the target key frame image, and determining the video frame sample set based on the optical flow vector;

[0007] Step S2, establishing an estimation-optimization network structure based on optical flow and physical constraints according to the video frame sample set, preliminarily estimating a photometric parameter set, setting constraints, and optimizing the photometric parameter set using the least squares method, substituting the optimized photometric parameter set into a photometric calibration formula, and using the obtained photometric parameter set result to perform photometric calibration on the video frame to obtain a photometric feature calibration value; iterating the estimation-optimization network structure to obtain an estimation-optimization framework with the video frame in the video frame sample set as input and the photometric feature calibration value as output; wherein the photometric parameter set includes CRF, exposure time, and vignetting effect parameters;

[0008] Step S3, establish a two-layer processing framework for dynamic adjustment of photometry including a front end and a back end; wherein the front end of the two-layer processing framework for dynamic adjustment of photometry includes an estimation-optimization framework for real-time processing of received video frames and outputting photometric feature calibration values ​​to the back end; the back end of the two-layer processing framework for dynamic adjustment of photometry outputs photometric feature calibration values ​​to video frames, and simultaneously iterates the photometric parameter set and transmits it to the front end.

[0009] In one embodiment, step S1 comprises:

[0010] At least one target key frame whose exposure difference reaches a preset threshold is selected in the target video sequence. The KLT feature tracking technology is used in the selected target key frame. The position of the scene point is represented by a coordinate system. The brightness constant equation is shown in formula (1):

[0011] I(x,y,t)=I(x+dx,y+dy,t+dt) (1)

[0012] In formula (1), I(x, y, t) is the pixel intensity at position (x, y) at time t, dx and dy are the displacement vectors from time t to t+dt;

[0013] Through Taylor expansion approximation, as shown in formula (2):

[0014]

[0015] In formula (2), the partial derivative and Represents the gradient of the image in the horizontal and vertical directions of the coordinate system;

[0016] The spatial and temporal gradients of the image brightness in the target key frame are linked to the optical flow vector to form the optical flow equation, as shown in formula (3):

[0017]

[0018] In formula (3), v x and v yare the components of the optical flow vector in the horizontal and vertical directions of the specified coordinate system, is the rate of change of image pixel intensity over time;

[0019] By defining an error function, the difference between the actual target key frame image and the target key frame image predicted by the optical flow is quantified. The error function is shown in formula (4):

[0020] ∈=∑ (x,y)∈W [I(x,y,t)-I(x+dx,y+dy,t+dt)] 2 (4)

[0021] In formula (4), W is the window centered on the feature point;

[0022] By solving the displacement vector that minimizes the error function, the optical flow vector representing the movement of the actual object is determined, and then iterative optimization is performed until the algorithm converges to the optimal solution to obtain a video frame sample set.

[0023] In one embodiment, step S2 comprises:

[0024] Step S2.1, respectively preliminarily estimating the CRF, exposure time and vignetting effect parameters in the photometric parameter set;

[0025] Step S2.2, based on the result of the preliminary estimation of the photometric parameter set, set the estimation-optimization network structure; adjust the photometric parameter set by nonlinear least squares method and introduce physical constraints; add regularization terms and target linear constraints in the process of adjusting the photometric parameter set by nonlinear least squares method and introducing physical constraints, and use gradient descent method to solve the nonlinear least squares method, calculate the gradient and update the photometric parameter set, stop iteration after reaching a preset threshold, and obtain the optimized photometric parameter set;

[0026] Step S2.3, substitute the optimized photometric parameter set into the photometric calibration formula to obtain the results of each photometric parameter in the photometric parameter set, use the results of the photometric parameter set to perform photometric calibration on the video frame, and obtain the photometric feature calibration value; iteratively estimate and optimize the network structure to obtain an estimation-optimization framework with the video frames in the video frame sample set as input and the photometric feature calibration value as output.

[0027] In one embodiment, step S2.1 comprises:

[0028] Step S2.1.1, performing a preliminary estimation of each photometric parameter in a photometric parameter set, where the photometric parameter set includes CRF, exposure time, and vignetting effect parameters;

[0029] Among them, the expression of CRF is shown in formula (5):

[0030] L = f-1 (M) (5)

[0031] In formula (5), L is the scene radiation, M is the image intensity value, and the specific expression is shown in formula (6):

[0032] f -1 (M) = c 0 +c 1 M+c 2 M 2 +c 3 M 3 (6)

[0033] In formula (6), c 0 、c 1 、c 2 、c 3 is the coefficient of the CRF polynomial to be estimated; c 0 is a constant term used to describe the basic offset between scene brightness and image brightness; c 1 is the linear coefficient, which is used to describe the linear relationship between the image brightness M and the scene brightness; c 2 is the quadratic term coefficient, describing a slight nonlinear change; c 3 is the cubic coefficient, which handles the high-order nonlinear response under complex illumination;

[0034] In step S2.1.2, the exposure time e is adjusted as shown in formula (7):

[0035]

[0036] In formula (7), e is the exposure time, L(t) is the radiation intensity of the scene at time t;

[0037] Step S2.1.3, the vignetting effect function is shown in formula (8):

[0038] V(R)=1+v 1 R 2 +v 2 R 4 +v 3 R 6 (8)

[0039] In formula (8), R is the normalized distance to the center of the image, v 1 v is the coefficient of the quadratic term in the vignetting polynomial, which is used to simulate the illumination changes within a small radius near the center of the image; 2 is the coefficient of the fourth order term, used to adjust the ratio v 1 Describes the lighting changes at a farther radius; v 3 It is the sixth-order coefficient, which affects the peripheral area of ​​the image.

[0040] In one embodiment, the introduction of physical constraints in step S2.2 includes:

[0041] Monotonic physical constraints are added to ensure that as the input increases, the CRF output should also increase accordingly. The expression is shown in formula (9):

[0042]

[0043] Formula (9) requires that the CRF function increases monotonically with M, that is, the derivative should always be greater than zero;

[0044] The non-negativity constraint is added to ensure that the output will not have unrealistic negative values ​​under any circumstances. The expression is shown in formula (10):

[0045]

[0046] Formula (10) requires that the output of CRF is always non-negative.

[0047] In one embodiment, step S2.2 comprises:

[0048] Step S2.2.1, introduce the result of the preliminary estimation of the photometric parameter set into the estimation-optimization network structure, and adjust the CRF, exposure time and vignetting effect parameters through the nonlinear least squares method. The least squares method expression is shown in formula (11):

[0049]

[0050] In formula (11), A is the target matrix, b is the observed data vector, and c is the coefficient to be estimated. This formula is used to determine the parameter c so that the sum of the squares of the differences between the predicted value Ac and the actual value b is minimized;

[0051] In step S2.2.2, the formula is expanded by adding regularization terms and target linear constraints as shown in formula (12):

[0052]

[0053] In formula (12), λ is the regularization parameter used to adjust the influence of the regularization term, and E and d represent additional linear constraints;

[0054] The gradient descent method is used to solve the nonlinear least squares method, and the expression is shown in formula (13):

[0055] J(c)=c T A T Ac-2b T Ac+λ(Ec-d) T (Ec-d) (13)

[0056] In formula (13), the update rule of the gradient descent method is shown in formula (14):

[0057]

[0058] In formula (14), α is the learning rate, which determines the step size of each update to ensure convergence;

[0059] The photometric parameter set update is repeated until a preset threshold is reached and then the optimized photometric parameter set is obtained.

[0060] In one embodiment, step S2.3 of substituting the optimized photometric parameter set into the photometric calibration formula includes:

[0061] Substitute the initially estimated and optimized photometric parameter set into the photometric calibration formula, as shown in formula (15):

[0062]

[0063] In formula (15), M 1 、M 2 is the measured intensity under different exposure conditions, e 1 、e 2 is the corresponding exposure time;

[0064] After linearization, the proportional relationship is re-expressed in matrix form, as shown in formula (16):

[0065]

[0066] in,

[0067] In one embodiment, step S3 comprises:

[0068] Establish a two-layer processing framework for dynamic photometric adjustment including front-end and back-end;

[0069] The front end of the dual-layer processing framework for dynamic photometric adjustment includes an estimation-optimization framework for real-time processing of received video frames and photometric calibration using a photometric parameter set, and outputting photometric feature calibration values ​​to the back end;

[0070] The back end of the dual-layer processing framework for dynamic photometric adjustment receives photometric feature calibration values ​​and outputs the photometric feature calibration values ​​to video frames. At the same time, the back end updates and iterates the photometric parameter set and transmits the updated and iterated photometric parameter set to the front end.

[0071] In a second aspect, the present application also provides a third-order dynamic photometric calibration device using an estimation-optimization and dual-layer processing architecture, comprising:

[0072] A video frame sample set determination module is used to select at least one target key frame in the target video sequence, capture the pixel intensity of the field light in the target key frame by using the KLT feature tracking technology, solve the optical flow vector by estimating the movement direction and speed of each point in the target key frame image, and determine the video frame sample set based on the optical flow vector;

[0073] The estimation-optimization framework iteration module is used to establish an estimation-optimization network structure based on optical flow and physical constraints according to the video frame sample set, preliminarily estimate the photometric parameter set, set the constraint conditions, and use the least square method to optimize the photometric parameter set, substitute the optimized photometric parameter set into the photometric calibration formula, and use the result of the obtained photometric parameter set to perform photometric calibration on the video frame to obtain the photometric feature calibration value; iterate the estimation-optimization network structure to obtain an estimation-optimization framework with the video frame in the video frame sample set as input and the photometric feature calibration value as output; wherein the photometric parameter set includes CRF, exposure time and vignetting effect parameters;

[0074] The photometric dynamic adjustment double-layer processing framework training module is used to establish a photometric dynamic adjustment double-layer processing framework including a front end and a back end; wherein the front end of the photometric dynamic adjustment double-layer processing framework includes an estimation-optimization framework, which is used to process the received video frames in real time and output photometric feature calibration values ​​to the back end; the back end of the photometric dynamic adjustment double-layer processing framework outputs photometric feature calibration values ​​to the video frames, and simultaneously iterates the photometric parameter set and transmits it to the front end.

[0075] In a third aspect, the present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of a third-order dynamic photometric calibration method using an estimation-optimization and a two-layer processing architecture.

[0076] The above-mentioned third-order dynamic photometric calibration method, device and computer equipment using estimation-optimization and two-layer processing architecture selects at least one target key frame in the target video sequence, uses KLT feature tracking technology to capture the pixel intensity of the field light in the target key frame, solves the optical flow vector by estimating the movement direction and speed of each point in the target key frame image, and determines the video frame sample set based on the optical flow vector; establishes an estimation-optimization network structure based on optical flow and physical constraints according to the video frame sample set, preliminarily estimates the photometric parameter set, sets constraints, and uses the least squares method to optimize the photometric parameter set, substitutes the optimized photometric parameter set into the photometric calibration formula, and uses the obtained The result of the photometric parameter set is used to calibrate the video frame to obtain the photometric feature calibration value; the estimation-optimization network structure is iterated to obtain an estimation-optimization framework with the video frame in the video frame sample set as input and the photometric feature calibration value as output; wherein the photometric parameter set includes CRF, exposure time and vignetting effect parameters; a two-layer processing framework for dynamic photometric adjustment is established, which includes a front-end and a back-end; wherein the front-end of the two-layer processing framework for dynamic photometric adjustment includes an estimation-optimization framework for real-time processing of received video frames and outputting photometric feature calibration values ​​to the back-end; the back-end of the two-layer processing framework for dynamic photometric adjustment outputs photometric feature calibration values ​​to video frames, and iterates the photometric parameter set and transmits it to the front-end. This method can quickly adapt to lighting changes and automatically adjust photometric parameters, and is suitable for high-precision visual navigation and spatial reconstruction in dynamic lighting or uneven lighting environments. By introducing online sequential photometric calibration technology, this method independently estimates and optimizes CRF, exposure time and vignetting effect, thereby providing stable and reliable image processing results under unknown or changing lighting conditions, meeting the high standard requirements of industrial applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 A schematic diagram of a flow chart of a third-order dynamic photometric calibration method using an estimation-optimization and dual-layer processing architecture in one embodiment;

[0078] Figure 2 A schematic diagram of a process flow of a third-order dynamic photometric calibration, estimation-optimization and dual-layer processing architecture method for visual SLAM in one embodiment;

[0079] Figure 3 is a flow chart of an estimation-optimization framework based on optical flow and physical constraints in one embodiment;

[0080] Figure 4 A schematic diagram of a dual-layer processing framework for dynamic photometric adjustment based on real-time and optimized fusion in one embodiment;

[0081] Figure 5 FIG. 4 is a schematic diagram of a module of a third-order dynamic photometric calibration device using an estimation-optimization and dual-layer processing architecture in one embodiment. DETAILED DESCRIPTION

[0082] DTAM (Dense Tracking and Mapping) is a monocular SLAM algorithm proposed in 2011 and is considered the ancestor of the direct method. DTAM effectively reduces the uncertainty of pose by performing probabilistic depth measurement on each pixel, and obtains dense maps and camera poses by aligning the entire image. However, this method requires GPU (Graphics Processing Unit) acceleration to meet its computing requirements. Based on the principle of DTAM, a visual odometry system based on direct tracking was further developed. This is the first real-time visual odometry (VO) system that does not use features. Subsequently, map optimization was integrated into the VO system and expanded to LSD-SLAM (Large-Scale Direct Monocular SLAM, a direct method algorithm), becoming a real-time SLAM system that does not use features. LSD-SLAM generates a map with global consistency by directly registering the image photometry and using a probabilistic model to represent the semi-dense depth map. As a direct SLAM system, LSD-SLAM is supported by binocular cameras and panoramic cameras, but it is very sensitive to the camera's internal parameters and exposure, and is not as accurate as ORB-SLAM (OrientedFAST and Rotated BRIEF, a feature point method algorithm), and is not as fast as DSO. Continuing on this basis, researchers released DSO (Direct Sparse Odometry). DSO focuses on how to use the direct method to process pixels with intensity gradients in the image. Since no closed loop is achieved, DSO is more regarded as a module in the SLAM system. Before developing DSO, the researchers conducted an in-depth study of photometric calibration and believed that the robustness of the direct method can be significantly improved by accurately calibrating the camera's exposure time, vignetting, gamma response and other parameters. DSO incorporates the photometric error model and all model parameters into the optimization function for joint optimization, and proposes a complete photometric calibration method, which has shown good accuracy and speed on multiple data sets. In addition, researchers have also studied the binocular version of DSO. Although there is no open source code, other researchers have expanded on this basis and tried to add loop detection and map reuse modules to DSO to improve the overall performance and practicality of the system. These developments fully demonstrate the technical progress and application prospects of direct methods in the field of visual SLAM.

[0083] In the development of direct visual positioning and SLAM algorithms, these technologies require online photometric calibration to achieve efficient photometric invariance processing. Traditionally, many visual systems rely on the assumption of constant image brightness, but in automatic exposure video capture, this assumption often does not hold, resulting in the need for real-time adjustment of photometric parameters to adapt to the changing visual environment. In past studies, although a variety of photometric calibration methods have been proposed, such as multi-image approximation and linear optimization, these methods are usually only applicable to offline applications such as panoramic stitching and photo synthesis, and cannot effectively cope with dynamic changes in video sequences. The limitation of these methods is that they often require image alignment or assume no vignetting effects, which is often unrealistic in practical applications.

[0084] Therefore, new solutions are needed in this field that can automatically process and correct the camera's video data for exposure in real time without relying on static image alignment and complex preprocessing, thereby improving the accuracy and robustness of SLAM and visual positioning systems.

[0085] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0086] In one embodiment, Figure 1 As shown, a third-order dynamic photometric calibration method using an estimation-optimization and a two-layer processing architecture is provided to achieve photometric calibration of a target video sequence, including:

[0087] Step S1, selecting at least one target key frame in the target video sequence, using KLT feature tracking technology to capture the pixel intensity of the field light in the target key frame, solving the optical flow vector by estimating the movement direction and speed of each point in the target key frame image, and determining the video frame sample set based on the optical flow vector;

[0088] Step S2, establishing an estimation-optimization network structure based on optical flow and physical constraints according to the video frame sample set, preliminarily estimating a photometric parameter set, setting constraints, and optimizing the photometric parameter set using the least squares method, substituting the optimized photometric parameter set into a photometric calibration formula, and using the obtained photometric parameter set result to perform photometric calibration on the video frame to obtain a photometric feature calibration value; iterating the estimation-optimization network structure to obtain an estimation-optimization framework with the video frame in the video frame sample set as input and the photometric feature calibration value as output; wherein the photometric parameter set includes CRF, exposure time, and vignetting effect parameters;

[0089] Step S3, establish a two-layer processing framework for dynamic adjustment of photometry including a front end and a back end; wherein the front end of the two-layer processing framework for dynamic adjustment of photometry includes an estimation-optimization framework for real-time processing of received video frames and outputting photometric feature calibration values ​​to the back end; the back end of the two-layer processing framework for dynamic adjustment of photometry outputs photometric feature calibration values ​​to video frames, and simultaneously iterates the photometric parameter set and transmits it to the front end.

[0090] In an exemplary embodiment, Figure 2 As shown in Figure 2, a third-order dynamic photometric calibration, estimation-optimization and two-layer processing architecture method for visual SLAM is proposed, which is divided into the following Figure 2 The three steps shown include:

[0091] The key frames with significant exposure changes are selected, and the Kanade-Lucas-Tomasi (KLT) feature tracking technique is applied to capture the pixel intensity of scene point lights in the auto-exposure video sequence, and then the optical flow vector is solved by estimating the motion direction and speed of each point in the image.

[0092] Establishing an estimation-optimization framework based on optical flow and physical constraints, initializing parameters of camera response function (CRF), exposure time and vignetting effect, setting constraints, and optimizing parameters using least square method, and substituting the optimized parameters into the photometric calibration formula to improve the system; illustratively, the system includes an estimation-optimization framework based on optical flow and physical constraints;

[0093] A two-layer processing framework for dynamic adjustment of photometry based on the fusion of real-time and optimization is run. The front end processes the received video frames in real time and applies photometry calibration parameters. The background runs the optimization algorithm, analyzes the data and updates the calibration parameters to maintain the calibration accuracy of the system. Exemplarily, the system includes a two-layer processing framework for dynamic adjustment of photometry based on the fusion of real-time and optimization. Among them, the parameters of the camera response function, exposure time and vignetting effect are the photometry calibration parameters.

[0094] In this embodiment, an online photometric calibration technique is designed to quickly estimate CRF, vignetting effect and exposure value. By utilizing exposure data acquired in real time, this method avoids the reliance on ground truth data and adapts to unknown or changing lighting conditions. At the same time, the use of an optimization framework improves the efficiency and stability of parameter estimation, enabling the system to perform photometric calibration online, which is suitable for video sequence processing and real-time vision applications.

[0095] In one embodiment, in step S1, an exposure difference threshold is preset, and a key frame with a significant exposure difference is selected, that is, a key frame with an exposure difference reaching a preset threshold. The gain-robust KLT feature tracking technology is used in the selected key frame. The basis of this algorithm is an optimization problem based on minimizing the error in the optical flow equation, and its brightness constant equation is shown in formula (1). Exemplarily, in order to ensure the image processing capability of the visual SLAM system in various environments to ensure the accuracy and stability of the system, based on this, this embodiment proposes a third-order dynamic photometric calibration method and an estimation-optimization framework for visual SLAM, the purpose of which is to improve the image processing capability of the SLAM system in complex environments, as shown in formula (1):

[0096] I(x,y,t)=I(x+dx,y+dy,t+dt) (1)

[0097] In formula (1), I(x,y,t) is the pixel intensity at position (x,y) at time t, and dx and dy are the displacement vectors from time t to t+d. For example, I(x,y,t) is the pixel intensity at position (x,y) at scene point t, and (dx,dy) is the displacement vector from time t to t+dt. This equation is based on the assumption that the brightness of an object is constant over a short period of time, even if its position in the image changes. This is the basic assumption of optical flow calculation.

[0098] Then, Taylor expansion is used for approximation, as shown in formula (2):

[0099]

[0100] In formula (2), the partial derivative and Represents the gradient of the image in the x and y directions. This method approximates the change in pixel value of the image under small movement. Through Taylor expansion, the optical flow problem can be simplified into a linear equation, which is easy to calculate and solve. Then, the spatial and temporal gradients of the image brightness are linked to the optical flow vector to form a solvable equation to estimate the movement speed and direction of each pixel. This equation is the optical flow equation, as shown in formula (3):

[0101]

[0102] In formula (3), v x and v y are the x and y components of the optical flow vector, is the rate of change of image pixel intensity over time.

[0103] Finally, the difference between the actual image and the image predicted by the optical flow is quantified by defining an error function, which is shown in formula (4):

[0104] ∈=∑ (x,y)∈W [I(x,y,t)-I(x+dx,y+dy,t+dt)] 2 (4)

[0105] In formula (4), W is the window centered on the feature point. By solving the displacement vector that minimizes the error function, the optical flow vector that is most likely to represent the movement of the actual object is found, and then iterative optimization is performed to gradually fine-tune until the algorithm converges to the optimal solution and obtains a video frame sample set.

[0106] In one embodiment, if Figure 3 As shown, the solution steps of the fast estimation-optimization framework in step S2 include:

[0107] Step S2.1, preliminarily estimate the CRF, exposure time and vignetting effect parameters in the photometric parameter set respectively.

[0108] Exemplarily, a quick preliminary estimation of photometric parameters is performed based on key frames, including CRF, exposure time, and vignetting effect. Specifically, each photometric parameter is estimated independently. For CRF estimation, the image intensity value M is obtained; for exposure time estimation, the exposure time e is obtained; for vignetting effect estimation, the vignetting effect parameter v is obtained. The input includes M 1 、M 2 、e 1 、e 2 、v 1 、v 2 The photometric parameters, including , are substituted into the photometric correction formula for calculation. After solving the matrix, linearization processing and solution are performed to output the preliminary estimated value.

[0109] Step S2.2, based on the results of the preliminary estimation of the photometric parameter set, set the estimation-optimization network structure; adjust the photometric parameter set by the nonlinear least squares method and introduce physical constraints; add regularization terms and target linear constraints in the process of adjusting the photometric parameter set by the nonlinear least squares method and introducing physical constraints, and use the gradient descent method to solve the nonlinear least squares method, calculate the gradient and update the photometric parameter set, stop the iteration after reaching the preset threshold, and obtain the optimized photometric parameter set.

[0110] Specifically, based on the results of the fast preliminary estimation, an estimation-optimization framework is designed to further fine-tune the parameters of CRF, exposure time, and vignetting model through nonlinear least squares. All relevant parameters are considered in the optimization process, and practical physical constraints are introduced. The optimization process is extended to include regularization terms and additional linear constraints, and the gradient descent method is applied to solve the nonlinear least squares problem, calculate the gradient and update the parameters to stop the iteration after reaching a preset threshold.

[0111] Step S2.3, substitute the optimized photometric parameter set into the photometric calibration formula to obtain the results of each photometric parameter in the photometric parameter set, use the results of the photometric parameter set to perform photometric calibration on the video frame, and obtain the photometric feature calibration value; iteratively estimate and optimize the network structure to obtain an estimation-optimization framework with the video frames in the video frame sample set as input and the photometric feature calibration value as output.

[0112] Specifically, the parameters that are initially estimated and optimized are substituted into the photometric calibration formula to improve the system's ability to process images.

[0113] In one embodiment, the solution steps of the fast estimation-optimization framework in step S2.1 are as follows:

[0114] Step S2.1.1, performing a preliminary estimation of each photometric parameter in a photometric parameter set, where the photometric parameter set includes CRF, exposure time, and vignetting effect parameters;

[0115] Specifically, each photometric parameter is first independently estimated quickly and preliminarily, including CRF, exposure time and vignetting effect; the expression of the camera response function CRF is shown in formula (5):

[0116] L = f -1 (M) (5)

[0117] In formula (5), L is the scene radiation and M is the image intensity value, which describes how the camera converts the captured scene radiation L into the image intensity value M. Its specific expression is shown in formula (6):

[0118] f -1 (M) = c 0 +c 1 M+c 2 M 2 +c 3 M 3 (6)

[0119] In formula (6), c 0 , c 1 , c 2 , c 3 are the coefficients of the CRF polynomial to be estimated, which are the parameters of the model and are used to describe the response characteristics of the camera. 0 is a constant term used to describe the basic offset between scene brightness and image brightness and quantify the camera response characteristics. 1 It is a linear coefficient, which is used to describe the linear relationship between the image brightness M and the scene brightness. 2 is the coefficient of the quadratic term, describing a slightly nonlinear change. 3 is the cubic coefficient, which handles the high-order nonlinear response under complex lighting.

[0120] Step S2.1.2, the exposure time parameter directly affects the value of the measured intensity M. The adjustment of the exposure time e can be considered in combination with the CRF through the following mathematical model, as shown in formula (7):

[0121]

[0122] In formula (7), e represents the exposure time, and L(t) is the radiation intensity of the scene at time t.

[0123] Step S2.1.3, at the same time, the initial parameters of the vignetting effect are estimated, and the vignetting effect function is shown in formula (8):

[0124] V(R)=1+v 1 R 2 +v 2 R 4 +v 3 R 6 (8)

[0125] In formula (8), R represents the normalized distance to the center of the image, v 1 v is the coefficient of the quadratic term in the vignetting polynomial, which is mainly responsible for simulating the illumination changes within a small radius near the center of the image; 2 is the coefficient of the fourth order term, used to adjust the ratio v 1 describes the illumination variation at a farther radius, which plays a dominant role in the middle area of ​​the image; v 3 It is the sixth-order coefficient, which mainly affects the peripheral areas of the image, especially those close to the edges. It helps to more accurately adjust the light attenuation caused by lens shading.

[0126] In one embodiment, the introduction of physical constraints in step S2.2 includes:

[0127] Monotonic physical constraints are added to ensure that as the input (i.e., light intensity) increases, the CRF output should also increase accordingly. Its expression is shown in formula (9):

[0128]

[0129] Formula (9) requires that the CRF function increases monotonically with M, that is, its derivative should always be greater than zero.

[0130] Non-negativity constraints are added to ensure that the output of the model will not have unrealistic negative values ​​under any circumstances. Its expression is shown in formula (10):

[0131]

[0132] Formula (10) requires that the output of CRF is always non-negative.

[0133] In this embodiment, by adding these constraints to the optimization model, the algorithm can be guided to avoid solutions that are mathematically feasible but physically unreasonable.

[0134] In one embodiment, step S2.2 comprises:

[0135] Step S2.2.1, introduce the results of the preliminary estimation of the photometric parameter set into the estimation-optimization network structure, adjust the parameters of CRF, exposure time and vignetting model by nonlinear least squares method, jointly consider all relevant parameters, and consider the actual physical constraints, where the parameters of the vignetting model are the vignetting effect parameters. The optimization process is expanded to include regularization terms and additional linear constraints.

[0136] Specifically, the mathematical formula of the least squares method is extended from the basic least squares problem, and its expression is shown in formula (11):

[0137]

[0138] In formula (11), A is the target matrix, c is the coefficient to be estimated, and b is the observed data vector. The purpose of this formula is to find the parameter c so that the sum of the squares of the differences between the predicted value Ac and the actual value b is minimized.

[0139] Step S2.2.2, in order to introduce more flexibility and control over the problem, the formula is expanded by adding a regularization term as shown in formula (12):

[0140]

[0141] In formula (12), λ is a regularization parameter used to adjust the influence of the regularization term, while E and d represent additional linear constraints to meet specific camera and environment characteristics;

[0142] Next, the gradient descent method is used to solve the nonlinear least squares problem, and its mathematical expression is shown in formula (13):

[0143] J(c)=c T A T Ac-2b T Ac+λ(Ec-d) T (Ec-d) (13)

[0144] In formula (13), the update rule of the gradient descent method is shown in formula (14):

[0145]

[0146] In formula (14), α is the learning rate, which determines the step size of each update to ensure convergence.

[0147] Repeat the photometric parameter set update until it reaches the threshold preset by this framework and stops to obtain the optimized photometric parameter set.

[0148] In one embodiment, step S2.3 of substituting the optimized photometric parameter set into the photometric calibration formula includes:

[0149] Substituting the initially estimated and optimized parameters into the photometric calibration formula improves the image processing capability of the system. Its mathematical expression is shown in formula (15):

[0150]

[0151] In formula (15), M 1 , M 2 is the measured intensity under different exposure conditions, e 1 , e 2 is the corresponding exposure time;

[0152] Then, after linearization, this proportional relationship is re-expressed in matrix form to solve the CRF and vignetting parameters, as shown in formula (16):

[0153]

[0154] in,

[0155] In this embodiment, after obtaining the preliminary estimates of each parameter, the system will use these preliminary values ​​as input to provide the basis for the optimization framework of step S2.3. The key to this method is that it can quickly generate preliminary, real-time photometric parameter estimates, even if these estimates may not reach the final required accuracy, they are sufficient to provide effective adaptability for the system in a dynamically changing environment.

[0156] In one embodiment, step S3 comprises:

[0157] Establish a two-layer processing framework for dynamic photometric adjustment including front-end and back-end;

[0158] The front end of the dual-layer processing framework for dynamic photometric adjustment includes an estimation-optimization framework for real-time processing of received video frames and photometric calibration using a photometric parameter set, and outputting photometric feature calibration values ​​to the back end;

[0159] The back end of the dual-layer processing framework for dynamic photometric adjustment receives photometric feature calibration values ​​and outputs the photometric feature calibration values ​​to video frames. At the same time, the back end updates and iterates the photometric parameter set and transmits the updated and iterated photometric parameter set to the front end.

[0160] For example, Figure 4 As shown in the figure, after being improved by the estimation-optimization framework, the designed two-layer processing architecture is run. The system front end receives the video frame and performs photometric calibration, and then wirelessly transmits the calibrated result to the system back end, which performs data optimization and updates parameters, and then inputs the parameters back to the front end and outputs the correction result at the same time; a loop is set to continue receiving and processing new video frames.

[0161] Specifically, the front end processes the received video frames in real time and applies the photometric calibration parameters, while the background runs the optimization algorithm to analyze the data and update the calibration parameters to maintain the system calibration accuracy. This operation quickly adjusts the photometric features of the image, such as brightness and contrast, to adapt to the changing ambient lighting, thereby maintaining the continuity and reliability of the visual information. Its mathematical expression is shown in formula (17):

[0162] M'=a 0 +a 1 M raw (17)

[0163] In formula (17), M raw is the brightness value of the original image frame, a 0 and a 1 is the current optimal photometric calibration parameter received from the backend, used to adjust the image brightness. M' is the corrected image brightness value.

[0164] At the back end of the system, optimization algorithms are used to perform in-depth analysis on the accumulated large amounts of image data. Through advanced mathematical models and optimization techniques, the back end continuously evaluates and adjusts the photometric calibration parameters. The mathematical model of the back end can be expressed as an optimization problem, the goal of which is to minimize the prediction error and achieve physical constraints, as shown in formula (18):

[0165]

[0166] In formula (18), f(c,M i ) is the predicted image brightness calculated based on the current parameter c, M' i is the measured image brightness value, R(c) is a regularization term containing physical constraints, namely monotonicity and non-negativity constraints. c is a vector containing all calibration parameters. The data processing results of the backend are periodically updated to the frontend, and new calibration parameters are periodically sent through the network or data link. This dynamic update mechanism of parameters enables the frontend to obtain optimized calibration parameters in a timely manner, ensuring the system's rapid adaptability to environmental changes. This two-layer architecture design enables the system to ensure real-time response while also improving the stability and accuracy of long-term operation through continuous backend processing, and adapting to various complex application scenarios.

[0167] This application has the following technical effects: an estimation-optimization framework based on optical flow and physical constraints is designed, which is more suitable for real-time application scenarios; a two-layer processing architecture for dynamic adjustment of photometric information based on the fusion of real-time and optimization is designed, providing a continuous and adaptive photometric calibration strategy. Specifically, this application, by designing a framework based on optical flow and physical constraints, enables the visual SLAM system to better cope with image quality fluctuations caused by changes in illumination, thereby improving the accuracy of positioning and mapping under different lighting conditions. At the same time, a two-layer processing framework for dynamic adjustment of photometric information based on the fusion of real-time and optimization is designed. By separating real-time processing and back-end optimization, it not only ensures a fast response to input video frames, but also enables in-depth data analysis and parameter optimization. This design allows the system to continuously improve processing quality and accuracy through background algorithms while maintaining high real-time performance.

[0168] At least a portion of the steps in the flowcharts involved in the embodiments described above may include multiple steps or multiple stages. These steps or stages may be executed at different times, or may be executed in turn or alternation with other steps or at least a portion of the steps or stages in other steps.

[0169] Based on the same inventive concept, the embodiment of the present application also provides a third-order dynamic photometric calibration device using an estimation-optimization and double-layer processing architecture for implementing the third-order dynamic photometric calibration method using an estimation-optimization and double-layer processing architecture involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the third-order dynamic photometric calibration device using an estimation-optimization and double-layer processing architecture provided below can refer to the limitations of the third-order dynamic photometric calibration method using an estimation-optimization and double-layer processing architecture above, and will not be repeated here.

[0170] In an exemplary embodiment, Figure 5 As shown, a third-order dynamic photometric calibration device using an estimation-optimization and double-layer processing architecture is provided, comprising:

[0171] The video frame sample set determination module 501 is used to select at least one target key frame in the target video sequence, capture the pixel intensity of the field light in the target key frame by using the KLT feature tracking technology, solve the optical flow vector by estimating the movement direction and speed of each point in the target key frame image, and determine the video frame sample set based on the optical flow vector;

[0172] The estimation-optimization framework iteration module 502 is used to establish an estimation-optimization network structure based on optical flow and physical constraints according to the video frame sample set, preliminarily estimate the photometric parameter set, set the constraint conditions, and use the least square method to optimize the photometric parameter set, substitute the optimized photometric parameter set into the photometric calibration formula, and use the result of the obtained photometric parameter set to perform photometric calibration on the video frame to obtain the photometric feature calibration value; iterate the estimation-optimization network structure to obtain an estimation-optimization framework with the video frame in the video frame sample set as input and the photometric feature calibration value as output; wherein the photometric parameter set includes CRF, exposure time and vignetting effect parameter;

[0173] The photometric dynamic adjustment double-layer processing framework training module 503 is used to establish a photometric dynamic adjustment double-layer processing framework including a front end and a back end; wherein the front end of the photometric dynamic adjustment double-layer processing framework includes an estimation-optimization framework, which is used to process the received video frames in real time and output photometric feature calibration values ​​to the back end; the back end of the photometric dynamic adjustment double-layer processing framework outputs photometric feature calibration values ​​to the video frames, and simultaneously iterates the photometric parameter set and transmits it to the front end.

[0174] Each module in the third-order dynamic photometric calibration device using the estimation-optimization and dual-layer processing architecture can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0175] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0176] For those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these modifications and improvements are all within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A third-order dynamic photometric calibration method using an estimation-optimization and dual-layer processing architecture, characterized in that: Used to implement photometric calibration of a target video sequence, the method comprises: Step S1, selecting at least one target key frame in the target video sequence, using KLT feature tracking technology to capture the pixel intensity of the field light in the target key frame, solving the optical flow vector by estimating the movement direction and speed of each point in the target key frame image, and determining the video frame sample set based on the optical flow vector; Step S2, establishing an estimation-optimization network structure based on optical flow and physical constraints according to the video frame sample set, preliminarily estimating a photometric parameter set, setting constraints, and optimizing the photometric parameter set using the least squares method, substituting the optimized photometric parameter set into a photometric calibration formula, and using the obtained photometric parameter set result to perform photometric calibration on the video frame to obtain a photometric feature calibration value; iterating the estimation-optimization network structure to obtain an estimation-optimization framework with the video frame in the video frame sample set as input and the photometric feature calibration value as output; wherein the photometric parameter set includes CRF, exposure time, and vignetting effect parameters; Step S3, establish a two-layer processing framework for dynamic adjustment of photometry including a front end and a back end; wherein the front end of the two-layer processing framework for dynamic adjustment of photometry includes an estimation-optimization framework for real-time processing of received video frames and outputting photometric feature calibration values ​​to the back end; the back end of the two-layer processing framework for dynamic adjustment of photometry outputs photometric feature calibration values ​​to video frames, and simultaneously iterates the photometric parameter set and transmits it to the front end.

2. The method according to claim 1, characterized in that The step S1 comprises: At least one target key frame whose exposure difference reaches a preset threshold is selected in the target video sequence. The KLT feature tracking technology is used in the selected target key frame. The position of the scene point is represented by a coordinate system. The brightness constant equation is shown in formula (1): I(x,y,t)=I(x+dx,y+dy,t+dt) (1) In formula (1), I(x, y, t) is the pixel intensity at position (x, y) at time t, dx and dy are the displacement vectors from time t to t+dt; Through Taylor expansion approximation, as shown in formula (2): In formula (2), the partial derivative and Represents the gradient of the image in the horizontal and vertical directions of the coordinate system; The spatial and temporal gradients of the image brightness in the target key frame are linked to the optical flow vector to form the optical flow equation, as shown in formula (3): In formula (3), v x and v y are the components of the optical flow vector in the horizontal and vertical directions of the specified coordinate system, is the rate of change of image pixel intensity over time; By defining an error function, the difference between the actual target key frame image and the target key frame image predicted by the optical flow is quantified. The error function is shown in formula (4): ∈=∑ (x,y)∈W [I(x,y,t)-I(x+dx,y+dy,t+dt) 2 (4) In formula (4), W is the window centered on the feature point; By solving the displacement vector that minimizes the error function, the optical flow vector representing the movement of the actual object is determined, and then iterative optimization is performed until the algorithm converges to the optimal solution to obtain a video frame sample set.

3. The method according to claim 2, characterized in that The step S2 comprises: Step S2.1, respectively preliminarily estimating the CRF, exposure time and vignetting effect parameters in the photometric parameter set; Step S2.2, based on the result of the preliminary estimation of the photometric parameter set, set the estimation-optimization network structure; adjust the photometric parameter set by nonlinear least squares method and introduce physical constraints; add regularization terms and target linear constraints in the process of adjusting the photometric parameter set by nonlinear least squares method and introducing physical constraints, and use gradient descent method to solve the nonlinear least squares method, calculate the gradient and update the photometric parameter set, stop iteration after reaching a preset threshold, and obtain the optimized photometric parameter set; Step S2.3, substitute the optimized photometric parameter set into the photometric calibration formula to obtain the results of each photometric parameter in the photometric parameter set, use the results of the photometric parameter set to perform photometric calibration on the video frame, and obtain the photometric feature calibration value; iteratively estimate and optimize the network structure to obtain an estimation-optimization framework with the video frames in the video frame sample set as input and the photometric feature calibration value as output.

4. The method according to claim 3, characterized in that The step S2.1 comprises: Step S2.1.1, performing a preliminary estimation of each photometric parameter in a photometric parameter set, where the photometric parameter set includes CRF, exposure time, and vignetting effect parameters; Among them, the expression of CRF is shown in formula (5): L = f -1 (M) (5) In formula (5), L is the scene radiation, M is the image intensity value, and the specific expression is shown in formula (6): f -1 (M) = c0 + c1M + c2M 2 +c3M 3 (6) In formula (6), c0, c1, c2, and c3 are the coefficients of the CRF polynomial to be estimated; c0 is a constant term used to describe the basic offset between scene brightness and image brightness; c1 is a linear term coefficient used to describe the linear relationship between image brightness M and scene brightness; c2 is a quadratic term coefficient used to describe slight nonlinear changes; and c3 is a cubic term coefficient used to handle high-order nonlinear responses under complex illumination. In step S2.1.2, the exposure time e is adjusted as shown in formula (7): In formula (7), e is the exposure time, L(t) is the radiation intensity of the scene at time t; Step S2.1.3, the vignetting effect function is shown in formula (8): V(R)=1+v1R 2 +v2R 4 +v3R 6 (8) In formula (8), R is the normalized distance to the center of the image, v1 is the quadratic term coefficient in the vignetting polynomial, which is used to simulate the illumination changes within a smaller radius near the center of the image; v2 is the quartic term coefficient, which is used to adjust the illumination changes at a radius farther than that described by v1; and v3 is the sixth term coefficient, which affects the peripheral area of ​​the image.

5. The method according to claim 4, characterized in that The physical constraints introduced in step S2.2 include: Monotonic physical constraints are added to ensure that as the input increases, the CRF output should also increase accordingly. The expression is shown in formula (9): Formula (9) requires that the CRF function increases monotonically with M, that is, the derivative should always be greater than zero; The non-negativity constraint is added to ensure that the output will not have unrealistic negative values ​​under any circumstances. The expression is shown in formula (10): Formula (10) requires that the output of CRF is always non-negative.

6. The method according to claim 5, characterized in that The step S2.2 comprises: Step S2.2.1, introduce the result of the preliminary estimation of the photometric parameter set into the estimation-optimization network structure, and adjust the CRF, exposure time and vignetting effect parameters through the nonlinear least squares method. The least squares method expression is shown in formula (11): In formula (11), A is the target matrix, b is the observed data vector, and c is the coefficient to be estimated. This formula is used to determine the parameter c so that the sum of the squares of the differences between the predicted value Ac and the actual value b is minimized; In step S2.2.2, the formula is expanded by adding regularization terms and target linear constraints as shown in formula (12): In formula (12), λ is the regularization parameter used to adjust the influence of the regularization term, and E and d represent additional linear constraints; The gradient descent method is used to solve the nonlinear least squares method, and the expression is shown in formula (13): J(c)=c T A T Ac-2b T Ac+λ(Ec-d) T (Ec-d) (13) In formula (13), the update rule of the gradient descent method is shown in formula (14): In formula (14), α is the learning rate, which determines the step size of each update to ensure convergence; The photometric parameter set update is repeated until a preset threshold is reached and then the optimized photometric parameter set is obtained.

7. The method according to claim 6, characterized in that Substituting the optimized photometric parameter set into the photometric calibration formula in step S2.3 includes: Substitute the initially estimated and optimized photometric parameter set into the photometric calibration formula, as shown in formula (15): In formula (15), M1 and M2 are the measured intensities under different exposure conditions, and e1 and e2 are the corresponding exposure times; After linearization, the proportional relationship is re-expressed in matrix form, as shown in formula (16): in, 8. The method according to claim 7, characterized in that The step S3 comprises: Establish a two-layer processing framework for dynamic photometric adjustment including front-end and back-end; The front end of the dual-layer processing framework for dynamic photometric adjustment includes an estimation-optimization framework for real-time processing of received video frames and photometric calibration using a photometric parameter set, and outputting photometric feature calibration values ​​to the back end; The back end of the dual-layer processing framework for dynamic photometric adjustment receives photometric feature calibration values ​​and outputs the photometric feature calibration values ​​to video frames. At the same time, the back end updates and iterates the photometric parameter set and transmits the updated and iterated photometric parameter set to the front end.

9. A third-order dynamic photometric calibration device using an estimation-optimization and dual-layer processing architecture, characterized in that: The device comprises: A video frame sample set determination module is used to select at least one target key frame in the target video sequence, capture the pixel intensity of the field light in the target key frame by using the KLT feature tracking technology, solve the optical flow vector by estimating the movement direction and speed of each point in the target key frame image, and determine the video frame sample set based on the optical flow vector; The estimation-optimization framework iteration module is used to establish an estimation-optimization network structure based on optical flow and physical constraints according to the video frame sample set, preliminarily estimate the photometric parameter set, set the constraint conditions, and use the least square method to optimize the photometric parameter set, substitute the optimized photometric parameter set into the photometric calibration formula, and use the result of the obtained photometric parameter set to perform photometric calibration on the video frame to obtain the photometric feature calibration value; iterate the estimation-optimization network structure to obtain an estimation-optimization framework with the video frame in the video frame sample set as input and the photometric feature calibration value as output; wherein the photometric parameter set includes CRF, exposure time and vignetting effect parameters; The photometric dynamic adjustment double-layer processing framework training module is used to establish a photometric dynamic adjustment double-layer processing framework including a front end and a back end; wherein the front end of the photometric dynamic adjustment double-layer processing framework includes an estimation-optimization framework, which is used to process the received video frames in real time and output photometric feature calibration values ​​to the back end; the back end of the photometric dynamic adjustment double-layer processing framework outputs photometric feature calibration values ​​to the video frames, and simultaneously iterates the photometric parameter set and transmits it to the front end.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images