Third-order dynamic photometric calibration method using estimation-optimization and double-layer processing architecture

Abstract

The application relates to a third-order dynamic photometric calibration method, device and computer equipment adopting an estimation-optimization and double-layer processing architecture. The method comprises the following steps: selecting at least one target key frame in a target video sequence, and determining a video frame sample set; establishing an estimation-optimization network structure based on optical flow properties and physical constraints according to the video frame sample set, preliminarily estimating a photometric parameter set, setting a constraint condition, and performing photometric calibration on the video frame by using the obtained photometric parameter set, so as to obtain a photometric feature calibration value and an estimation-optimization framework; and establishing a photometric dynamic adjustment double-layer processing framework comprising a front end and a rear end, which is used for real-time processing of received video frames and output of the photometric feature calibration value, and simultaneously iterates the photometric parameter set and transmits the photometric parameter set to the front end. The method provides a continuous and adaptive photometric calibration strategy, and is more suitable for real-time application scenarios.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision and VSLAM (Visual SLAM, Simultaneous Localization and Mapping), and in particular to a third-order dynamic photometric calibration method, apparatus, and computer equipment using an estimation-optimization and dual-layer processing architecture. Background Art

[0002] In the development of direct visual localization and SLAM algorithms, these technologies require online photometric calibration to achieve efficient photometric invariance processing. Traditionally, many visual systems rely on the assumption of constant image brightness, but this assumption often does not hold in auto-exposure video capture, resulting in the need for real-time adjustment of photometric parameters to adapt to the changing visual environment. In past studies, although various photometric calibration methods have been proposed, such as multi-image approximation and linear optimization, these methods are generally only suitable for offline applications such as panoramic stitching and photo synthesis, and cannot effectively cope with dynamic changes in video sequences. The limitation of these methods is that they often require image alignment or assume no vignetting effects, which is often unrealistic in practical applications.

[0003] Therefore, new solutions are needed in this field that can process and correct the video data of the automatic exposure camera in real time without relying on static alignment of images and complex preprocessing, thereby improving the accuracy and robustness of SLAM and visual positioning systems. Summary of the Invention

[0004] Based on this, it is necessary to provide a third-order dynamic photometric calibration method, device and computer equipment that uses estimation-optimization and two-layer processing architecture, which can quickly adapt to lighting changes and automatically adjust photometric parameters, and is suitable for high-precision visual navigation and spatial reconstruction in dynamic lighting or uneven lighting environments.

[0005] In a first aspect, the present application provides a third-order dynamic photometric calibration method using an estimation-optimization and two-layer processing architecture, for achieving photometric calibration of a target video sequence, comprising:

[0006] Step S1, selecting at least one target key frame in the target video sequence, using KLT feature tracking technology to capture the pixel intensity of the field light in the target key frame, solving the optical flow vector by estimating the motion direction and speed of each point in the target key frame image, and determining a video frame sample set based on the optical flow vector;

[0007] Step S2, an estimation-optimization network structure based on optical flow properties and physical constraints is established according to the video frame sample set, a preliminary estimation of the set of photometric parameters is made, a constraint condition is set, and a least square method is used for optimization of the set of photometric parameters, the optimized set of photometric parameters is substituted into a photometric calibration formula, and a result of the set of photometric parameters is used to perform photometric calibration on the video frame to obtain a photometric feature calibration value; an estimation-optimization framework taking the video frame in the video frame sample set as input and the photometric feature calibration value as output is obtained by iterative estimation-optimization of the network structure; wherein the set of photometric parameters includes CRF, exposure time, and vignetting effect parameters;

[0008] Step S3, a photometric dynamic adjustment double-layer processing framework including a front end and a back end is established; wherein the front end of the photometric dynamic adjustment double-layer processing framework includes the estimation-optimization framework, which is used to process the received video frame in real time and output the photometric feature calibration value to the back end; the back end of the photometric dynamic adjustment double-layer processing framework outputs the photometric feature calibration value to the video frame, and iteratively updates the set of photometric parameters and transmits the set of photometric parameters to the front end.

[0009] In one embodiment, step S1 includes:

[0010] In the target video sequence, at least one target key frame with exposure difference reaching a preset threshold is selected, and a KLT feature tracking technology is used in the selected target key frame to represent the position of a scene point in a coordinate system, and a brightness constancy equation is as shown in formula (1):

[0011] I(x,y,t)=I(x+dx,y+dy,t+dt) (1)

[0012] In formula (1), I(x,y,t) is a pixel intensity of position (x,y) at time t, and dx and dy are displacement vectors from time t to t+dt;

[0013] Through Taylor expansion approximation, it is specifically as shown in formula (2):

[0014]

[0015] In formula (2), the partial derivatives and represent the gradients of the image in the horizontal and vertical directions of the coordinate system;

[0016] The spatial and temporal gradients of the image brightness in the target key frame are related to the optical flow vector to form an optical flow equation, which is specifically as shown in formula (3):

[0017]

[0018] In formula (3), v x and v yis the horizontal component of the optical flow vector in the specified coordinate system, is the rate of change of the image pixel intensity over time;

[0019] By defining an error function to quantify the difference between the actual target key frame image and the target key frame image predicted according to the optical flow, the error function is as shown in equation (4):

[0020] ∑ (x,y)∈W [I(x,y,t)-I(x+dx,y+dy,t+dt)] 2 (4)

[0021] In equation (4), W is a window centered on the feature point;

[0022] By solving the displacement vector that minimizes the error function, the optical flow vector representing the actual movement of the object is determined, and iterative optimization is performed until the algorithm converges to the optimal solution, and the video frame sample set is obtained.

[0023] In one embodiment, step S2 includes:

[0024] Step S2.1, preliminary estimates are made for the CRF, exposure time and vignetting effect parameters in the set of photometric parameters, respectively;

[0025] Step S2.2, based on the results of the preliminary estimates of the set of photometric parameters, the estimation-optimization network structure is set; the set of photometric parameters is adjusted by a nonlinear least squares method and physical constraints are introduced; regularization terms and target linear constraints are added during the adjustment of the set of photometric parameters by the nonlinear least squares method and the introduction of physical constraints, and a gradient descent method is used to solve the nonlinear least squares method, the gradient is calculated and the set of photometric parameters is updated, the iteration is stopped after reaching a preset threshold, and the optimized set of photometric parameters is obtained;

[0026] Step S2.3, the optimized set of photometric parameters is substituted into the photometric calibration formula to obtain the results of each photometric parameter in the set of photometric parameters, and the results of the set of photometric parameters are used to perform photometric calibration on the video frames to obtain photometric feature calibration values; the estimation-optimization network structure is iteratively estimated to obtain an estimation-optimization framework with video frames in the video frame sample set as input and photometric feature calibration values as output.

[0027] In one embodiment, step S2.1 includes:

[0028] Step S2.1.1, preliminary estimates are made for each photometric parameter in the set of photometric parameters, which includes the CRF, exposure time and vignetting effect parameters;

[0029] In which, the expression of CRF is as shown in equation (5):

[0030] L=f-1 (M) (5)

[0031] In formula (5), L is the scene radiance, and M is the image intensity value, and the specific expression is shown in formula (6):

[0032] f -1 (M)=c0+c1M+c2M 2 +c3M 3 (6)

[0033] In formula (6), c0, c1, c2, and c3 are coefficients of the CRF polynomial to be estimated; c0 is a constant term, which is used to describe the basic offset between the scene brightness and the image brightness; c1 is a first-order term coefficient, which is used to describe the linear relationship between the image brightness M and the scene brightness; c2 is a second-order term coefficient, which describes a slight nonlinear change; and c3 is a third-order term coefficient, which processes high-order nonlinear response under complex light.

[0034] In step S2.1.2, the adjustment of the exposure time e is shown in formula (7):

[0035]

[0036] In formula (7), e is the exposure time, and L(t) is the radiation intensity of the scene at time t.

[0037] In step S2.1.3, the vignetting effect function is shown in formula (8):

[0038] V(R)=1+v1R 2 +v2R 4 +v3R 6 (8)

[0039] In formula (8), R is a normalized distance to the center of the image, v1 is a second-order term coefficient in the vignetting polynomial, which is used to simulate the light change in a small radius range near the center of the image; v2 is a fourth-order term coefficient, which is used to adjust the light change of a farther radius than v1; and v3 is a sixth-order term coefficient, which affects the peripheral area of the image.

[0040] In one of the embodiments, the introduction of the physical constraint condition in step S2.2 includes:

[0041] The monotonicity physical constraint is added, which is used to ensure that the CRF output should also increase accordingly as the input increases, and the expression is shown in formula (9):

[0042]

[0043] In formula (9), it is required that the CRF function monotonically increases with M, that is, the derivative should always be greater than zero;

[0044] A non-negativity constraint is added to ensure that the output will not appear unrealistic negative value in any case, the expression is shown as formula (10):

[0045]

[0046] Formula (10) requires the output of CRF to be non-negative at all times.

[0047] In one embodiment, step S2.2 includes:

[0048] Step S2.2.1, the results of the preliminary estimation of the set of photometric parameters are introduced into the estimation-optimization network structure, and the CRF, exposure time and vignetting effect parameters are adjusted by a nonlinear least squares method, the expression of the least squares method is shown as formula (11):

[0049]

[0050] In formula (11), A is the target matrix, b is the observation data vector, and c is the coefficient to be estimated, and the formula is used to determine the parameter c so that the sum of squares of the difference between the predicted value Ac and the actual value b is minimized;

[0051] Step S2.2.2, the formula is extended as shown in formula (12) by adding a regularization term and a target linear constraint:

[0052]

[0053] In formula (12), λ is a regularization parameter for adjusting the influence of the regularization term, and E and d represent additional linear constraints;

[0054] The gradient descent method is used to solve the nonlinear least squares method, and the expression is shown as formula (13):

[0055] J(c)=c T A T Ac-2b T Ac+λ(Ec-d) T (Ec-d) (13)

[0056] In formula (13), the update rule of the gradient descent method is shown as formula (14):

[0057]

[0058] In formula (14), α is the learning rate, which determines the step size of each update to ensure convergence;

[0059] The photometric parameter set update is repeatedly performed until a preset threshold is reached and stopped, and the optimized photometric parameter set is obtained.

[0060] In one embodiment, step S2.3 of substituting the optimized photometric parameter set into the photometric calibration formula includes:

[0061] Substitute the initially estimated and optimized photometric parameter set into the photometric calibration formula, as shown in formula (15):

[0062]

[0063] In formula (15), M1 and M2 are the measured intensities under different exposure conditions, and e1 and e2 are the corresponding exposure times;

[0064] After linearization, the proportional relationship is re-expressed in matrix form, as shown in formula (16):

[0065]

[0066] in,

[0067] In one embodiment, step S3 includes:

[0068] Establish a two-layer processing framework for dynamic photometric adjustment including front-end and back-end;

[0069] The front end of the two-layer processing framework for dynamic photometric adjustment includes an estimation-optimization framework for real-time processing of received video frames and performing photometric calibration using a photometric parameter set, and outputting photometric feature calibration values ​​to the back end.

[0070] The backend of the dual-layer processing framework for dynamic photometric adjustment receives photometric feature calibration values ​​and outputs them to video frames. At the same time, the backend updates and iterates the photometric parameter set and transmits the updated photometric parameter set to the frontend.

[0071] In a second aspect, the present application further provides a third-order dynamic photometric calibration device using an estimation-optimization and dual-layer processing architecture, comprising:

[0072] A video frame sample set determination module is configured to select at least one target key frame in a target video sequence, capture the pixel intensity of the midfield light in the target key frame using KLT feature tracking technology, solve the optical flow vector by estimating the motion direction and speed of each point in the target key frame image, and determine the video frame sample set based on the optical flow vector;

[0073] An estimation-optimization framework iteration module is used to establish an estimation-optimization network structure based on optical flow and physical constraints based on a video frame sample set, preliminarily estimate a photometric parameter set, set constraints, and optimize the photometric parameter set using the least squares method. The optimized photometric parameter set is substituted into a photometric calibration formula, and the obtained photometric parameter set is used to perform photometric calibration on the video frame to obtain photometric feature calibration values. The estimation-optimization network structure is iterated to obtain an estimation-optimization framework with video frames in the video frame sample set as input and photometric feature calibration values ​​as output; wherein the photometric parameter set includes CRF, exposure time, and vignetting effect parameters.

[0074] The photometric dynamic adjustment dual-layer processing framework training module is used to establish a photometric dynamic adjustment dual-layer processing framework including a front end and a back end; the front end of the photometric dynamic adjustment dual-layer processing framework includes an estimation-optimization framework for real-time processing of received video frames and outputting photometric feature calibration values ​​to the back end; the back end of the photometric dynamic adjustment dual-layer processing framework outputs photometric feature calibration values ​​to video frames, and simultaneously iterates the photometric parameter set and transmits it to the front end.

[0075] In a third aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of a third-order dynamic photometric calibration method using an estimation-optimization and a two-layer processing architecture.

[0076] The third-order dynamic photometric calibration method, device and computer equipment using the estimation-optimization and double-layer processing architecture, by selecting at least one target key frame in the target video sequence, using the KLT feature tracking technology to capture the pixel intensity of the scene point light in the target key frame, solving the optical flow vector by estimating the motion direction and speed of each point in the target key frame image, and determining the video frame sample set based on the optical flow vector; the estimation-optimization network structure based on the optical flow property and physical constraints is established according to the video frame sample set, the photometric parameter set is preliminarily estimated, the constraint condition is set, and the photometric parameter set is optimized using the least square method, the optimized photometric parameter set is substituted into the photometric calibration formula, and the video frame is calibrated by using the photometric parameter set, and the photometric feature calibration value is obtained; the estimation-optimization network structure is iterated to obtain the estimation-optimization framework with the video frame in the video frame sample set as the input and the photometric feature calibration value as the output; wherein the photometric parameter set includes CRF, exposure time and vignetting effect parameters; the photometric dynamic adjustment double-layer processing framework including the front end and the back end is established; wherein the front end of the photometric dynamic adjustment double-layer processing framework includes the estimation-optimization framework, which is used for real-time processing of the received video frame and outputting the photometric feature calibration value to the back end; the back end of the photometric dynamic adjustment double-layer processing framework outputs the photometric feature calibration value to the video frame, and iterates the photometric parameter set and transmits it to the front end. This method can quickly adapt to light changes and automatically adjust the photometric parameters, and is suitable for high-precision visual navigation and spatial reconstruction in dynamic light or uneven light environment. By introducing the online sequential photometric calibration technology, the CRF, exposure time and vignetting effect are independently estimated and optimized, so as to provide stable and reliable image processing results under unknown or changing light conditions, and meet the high standard requirements of industrial applications. BRIEF DESCRIPTION OF DRAWINGS

[0077] Figure 1 A flowchart of a third-order dynamic photometric calibration method using an estimation-optimization and double-layer processing architecture in an embodiment;

[0078] Figure 2 A flowchart of a third-order dynamic photometric calibration, estimation-optimization and double-layer processing architecture method for visual SLAM in an embodiment;

[0079] Figure 3 A flowchart of an estimation-optimization framework based on optical flow property and physical constraints in an embodiment;

[0080] Figure 4 A schematic diagram of a photometric dynamic adjustment double-layer processing framework based on real-time and optimization fusion in an embodiment;

[0081] Figure 5 A module schematic diagram of a third-order dynamic photometric calibration device using an estimation-optimization and double-layer processing architecture in an embodiment. DETAILED DESCRIPTION

[0082] DTAM (Dense Tracking and Mapping, a real-time dense localization and tracking algorithm) is a monocular SLAM algorithm proposed in 2011 and is considered as the progenitor of direct method. DTAM effectively reduces the uncertainty of pose by performing probabilistic depth measurement on each pixel, and obtains dense map and camera pose through alignment of the entire image. However, this method requires GPU (Graphics Processing Unit) acceleration to meet its computational requirements. Based on the principle of DTAM, a direct tracking-based visual odometry system was further developed, which is the first real-time visual odometry (VO) system without using features. Subsequently, map optimization was integrated into the VO system, which was extended to LSD-SLAM (Large-Scale Direct Monocular SLAM, a direct method algorithm), becoming a real-time SLAM system without using features. LSD-SLAM generates a globally consistent map by directly registering the image luminosity and using a probabilistic model to represent the semi-dense depth map. As a direct method SLAM system, although LSD-SLAM supports binocular cameras and panoramic cameras, it is very sensitive to camera intrinsic parameters and exposure, and is less accurate than ORB-SLAM (Oriented FAST and Rotated BRIEF, a feature point method algorithm) and less fast than DSO. On this basis, researchers released DSO (Direct Sparse Odometry). DSO mainly focuses on how to process the pixels with intensity gradient in the image using direct method. Since no loop closure is implemented, DSO is more considered as a module in the SLAM system. Before developing DSO, researchers conducted in-depth research on photometric calibration and believed that accurate calibration of camera parameters such as exposure time, dark corner, and gamma response could significantly improve the robustness of direct method. DSO integrates the photometric error model and all model parameters into the optimization function for joint optimization and proposes a complete photometric calibration method, which has shown good accuracy and speed on multiple datasets. In addition, researchers have also studied binocular DSO, although there is no open source code, but other researchers have extended it on this basis and tried to add loop detection and map reuse modules to DSO to improve the overall performance and practicality of the system. These developments fully demonstrate the technical progress and application prospects of direct method in the field of visual SLAM.

[0083] In the development of direct visual positioning and SLAM algorithms, these techniques require online photometric calibration to achieve efficient photometric invariance processing. Traditionally, many vision systems rely on the assumption of constant image brightness, but in auto-exposure video capture, this assumption often does not hold, resulting in the need for real-time adjustment of photometric parameters to adapt to the changing visual environment. In past research, although various photometric calibration methods have been proposed, such as multi-image approximation, linear optimization, etc., these methods are usually only applicable to offline applications such as panorama stitching and photo synthesis, and cannot effectively cope with dynamic changes in video sequences. The limitation of these methods is that they often require image alignment or assume no vignetting effect, which is often unrealistic in practical applications.

[0084] Therefore, there is a need in the art for new solutions that can process and correct video data from auto-exposure cameras in real time without relying on image static alignment and complex preprocessing, thereby improving the accuracy and robustness of SLAM and visual positioning systems.

[0085] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0086] In one embodiment, as shown in Figure 1 A third-order dynamic photometric calibration method using an estimation-optimization and double-layer processing architecture is provided for implementing photometric calibration of a target video sequence, comprising:

[0087] Step S1, selecting at least one target key frame in the target video sequence, capturing the pixel intensity of the scene point light in the target key frame using KLT feature tracking technology, solving the optical flow vector by estimating the motion direction and speed of each point in the target key frame image, and determining the video frame sample set based on the optical flow vector;

[0088] Step S2, establishing an estimation-optimization network structure based on optical flow properties and physical constraints according to the video frame sample set, preliminarily estimating the photometric parameter set, setting the constraint condition, and using the least squares method to optimize the photometric parameter set, substituting the optimized photometric parameter set into the photometric calibration formula, and using the resulting photometric parameter set to perform photometric calibration on the video frame to obtain the photometric feature calibration value; iteratively estimating the optimization network structure to obtain an estimation-optimization framework with the video frames in the video frame sample set as input and the photometric feature calibration value as output; wherein the photometric parameter set includes CRF, exposure time and vignetting effect parameters;

[0089] Step S3, a photometric dynamic adjustment double-layer processing framework including a front end and a back end is established; wherein the front end of the photometric dynamic adjustment double-layer processing framework includes an estimation-optimization framework, which is used for processing the received video frames in real time and outputting photometric feature calibration values to the back end; the back end of the photometric dynamic adjustment double-layer processing framework outputs the photometric feature calibration values to the video frames, and iterates a photometric parameter set and transmits it to the front end.

[0090] In one exemplary embodiment, as shown in Figure 2 A third-order dynamic photometric calibration, estimation-optimization and double-layer processing architecture method for visual SLAM is proposed, which is divided into three steps as shown in Figure 2 The specific steps include:

[0091] Key frames with significant exposure changes are selected, the Kanade-Lucas-Tomasi (KLT) feature tracking technology is applied, the pixel intensity of scene point light in the auto exposure video sequence is captured, and the optical flow vector is solved by estimating the motion direction and speed of each point in the image;

[0092] An estimation-optimization framework based on optical flow properties and physical constraints is established, the parameters of the camera response function (CRF), exposure time and vignetting effect are initialized, the constraint conditions are set, the least square method is used for parameter optimization, and the optimized parameters are substituted into the photometric calibration formula to improve the system; an exemplary system includes an estimation-optimization framework based on optical flow properties and physical constraints;

[0093] A photometric dynamic adjustment double-layer processing framework based on real-time and optimization fusion is run, the front end processes the received video frames in real time and applies photometric calibration parameters, the optimization algorithm is run in the background, the data is analyzed and the calibration parameters are updated, and the system calibration accuracy is maintained; an exemplary system includes a photometric dynamic adjustment double-layer processing framework based on real-time and optimization fusion; wherein the parameters of the camera response function, exposure time and vignetting effect are photometric calibration parameters.

[0094] In this embodiment, an online photometric calibration technology is designed to quickly estimate the CRF, vignetting effect and exposure value. By using the real-time exposure data, this method avoids the dependence on ground truth data and adapts to unknown or changing lighting conditions. At the same time, the use of the optimization framework improves the efficiency and stability of parameter estimation, so that the system can perform online photometric calibration and is suitable for video sequence processing and real-time visual applications.

[0095] In one embodiment, in step S1, an exposure difference threshold is preset, and key frames with significant exposure differences are selected, that is, key frames with exposure differences reaching the preset threshold. The gain-robust KLT feature tracking technology is used in the selected key frames. The basis of this algorithm is an optimization problem based on minimizing the error in the optical flow equation, and its brightness constant equation is shown in formula (1). For example, in order to ensure the image processing capability of the visual SLAM system in various environments to ensure the accuracy and stability of the system, based on this, this embodiment proposes a third-order dynamic photometric calibration method and an estimation-optimization framework for visual SLAM, the purpose of which is to improve the image processing capability of the SLAM system in complex environments, as shown in formula (1):

[0096] I(x,y,t)=I(x+dx,y+dy,t+dt) (1)

[0097] In formula (1), I(x,y,t) is the pixel intensity at position (x,y) at time t, and dx and dy are the displacement vectors from time t to t+d. For example, I(x,y,t) is the pixel intensity at position (x,y) at scene point t, and (dx,dy) is the displacement vector from time t to t+dt. This equation is based on the assumption that the brightness of an object is constant over a short period of time, even if its position in the image changes. This is the basic assumption of optical flow calculation.

[0098] Then, Taylor expansion is used for approximation, as shown in formula (2):

[0099]

[0100] In formula (2), the partial derivative and Represents the gradient of the image in the x and y directions. This method approximates the change in pixel value of the image under small movement. Through Taylor expansion, the optical flow problem can be simplified into a linear equation, which is easy to calculate and solve. Then, the spatial and temporal gradients of the image brightness are linked to the optical flow vector to form a solvable equation to estimate the movement speed and direction of each pixel. This equation is the optical flow equation, as shown in formula (3):

[0101]

[0102] In formula (3), v x and v y are the x and y components of the optical flow vector, is the rate of change of image pixel intensity over time.

[0103] Finally, the difference between the actual image and the image predicted by the optical flow is quantified by defining an error function, which is shown in formula (4):

[0104] ∈=∑ (x,y)∈W [I(x,y,t)-I(x+dx,y+dy,t+dt)] 2 (4)

[0105] In formula (4), W is the window centered on the feature point. By solving the displacement vector that minimizes the error function, the optical flow vector that is most likely to represent the actual object movement is found. Then, iterative optimization is performed and fine-tuned until the algorithm converges to the optimal solution, obtaining a video frame sample set.

[0106] In one embodiment, Figure 3 As shown, the solution steps of the fast estimation-optimization framework in step S2 include:

[0107] In step S2.1, preliminary estimation is performed on the CRF, exposure time and vignetting effect parameters in the photometric parameter set.

[0108] For example, a fast preliminary estimation of photometric parameters, including CRF, exposure time, and vignetting, is performed based on keyframes. Specifically, each photometric parameter is estimated independently. For CRF estimation, the image intensity value M is obtained; for exposure time estimation, the exposure time e is obtained; and for vignetting, the vignetting effect parameter v is obtained. The photometric parameters, including M1, M2, e1, e2, v1, and v2, are input and substituted into the photometric correction formula. After the matrix is ​​solved, linearization is performed and the solution is obtained, outputting a preliminary estimate.

[0109] Step S2.2, based on the results of the preliminary estimation of the photometric parameter set, set the estimation-optimization network structure; adjust the photometric parameter set through the nonlinear least squares method and introduce physical constraints; add regularization terms and target linear constraints in the process of adjusting the photometric parameter set through the nonlinear least squares method and introducing physical constraints, and use the gradient descent method to solve the nonlinear least squares method, calculate the gradient and update the photometric parameter set, stop the iteration after reaching the preset threshold, and obtain the optimized photometric parameter set.

[0110] Specifically, based on the results of a fast initial estimation, an estimation-optimization framework is designed to further fine-tune the parameters of the CRF, exposure time, and vignetting model using a nonlinear least-squares approach. All relevant parameters are considered in the optimization process, and realistic physical constraints are introduced. The optimization process is expanded to include regularization terms and additional linear constraints, and gradient descent is applied to solve the nonlinear least-squares problem. The gradient is calculated and the parameters are updated, with iterations stopping when a preset threshold is reached.

[0111] Step S2.3, substituting the optimized set of photometric parameters into the photometric calibration formula to obtain the results of each photometric parameter in the set of photometric parameters, and using the results of the set of photometric parameters to perform photometric calibration on the video frame to obtain a photometric feature calibration value; iteratively estimating and optimizing the network structure to obtain an estimation-optimization framework taking the video frames in the video frame sample set as input and the photometric feature calibration value as output.

[0112] Specifically, substituting the preliminarily estimated and optimized parameters into the photometric calibration formula to improve the ability of the system to process images.

[0113] In one embodiment, the solving step of the fast estimation-optimization framework in step S2.1 is as follows:

[0114] Step S2.1.1, preliminarily estimating each photometric parameter in the set of photometric parameters, the set of photometric parameters including CRF, exposure time, and vignetting effect parameters;

[0115] Specifically, first, independently preliminarily estimate each photometric parameter, including CRF, exposure time, and vignetting effect; wherein the expression of the camera response function CRF is as shown in formula (5):

[0116] L = f -1 (M) (5)

[0117] In formula (5), L is the scene radiance, M is the image intensity value, and describes how the camera converts the captured scene radiance L into the image intensity value M, and the specific expression is as shown in formula (6):

[0118] f -1 (M) = c0 + c1M + c2M 2 + c3M 3 (6)

[0119] In formula (6), c0, c1, c2, and c3 are coefficients of the CRF polynomial to be estimated, which are parameters of the model and are used to describe the response characteristics of the camera, c0 is a constant term used to describe the basic offset between scene brightness and image brightness, and quantifies the response characteristics of the camera. c1 is a linear term coefficient used to describe the linear relationship between image brightness M and scene brightness. c2 is a quadratic term coefficient describing a slight nonlinear change. c3 is a cubic term coefficient handling high-order nonlinear responses under complex lighting.

[0120] Step S2.1.2, the exposure time parameter directly affects the value of the measured intensity M, and the adjustment of the exposure time e can be considered in combination with the CRF through the following mathematical model, as shown in formula (7):

[0121]

[0122] In formula (7), e represents the exposure time, and L(t) is the radiation intensity of the scene at time t.

[0123] In step S2.1.3, at the same time, the initial parameters of the vignetting effect are estimated. The vignetting effect function is shown in formula (8):

[0124] V(R)=1+v1R 2 +v2R 4 +v3R 6 (8)

[0125] In formula (8), R represents the normalized distance to the center of the image, v1 is the quadratic term coefficient in the vignetting polynomial, which is mainly responsible for simulating the illumination changes within a smaller radius near the center of the image; v2 is the quartic term coefficient, which is used to adjust the illumination changes at a radius farther than that described by v1. In the middle area of ​​the image, it plays a dominant role; v3 is the sixth term coefficient, which mainly affects the peripheral area of ​​the image, especially the part close to the edge. It helps to more accurately adjust the illumination attenuation caused by lens shading.

[0126] In one embodiment, the introduction of physical constraints in step S2.2 includes:

[0127] A monotonic physical constraint is added to ensure that as the input (i.e., light intensity) increases, the CRF output should also increase accordingly. Its expression is shown in formula (9):

[0128]

[0129] Formula (9) requires that the CRF function increases monotonically with M, that is, its derivative should always be greater than zero.

[0130] The non-negativity constraint is added to ensure that under any circumstances, the output of the model will not have unrealistic negative values. Its expression is shown in formula (10):

[0131]

[0132] Formula (10) requires that the output of CRF is always non-negative.

[0133] In this embodiment, by adding these constraints to the optimization model, the algorithm can be guided to avoid solutions that are mathematically feasible but physically unreasonable.

[0134] In one embodiment, step S2.2 includes:

[0135] In step S2.2.1, the results of the preliminary estimation of the photometric parameter set are introduced into the estimation-optimization network structure. The parameters of the CRF, exposure time, and vignetting model are adjusted using nonlinear least squares methods. All relevant parameters are jointly considered, taking into account practical physical constraints. The parameters of the vignetting model are the vignetting effect parameters. The optimization process is expanded to include regularization terms and additional linear constraints.

[0136] Specifically, the mathematical formula of the least squares method is extended from the basic least squares problem, and its expression is shown in formula (11):

[0137]

[0138] In formula (11), A is the target matrix, c is the coefficient to be estimated, and b is the observed data vector. The purpose of this formula is to find the parameter c so that the sum of the squares of the differences between the predicted value Ac and the actual value b is minimized.

[0139] In step S2.2.2, in order to introduce more flexibility and control over the problem, the formula is expanded by adding a regularization term as shown in formula (12):

[0140]

[0141] In formula (12), λ is a regularization parameter used to adjust the influence of the regularization term, while E and d represent additional linear constraints to meet specific camera and environment characteristics;

[0142] Next, the gradient descent method is used to solve the nonlinear least squares problem, and its mathematical expression is shown in formula (13):

[0143] J(c)=c T A T Ac-2b T Ac+λ(Ec-d) T (Ec-d) (13)

[0144] In formula (13), the update rule of the gradient descent method is shown in formula (14):

[0145]

[0146] In formula (14), α is the learning rate, which determines the step size of each update to ensure convergence.

[0147] Repeat the photometric parameter set update until it reaches the threshold preset by this framework and then stops to obtain the optimized photometric parameter set.

[0148] In one embodiment, step S2.3 of substituting the optimized photometric parameter set into the photometric calibration formula includes:

[0149] Substituting the initially estimated and optimized parameters into the photometric calibration formula improves the system's image processing capability. Its mathematical expression is shown in formula (15):

[0150]

[0151] In formula (15), M1 and M2 are the measured intensities under different exposure conditions, and e1 and e2 are the corresponding exposure times;

[0152] Then, after linearization, this proportional relationship is re-expressed in matrix form to solve the CRF and vignetting parameters, as shown in formula (16):

[0153]

[0154] in,

[0155] In this embodiment, after obtaining preliminary estimates for each parameter, the system uses these preliminary values ​​as input to form the basis for the optimization framework in step S2.3. The key to this approach is the ability to rapidly generate preliminary, real-time estimates of the photometric parameters. Even if these estimates may not achieve the ultimate required accuracy, they are sufficient to provide effective adaptability to the system in a dynamically changing environment.

[0156] In one embodiment, step S3 includes:

[0157] Establish a two-layer processing framework for dynamic photometric adjustment including front-end and back-end;

[0158] The front end of the two-layer processing framework for dynamic photometric adjustment includes an estimation-optimization framework for real-time processing of received video frames and performing photometric calibration using a photometric parameter set, and outputting photometric feature calibration values ​​to the back end.

[0159] The backend of the dual-layer processing framework for dynamic photometric adjustment receives photometric feature calibration values ​​and outputs them to video frames. At the same time, the backend updates and iterates the photometric parameter set and transmits the updated photometric parameter set to the frontend.

[0160] For example, Figure 4 As shown in the figure, after the improvement of the estimation-optimization framework, the designed two-layer processing architecture is implemented. The system front end receives video frames and performs photometric calibration, then wirelessly transmits the calibration results to the system back end. The back end performs data optimization and parameter updates, then inputs the parameters back to the front end and outputs the calibration results. A loop is set up to continue receiving and processing new video frames.

[0161] Specifically, the front-end processes the received video frames in real time and applies the photometric calibration parameters. The back-end runs the optimization algorithm to analyze the data and update the calibration parameters to maintain the system calibration accuracy. This operation quickly adjusts the photometric characteristics of the image, such as brightness and contrast, to adapt to the changing ambient lighting, thereby maintaining the continuity and reliability of the visual information. Its mathematical expression is shown in formula (17):

[0162] M'=a0+a1M raw (17)

[0163] In formula (17), M raw is the brightness value of the original image frame, a0 and a1 are the current optimal photometric calibration parameters received from the backend, used to adjust the image brightness. M' is the corrected image brightness value.

[0164] At the back end of the system, optimization algorithms are used to perform in-depth analysis of the accumulated large amounts of image data. Using advanced mathematical models and optimization techniques, the back end continuously evaluates and adjusts the photometric calibration parameters. The mathematical model at the back end can be expressed as an optimization problem, with the goal of minimizing prediction errors and achieving physical constraints, as shown in Equation (18):

[0165]

[0166] In formula (18), f(c,M i ) is the predicted image brightness calculated based on the current parameter c, M' i is the measured image brightness value, R(c) is a regularization term that includes physical constraints, namely monotonicity and non-negativity. c is a vector containing all calibration parameters. The backend's data processing results are periodically updated to the frontend, and new calibration parameters are periodically transmitted via the network or data link. This dynamic parameter update mechanism enables the frontend to obtain optimized calibration parameters in a timely manner, ensuring the system's rapid adaptability to environmental changes. This two-tier architecture design ensures real-time responsiveness while also improving long-term stability and accuracy through continuous backend processing, making it adaptable to a variety of complex application scenarios.

[0167] This application has the following technical effects: an estimation-optimization framework based on optical flow and physical constraints is designed, which is more suitable for real-time application scenarios; a two-layer processing architecture for dynamic adjustment of photometry based on the fusion of real-time and optimization is designed, providing a continuous and adaptive photometry calibration strategy. Specifically, this application, by designing a framework based on optical flow and physical constraints, enables the visual SLAM system to better cope with image quality fluctuations caused by changes in illumination, thereby improving the accuracy of positioning and mapping under different lighting conditions, and at the same time, a two-layer processing framework for dynamic adjustment of photometry based on the fusion of real-time and optimization is designed. By separating real-time processing and back-end optimization, it not only ensures a fast response to input video frames, but also enables in-depth data analysis and parameter optimization. This design allows the system to continuously improve processing quality and accuracy through background algorithms while maintaining high real-time performance.

[0168] At least a portion of the steps in the flowcharts involved in the various embodiments described above may include multiple steps or multiple stages. These steps or stages may be executed at different times, or may be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0169] Based on the same inventive concept, embodiments of the present application also provide a third-order dynamic photometric calibration device using an estimation-optimization and dual-layer processing architecture for implementing the aforementioned third-order dynamic photometric calibration method using an estimation-optimization and dual-layer processing architecture. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the third-order dynamic photometric calibration device using an estimation-optimization and dual-layer processing architecture provided below can be found in the above-mentioned limitations of the third-order dynamic photometric calibration method using an estimation-optimization and dual-layer processing architecture, and will not be repeated here.

[0170] In an exemplary embodiment, Figure 5 As shown, a third-order dynamic photometric calibration device using an estimation-optimization and dual-layer processing architecture is provided, comprising:

[0171] The video frame sample set determination module 501 is configured to select at least one target key frame in a target video sequence, capture the pixel intensity of the field light in the target key frame using the KLT feature tracking technique, calculate the optical flow vector by estimating the motion direction and speed of each point in the target key frame image, and determine the video frame sample set based on the optical flow vector.

[0172] The estimation-optimization framework iteration module 502 is configured to establish an estimation-optimization network structure based on optical flow properties and physical constraints according to the video frame sample set, preliminarily estimate a set of photometric parameters, set a constraint condition, and optimize the set of photometric parameters by using a least square method, substitute the optimized set of photometric parameters into a photometric calibration formula, and perform photometric calibration on the video frame by using the result of the set of photometric parameters to obtain a photometric feature calibration value; iteratively estimate and optimize the network structure to obtain an estimation-optimization framework taking the video frame in the video frame sample set as input and the photometric feature calibration value as output. The set of photometric parameters includes CRF, exposure time, and vignetting effect parameters.

[0173] The photometric dynamic adjustment double-layer processing framework training module 503 is configured to establish a photometric dynamic adjustment double-layer processing framework including a front end and a back end. The front end of the photometric dynamic adjustment double-layer processing framework includes the estimation-optimization framework, is configured to perform real-time processing on the received video frame, and outputs the photometric feature calibration value to the back end. The back end of the photometric dynamic adjustment double-layer processing framework outputs the photometric feature calibration value to the video frame, and iteratively processes the set of photometric parameters and transmits the set of photometric parameters to the front end.

[0174] The above-mentioned modules in the three-order dynamic photometric calibration device adopting the estimation-optimization and double-layer processing architecture can be realized by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above-mentioned modules.

[0175] In one embodiment, a computer device is also provided, which includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps in the above-mentioned method embodiments are implemented.

[0176] For those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A third-order dynamic photometric calibration method using an estimation-optimization and two-layer processing architecture, characterized by: For achieving photometric calibration of a target video sequence, the method comprises: Step S1, selecting at least one target key frame in the target video sequence, using KLT feature tracking technology to capture the pixel intensity of the field light in the target key frame, solving the optical flow vector by estimating the motion direction and speed of each point in the target key frame image, and determining a video frame sample set based on the optical flow vector; Step S2, establishing an estimation-optimization network structure based on optical flow and physical constraints based on the video frame sample set, preliminarily estimating a photometric parameter set, setting constraints, and optimizing the photometric parameter set using the least squares method, substituting the optimized photometric parameter set into a photometric calibration formula, and using the obtained photometric parameter set results to perform photometric calibration on the video frames to obtain photometric feature calibration values; iterating the estimation-optimization network structure to obtain an estimation-optimization framework with the video frames in the video frame sample set as input and the photometric feature calibration values ​​as output; wherein the photometric parameter set includes a CRF, exposure time, and vignetting effect parameters; Step S3, establishing a two-layer processing framework for dynamic photometric adjustment including a front end and a back end; wherein the front end of the two-layer processing framework for dynamic photometric adjustment includes an estimation-optimization framework for real-time processing of received video frames and outputting photometric feature calibration values ​​to the back end; the back end of the two-layer processing framework for dynamic photometric adjustment outputs photometric feature calibration values ​​to video frames, and simultaneously iterates the photometric parameter set and transmits it to the front end.

2. The method according to claim 1, characterized in that The step S1 comprises: At least one target key frame whose exposure difference reaches a preset threshold is selected in the target video sequence. KLT feature tracking technology is used in the selected target key frame. The position of the scene point is represented by a coordinate system. The brightness constant equation is shown in formula (1): I(x,y,t)=I(x+dx,y+dy,t+dt) (1) In formula (1), I(x,y,t) is the pixel intensity at position (x,y) at time t, dx and dy are the displacement vectors from time t to t+dt; By Taylor expansion approximation, as shown in formula (2): In formula (2), the partial derivative and Represents the gradient of the image in the horizontal and vertical directions of the coordinate system; The spatial and temporal gradients of the image brightness in the target key frame are linked to the optical flow vector to form the optical flow equation, as shown in formula (3): In formula (3), v x and v y It is the horizontal and vertical axis components of the optical flow vector in the specified coordinate system. is the rate of change of image pixel intensity over time; The difference between the actual target key frame image and the target key frame image predicted by optical flow is quantified by defining an error function, which is shown in formula (4): ∈=∑ (x,y)∈W [I(x,y,t)-I(x+dx,y+dy,t+dt) 2 (4) In formula (4), W is the window centered on the feature point; By solving the displacement vector that minimizes the error function, the optical flow vector representing the movement of the actual object is determined, and then iterative optimization is performed until the algorithm converges to the optimal solution to obtain a video frame sample set.

3. The method according to claim 2, characterized in that The step S2 comprises: Step S2.1, preliminarily estimating the CRF, exposure time and vignetting effect parameters in the photometric parameter set respectively; Step S2.2: Based on the results of the preliminary estimation of the photometric parameter set, an estimation-optimization network structure is set; the photometric parameter set is adjusted by a nonlinear least squares method and physical constraints are introduced; in the process of adjusting the photometric parameter set by the nonlinear least squares method and introducing physical constraints, a regularization term and a target linear constraint are added, and the nonlinear least squares method is solved by gradient descent, the gradient is calculated, and the photometric parameter set is updated. The iteration is stopped when a preset threshold is reached, and the optimized photometric parameter set is obtained; In step S2.3, the optimized photometric parameter set is substituted into the photometric calibration formula to obtain the results of each photometric parameter in the photometric parameter set, and the video frame is photometrically calibrated using the results of the photometric parameter set to obtain the photometric feature calibration value; the network structure is iteratively estimated and optimized to obtain an estimation-optimization framework with the video frames in the video frame sample set as input and the photometric feature calibration value as output.

4. The method according to claim 3, characterized in that The step S2.1 includes: Step S2.1.1, performing a preliminary estimation of each photometric parameter in the photometric parameter set, where the photometric parameter set includes CRF, exposure time, and vignetting effect parameters; Among them, the expression of CRF is shown in formula (5): L=f -1 (M) (5) In formula (5), L is the scene radiation and M is the image intensity value. The specific expression is shown in formula (6): f -1 (M) = c0 + c1M + c2M 2 +c3M 3 (6) In formula (6), c0, c1, c2, and c3 are the coefficients of the CRF polynomial to be estimated; c0 is a constant term used to describe the basic offset between scene brightness and image brightness; c1 is a linear term coefficient used to describe the linear relationship between image brightness M and scene brightness; c2 is a quadratic term coefficient used to describe slight nonlinear changes; and c3 is a cubic term coefficient used to handle high-order nonlinear responses under complex illumination. In step S2.1.2, the exposure time e is adjusted as shown in formula (7): In formula (7), e is the exposure time, L(t) is the radiation intensity of the scene at time t; In step S2.1.3, the vignetting effect function is shown in formula (8): V(R)=1+v1R 2 +v2R 4 +v3R 6 (8) In formula (8), R is the normalized distance to the image center, v1 is the quadratic term coefficient in the vignetting polynomial, which is used to simulate the illumination changes within a smaller radius near the image center; v2 is the quartic term coefficient, which is used to adjust the illumination changes at a radius farther than that described by v1; and v3 is the sixth term coefficient, which affects the peripheral area of ​​the image.

5. The method according to claim 4, characterized in that The physical constraints introduced in step S2.2 include: Monotonic physical constraints are added to ensure that as the input increases, the CRF output should also increase accordingly. The expression is shown in formula (9): Formula (9) requires that the CRF function increases monotonically with M, that is, the derivative should always be greater than zero; The non-negativity constraint is added to ensure that the output will not have unrealistic negative values ​​under any circumstances. The expression is shown in formula (10): Formula (10) requires that the output of CRF is always non-negative.

6. The method according to claim 5, characterized in that The step S2.2 includes: In step S2.2.1, the results of the preliminary estimation of the photometric parameter set are introduced into the estimation-optimization network structure, and the CRF, exposure time, and vignetting effect parameters are adjusted by the nonlinear least squares method. The least squares method expression is shown in formula (11): In formula (11), A is the target matrix, b is the observed data vector, and c is the coefficient to be estimated. This formula is used to determine the parameter c so that the sum of the squares of the differences between the predicted value Ac and the actual value b is minimized. In step S2.2.2, the formula is expanded by adding the regularization term and the target linear constraint as shown in formula (12): In formula (12), λ is the regularization parameter used to adjust the influence of the regularization term, and E and d represent additional linear constraints; The gradient descent method is used to solve the nonlinear least squares method, and the expression is shown in formula (13): J(c)=c T A T Ac-2b T Ac+λ(Ec-d) T (Ec-d) (13) In formula (13), the update rule of the gradient descent method is shown in formula (14): In formula (14), α is the learning rate, which determines the step size of each update to ensure convergence; Repeat the photometric parameter set update until it reaches a preset threshold and stops to obtain the optimized photometric parameter set.

7. The method according to claim 6, characterized in that Substituting the optimized photometric parameter set into the photometric calibration formula in step S2.3 includes: Substitute the initially estimated and optimized photometric parameter set into the photometric calibration formula, as shown in formula (15): In formula (15), M1 and M2 are the measured intensities under different exposure conditions, and e1 and e2 are the corresponding exposure times; After linearization, the proportional relationship is re-expressed in matrix form, as shown in formula (16): in, 8. The method according to claim 7, characterized in that The step S3 comprises: Establish a two-layer processing framework for dynamic photometric adjustment including front-end and back-end; The front end of the two-layer processing framework for dynamic photometric adjustment includes an estimation-optimization framework for real-time processing of received video frames and performing photometric calibration using a photometric parameter set, and outputting photometric feature calibration values ​​to the back end. The backend of the dual-layer processing framework for dynamic photometric adjustment receives photometric feature calibration values ​​and outputs them to video frames. At the same time, the backend updates and iterates the photometric parameter set and transmits the updated photometric parameter set to the frontend.

9. A third-order dynamic photometric calibration device using an estimation-optimization and dual-layer processing architecture, characterized in that: The device comprises: A video frame sample set determination module is configured to select at least one target key frame in a target video sequence, capture the pixel intensity of the midfield light in the target key frame using KLT feature tracking technology, solve the optical flow vector by estimating the motion direction and speed of each point in the target key frame image, and determine the video frame sample set based on the optical flow vector; An estimation-optimization framework iteration module is used to establish an estimation-optimization network structure based on optical flow and physical constraints based on a video frame sample set, preliminarily estimate a photometric parameter set, set constraints, and optimize the photometric parameter set using the least squares method. The optimized photometric parameter set is substituted into a photometric calibration formula, and the obtained photometric parameter set is used to perform photometric calibration on the video frame to obtain photometric feature calibration values. The estimation-optimization network structure is iterated to obtain an estimation-optimization framework with video frames in the video frame sample set as input and photometric feature calibration values ​​as output; wherein the photometric parameter set includes CRF, exposure time, and vignetting effect parameters. The photometric dynamic adjustment dual-layer processing framework training module is used to establish a photometric dynamic adjustment dual-layer processing framework including a front end and a back end; the front end of the photometric dynamic adjustment dual-layer processing framework includes an estimation-optimization framework for real-time processing of received video frames and outputting photometric feature calibration values ​​to the back end; the back end of the photometric dynamic adjustment dual-layer processing framework outputs photometric feature calibration values ​​to video frames, and simultaneously iterates the photometric parameter set and transmits it to the front end.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images