Method for assisting HDR (High Dynamic Range) of double-structure optical system based on offline-online dual-mode coordination strategy

Abstract

The invention discloses an HDR (High Dynamic Range) method of an auxiliary double-structure optical system based on an offline-online dual-mode cooperation strategy. The method comprises the following steps: in an offline mode, defining an intensity nonlinear mapping function INMF under a high dynamic range recovery model; in the off-line mode, calibrating a camera nonlinear response function CNRF to obtain an estimated camera nonlinear response function; in an off-line mode, a two-parameter unknown Sigmoid nonlinear function is established, the two parameters are fitted and estimated, and then the estimated two parameters are used as common parameters to nonlinearly adjust a brightness mapping function of a coding stripe image sequence EFIS in a low dynamic range; in an online mode, estimating an illumination matrix graph of the coded stripe image sequence EFIS, and obtaining an estimated illumination component graph and an exposure proportion matrix; and in an on-line mode, obtaining a high-dynamic recovery result from a coding stripe image sequence EFIS in a low dynamic range by using the estimated exposure proportion matrix and the two parameters. According to the invention, high-speed and high-quality reconstruction of the image can be realized.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more specifically, to an HDR (high dynamic range image) method for auxiliary dual-structured light systems using an offline-online dual-mode collaborative strategy. Background Art

[0002] Natural image enhancement techniques are usually divided into two categories: global enhancement and local enhancement. Global enhancement performs the same processing on all image pixels and ignores their spatial distribution. Representative solutions of this type of technology include linear amplification, nonlinear power-rate enhancement, histogram normalization and its variants. However, this global enhancement technology may cause loss of details in some local areas, mainly because the global processing method cannot ensure that all local areas can be well enhanced. Local enhancement technology takes the spatial distribution of pixels into account and can obtain better results. Classic solutions of this type of technology include local histogram normalization, Retinex theory-based enhancement methods and their variants. However, local enhancement technology may fall into the risk of under / over enhancement, mainly because the nonlinear response characteristics inside the camera are ignored.

[0003] Structured light technology has gradually played an important role in the field of 3D reconstruction with its advantages of non-contact, high precision and high reliability. At present, this technology has shown great application value in many fields such as industrial automation detection, film and television special effects production, medical diagnosis, virtual simulation, etc. Intrinsic structured light systems are often unable to balance accuracy and efficiency due to the limitations of factors such as hardware level and cost. How to achieve efficient reconstruction of inherent structured light systems has always been a concern of scientific research and industry. The most direct idea to solve this problem is to reduce the exposure of the imaging device and increase the sampling frequency. However, this method comes at the expense of imaging brightness or poor visual performance. Therefore, the structured light system with low exposure setting requires a high-quality enhancement method as auxiliary imaging to restore the encoded information submerged in darkness.

[0004] In the prior art, patent application 201180008464.1A discloses a high dynamic range image generation and rendering scheme. The HDR image generation system performs motion analysis on a collection of lower dynamic range images and derives the relative exposure level of the image based on the information obtained in the motion analysis. When integrating LDR (low dynamic range) images to form an HDR image, using these relative exposure levels, the HDR image renderer tone maps the sample values ​​in the HDR image to corresponding lower dynamic range values ​​and calculates local contrast values. A residual signal is derived based on the local contrast, and the sample values ​​of the LDR image are calculated based on the tone-mapped sample values ​​and the residual signal. User preference information can be used during various stages of HDR image generation or rendering.

[0005] Patent application 2013100084271A discloses a low-light image enhancement method. The method includes: using the bright channel prior and the dark channel prior to obtain the bright channel image and the dark channel image of the low-light image respectively; obtaining the adaptive atmospheric illumination map through the bright channel image; obtaining the adaptive transfer function map through the dark channel image and the adaptive atmospheric illumination map; restoring the scene image according to the low-light image, the adaptive atmospheric illumination map and the adaptive transfer function map in the atmospheric scattering physical model. The characteristic of this scheme is that it is based on the atmospheric scattering physical model, and can adaptively process various images captured at night or in a weak light environment. The enhanced image has an ideal contrast and visual effect, and the overall enhancement effect is better than the traditional image enhancement method.

[0006] In the development of augmented reality (AR) and virtual reality (VR) technologies such as the Metaverse, vision-based 3D reconstruction technology plays a vital role. Reconstructing the 3D surface contour of an object is crucial for a variety of visual 3D reconstruction systems (such as binocular stereo vision, laser scanning, time of flight, structured light systems, etc.), whether in active or passive projection coding mode. Among them, structured light systems have been widely used in industrial and commercial fields for their high-precision and high-density 3D reconstruction capabilities in active projection coding mode.

[0007] Traditional structured light systems usually include a projector for projecting coded stripe patterns and a camera for capturing coded stripe images. Ideally, the structured light system decodes the depth information of the target at a high exposure setting by calibrating parameters, and uses the principle of triangulation to accurately estimate the coordinates of 3D points on the surface of the projected object. When performing high-speed reconstruction of the structured light system, the exposure time of the camera needs to be precisely synchronized with the exposure time of the projector to support high frame rate reconstruction. It is worth noting that this high frame rate requires a low exposure time setting, but this type of setting will cause the coded information in the coded stripe image to fall into darkness, making it difficult to identify effective features, resulting in a decrease in the quality of 3D reconstruction, such as loss of local details and reduced accuracy. Obviously, how to solve the problem of underexposure recovery of coded stripe images under low exposure settings has become the key to affecting the quality of 3D reconstruction of structured light systems. To solve this problem, the most direct way is to increase the dynamic range of the coded stripe image.

[0008] Currently, most high dynamic range images are generated based on the method of using different exposure times under the same lighting conditions, which means that each frame of high dynamic range video is synthesized from multiple image frames with different exposures. This generation scheme places high demands on the high sampling frequency of the hardware device, which usually makes it difficult for the hardware device to respond, causing jitter in the generated video. In addition, ordinary cameras cannot quickly generate images with different exposures in "online mode".

[0009] Many computer vision algorithms assume that image intensity accurately records the radiance of the scene. However, camera manufacturers often improve the visual quality of captured images through nonlinear in-camera processing such as white balancing and demosaicing. These nonlinear processes compress the dynamic range in the image at the expense of irradiance and light intensity, and ignoring these processes may lead to degraded performance of vision algorithms.

[0010] Traditional high dynamic range imaging methods not only need to estimate CNRF, but also need to perform additional tone mapping operations to map the high dynamic range image to a low dynamic range (LDR) intensity image.

[0011] In summary, the 3D reconstruction method of the structured light system needs to be combined with high-visibility coded imaging. Limited by the hardware system conditions, the existing structured light system often needs to set the camera unit to a low exposure state to achieve high frame rate imaging, and then cooperate with the projection unit and its reconstruction algorithm to achieve a high frame rate 3D reconstruction effect. However, the coded stripe images captured by the system under low exposure settings are often accompanied by low visibility problems, which directly affects the discrimination ability of the structured light method that relies on significant coded features, while reducing its decoding accuracy and the completeness of 3D reconstruction. Summary of the invention

[0012] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide an HDR method for auxiliary dual-structured light system using an offline-online dual-mode collaborative strategy. The method comprises the following steps:

[0013] In offline mode, based on the ideal visual model, the intensity nonlinear mapping function INMF under the high dynamic range restoration model is defined;

[0014] In offline mode, the camera nonlinear response function CNRF is calibrated to obtain the estimated camera nonlinear response function;

[0015] In an offline mode, a Sigmoid nonlinear function with unknown dual parameters is established, and the dual parameters are estimated by fitting, and then the estimated dual parameters are used as common parameters to nonlinearly adjust the brightness mapping function of the coded fringe image sequence EFIS with a low dynamic range;

[0016] In the online mode, the illumination matrix diagram of the coded fringe image sequence EFIS is estimated to obtain the estimated illumination component diagram and exposure ratio matrix;

[0017] In an online mode, the estimated exposure ratio matrix and the dual parameters are used to obtain a high dynamic restoration result from a low dynamic range coded fringe image sequence EFIS.

[0018] Compared with the prior art, the advantage of the present invention is that, in view of the underexposure problem of EFIS (Encoded Fringe Image Sequence) caused by low exposure setting when reconstructing a dual structured light system at a high frame rate, the present invention proposes a high dynamic enhancement method for an auxiliary dual structured light system integrating offline-online dual-mode cooperation. First, in the "offline mode", the Sigmoid nonlinear function (SNF) solved by parameter optimization is used as the intensity nonlinear mapping function (INMF); then, in the "online mode", the INMF is used to nonlinearly map the captured low dynamic range (LDR) coded fringe image sequence (EFIS) to the high dynamic range (HDR) EFIS required by the structured light algorithm. Finally, combined with the HDR EFIS signal, the dual structured light system and its algorithm can easily realize high-density point cloud recovery and high-speed three-dimensional reconstruction.

[0019] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0021] Figure 1 It is a schematic diagram of the overall process of the HDR method of the auxiliary dual-structured light system of the offline-online dual-mode collaborative strategy according to one embodiment of the present invention;

[0022] Figure 2 is a schematic diagram of an exposure stack (left lens example) collected at an interval of 1000 us / frame in an offline state according to an embodiment of the present invention;

[0023] Figure 3 is a schematic diagram of an underexposed coding sequence captured in an online state and a ×10 intensity magnified version thereof according to an embodiment of the present invention;

[0024] Figure 4 is a schematic diagram of an HDR enhanced version in an online state according to an embodiment of the present invention;

[0025] Figure 5 This is a comparison diagram of the effects of implementing three-dimensional reconstruction using an enhanced solution assisted by a structured light algorithm according to an embodiment of the present invention;

[0026] Figure 6 is a schematic diagram of three color channel response curves and their average curves under a left lens according to an embodiment of the present invention;

[0027] Figure 7 is a schematic diagram of actual data (average response curve) and a fitting curve according to an embodiment of the present invention;

[0028] Figure 8 1 is a schematic diagram of a camera response function set of a DoRF data set and a Sigmoid function fitting curve of optimized parameters according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0030] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0031] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0032] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0033] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0034] In general, the HDR method of the auxiliary dual-structured light system of the offline-online dual-mode collaborative strategy provided by the present invention includes an offline mode stage and an online mode stage.

[0035] For example, the steps in offline mode include:

[0036] 1) DSLs (Dual Structured Light System) image exposure stack acquisition and ideal vision-HDR model. First, the acquisition method of DSLs acquisition image exposure stack is given. Then, starting from the classical imaging model and Mann theory mapping model, the INMF (intensity nonlinear mapping function) proposed in this invention is redefined, and the INMF in the HDR mode defined in this invention is further derived through the ideal vision exposure model.

[0037] 2) Analysis and calibration of camera nonlinear response function: The intrinsic properties of the classic CNRF are analyzed, and the CNRF of the current DSLs of the real scene is estimated in offline mode by combining offline exposure stacks with different exposure times using the Debevec method.

[0038] 3) Establishment, optimization fitting and parameter estimation of Sigmoid nonlinear function. A Sigmoid nonlinear function controlled by two unknown parameters is introduced as the approximate curve of the average statistical dorf data. Then, the SNF is obtained by fitting the real Debevec-CNRF of DSLs through the Quasi-Newton optimization technique. At the same time, the unknown parameters in the optimization objective function are estimated. Finally, the SNF with known parameters is obtained and used for intensity nonlinear mapping.

[0039] For example, the steps in online mode include:

[0040] 4) EFIS illumination matrix estimation: The proposed optimization objective function is solved step by step iteratively by combining alternating direction minimization and fast Fourier transform method to achieve a refined estimation of the initial illumination value.

[0041] 5) High dynamic restoration of EFIS. Combining the Sigmoid function with known parameters, the refined estimated illumination map and the ideal mapping conditions, a high dynamic restoration formula is established to map the low-exposure DSLs image sequence into an HDR image sequence in the online mode (stage).

[0042] Specifically, see Figure 1 As shown, the provided offline-online dual-mode collaborative strategy auxiliary dual-structured light system HDR method includes the following steps:

[0043] Step S1, in offline mode, collect DSLs image exposure stack and determine the ideal visual-HDR model.

[0044] 1) DSLs image exposure stack acquisition

[0045] For example, in offline mode, the software triggers the left / right lens of DSLs to respectively realize the interval "+1000us / frame" acquisition (200us~10000us left / right two groups of 22 frames in total). The two sets of image sequences can be approximately regarded as "exposure stacks" representing the dynamic range of the scene, such as Figure 2 and Figure 3 As shown, this can be used to estimate or calibrate the response function of DSLs under dual lenses.

[0046] 2) Ideal visual exposure - HDR model

[0047] a) Classical imaging model

[0048] According to the classic Retinex theoretical model, the amount of actual light reaching the observer's eyes can be understood as the image irradiance H(x) captured by the camera sensor, which can be composed of the reflectivity component R(x) and the illumination component L(x) of the actual scene:

[0049]

[0050] Operator In theory, Η(x) is a highly dynamic scene image. In practice, it is processed by CNRF f in almost all existing cameras:

[0051] S(x)=f(H) (2)

[0052] It is used to generate the back-end imaging S(x) that pleases the user's vision, which is also the LDR image captured by DSLs under low exposure settings. The CNRFf here describes the nonlinear relationship between the image irradiance H(x) and the pixel value S(x). When the exposure setting value of the camera changes, the irradiance H reaching the camera's CMOS sensor changes linearly. However, considering the nonlinear processing process inside the camera, the image intensity S does not change linearly with the irradiance H, but changes nonlinearly. Therefore, the mapping function between images with different exposures is also a nonlinear function, also called the brightness nonlinear mapping function, or INMF.

[0053] b) Mann Theory Mapping Model

[0054] The INMF proposed by Mann is expressed as:

[0055]

[0056] in is INMF, and e is the exposure ratio. Both CNRF and INMF are nonlinear functions that describe the camera imaging process, and the two constitute the camera response model.

[0057] Combining the above formulas, we can get the following corresponding equation:

[0058]

[0059] This equation is called the coparametric equation, and it describes both f and The relationship between them can also be used for mutual conversion between them.

[0060] c) Definition of INMF of the present invention

[0061] Different from the above methods, the present invention proposes to define the INMF of low dynamic EFIS collected by DSLs as:

[0062]

[0063] It is worth noting that, unlike the exposure ratio e in the previous text, E in the present invention is a matrix, which represents the expected exposure ratio of each pixel therein. The number of sequence forms of the encoded fringe image is j=1,...18.

[0064] d) Ideal visual exposure model

[0065] Underexposed DSLs images are usually divided into global underexposed and local underexposed images. Global underexposed images are mainly caused by low lighting, while local underexposed images are mainly caused by non-uniform lighting and the limited dynamic range of industrial cameras. Therefore, in one embodiment, it is assumed that the ideal exposure state is: adjusting the lighting to uniform and high intensity will not only restore the information "submerged" in low brightness, but also enable the camera to fully record the scene coding information, that is, HDR. Based on this, the ideal visual exposure model (or enhanced image restored by HDR) can be written as:

[0066]

[0067] in, It represents a matrix assuming that all elements in the illumination L are adjusted to 1, that is,

[0068] e) Definition of INMF under HDR restoration model

[0069] Combining formulas (1), (2), (5) and (6), the INMF result under ideal HDR restoration can be rewritten as:

[0070]

[0071] The symbol ⊙ represents pixel-by-pixel division. Next, how to construct a camera response function that approximates the actual scene under the assumed function form becomes the key. The following steps S2 and S3 are used to respectively implement the calibration of the actual CNRF and the SNF optimization fitting, and the exposure ratio matrix and INMF are estimated by steps S4 and S5, respectively, to further realize the high dynamic recovery of the EFIS captured by DSLs under low exposure settings.

[0072] Step S2: Analyze and calibrate the camera nonlinear response function in an offline mode.

[0073] 1) CNRF intrinsic properties

[0074] As described by researcher Nayar: CNRFξ should have three assumed properties: a) consistency, that is, it is the same for all pixels on the camera sensor. In other words, the response of each pixel of the camera under the same lighting conditions is consistent, which ignores the slight differences that may exist in the pixels; b) normalization, that is, its output can be scaled to a range of [0,1], which helps to compare and analyze different cameras or different encoded images; c) monotonicity, that is, it is monotonically increasing, which means that the pixel value will increase with the increase of the incident light intensity.

[0075] 2) CNRF definition

[0076] Combining the three assumptions, in one embodiment of the present invention, the theoretical space Φ of the camera nonlinear response function ξ is ξ Defined as:

[0077]

[0078] Where (x, y) represents the spatial coordinates. Obviously, formula (8) means the intensity nonlinear mapping function has the same assumed properties as ξ.

[0079] 3) Debevec-CNRF estimation

[0080] In offline state, the camera response function f is estimated by the Debevec author scheme for the image collected by adjusting the exposure debevec ,like Figure 6 As shown. This function can be used to fit the unknown parameters in the Sigmoid function below. It should be noted that only by fitting the Sigmoid function with parameters can it be converted into a brightness mapping function with parameters; if a nonlinear f is used debevec However, the function cannot derive the INMF directly used for brightness nonlinear mapping.

[0081] Step S3, in offline mode, establish a Sigmoid nonlinear function, and perform optimization fitting and parameter estimation.

[0082] 1) Reasons for choosing Sigmoid

[0083] a) First, the Sigmoid curve distribution conforms to the intrinsic properties of CNRF;

[0084] b) Secondly, the Sigmoid curve fitting result is infinitely close to the response curve calibrated by the real camera provided by Nayar, such as Figure 8 shown.

[0085] In the classic Debevec study, the authors proposed a method to recover a high dynamic range radiance map using multiple photos taken with different exposure times by traditional imaging equipment. This method involves taking multiple photos of the same scene with different exposures, and using these photos with different exposures through an algorithm to recover the response function of the imaging process. Once the response function is obtained, the multiple photos can be fused into a single high dynamic range radiance map, whose pixel values ​​are proportional to the true radiance values ​​in the scene. However, this method is not directly applicable to EFIS collected online by high dynamic DSLs, because: it is mainly used to recover the high dynamic range of the scene from multiple photos with different exposures, rather than directly enhancing the EFIS taken by DSLs under low exposure settings; this type of method is based on the same lighting and adjusting different exposure times, that is, each frame of the high dynamic range video is synthesized by several frames with different exposures. This generation method places high sampling frequency requirements on the hardware device, which often makes it difficult for the hardware device to respond, resulting in jitter in the generated video. It is impossible for ordinary industrial cameras to quickly generate images with different exposures.

[0086] 2) Establishment of dual-parameter (unknown) Sigmoid function

[0087] In one embodiment of the present invention, it is proposed to use a two-parameter (unknown) Sigmoid function based on the human visual system as a model for fitting the camera response function, namely SNF:

[0088]

[0089] Where Z represents the pixel intensity value of the input encoded image sequence, and α and β are unknown parameters of the model to be calculated. It should be noted that the CNRF f assumption is used to calculate Use f sigmoid As To calculate the characteristics of f assumptions and there is no ambiguity.

[0090] 3) Sigmoid function and CRF curve fitting estimated by Debevec method (estimated by atlases collected at different time periods under the same illumination)

[0091] Sigmoid function with two (unknown) parameters and nonlinear response function f estimated by Debevec method debevec The fit is as follows:

[0092]

[0093] 4) Optimize-fit and estimate the unknown dual parameters of the Sigmoid function

[0094] In one embodiment of the present invention, the Quasi-Newton optimization technique is used to solve the optimization problem in formula (10). In the experiment, an estimated value (α, β) = (0.03, 1.2) after optimization solution is provided.

[0095] It can be deduced from formulas (3) and (9) that the common parameters (α, β) are used to nonlinearly adjust the brightness mapping function of the LDR EFIS It can be expressed as:

[0096]

[0097] This expression is the intensity nonlinear mapping function with two (known) parameters SNF. It is worth noting that in formula (11) It is achieved mainly by the exposure ratio e.

[0098] Step S4, in online mode, estimating the EFIS illumination matrix diagram.

[0099] 1) Estimation of initial illumination value of EFIS

[0100] It can be seen from the above formulas (5) and (7) that the estimated exposure ratio matrix E can be directly obtained in an inverse manner according to the illumination component L. Considering that the exposure ratio matrix E is inversely proportional to the illumination component map, E can be solved by estimating the illumination component map L. In other words, the proposed global optimization solution for estimating the illumination component L can be used as an estimate of the exposure ratio matrix E. For simplicity, for each pixel value x, the EFIS collected online by DSLs is first used directly as the initial estimate of the illumination, that is, L = S. The physical meaning of the above operation is: keep the encoding information of the illumination component consistent with the original input EFIS; the initial value of the illumination is consistent with the intensity distribution information of the original input EFIS.

[0101] 2) EFIS illumination MAP priors and constraints

[0102] Theoretically, estimating the refined illumination of a scene is an ill-posed problem. Here, the key to solving this problem is to introduce a priori that conforms to the actual illumination. This invention assumes that the scene where the actual DSLs are located is natural illumination, which has a piecewise smooth characteristic in space and conforms to the actual illumination distribution priori. In addition, a small window weighting matrix is ​​designed as the structural weight guide of the optimization function below, which is expressed as:

[0103]

[0104] Where h represents the horizontal direction, v represents the vertical direction, and L(y) represents the light intensity at the neighborhood y with x as the center. Here, the operator |·| is an absolute value operation. is a local window centered at x and with a radius of 3, and a small constant θ = 0.001 is used to avoid the denominator being zero. The first-order difference filter in the formula Contains horizontal and vertical direction The refined illumination component map can be solved by the following optimization function:

[0105]

[0106] in, represents the estimated initial illumination value, L(x) represents the illumination component to be optimized, and W L，φ represents the structural weight. Here, the regularization coefficient λ L It is used to optimize the two balanced terms in the function. It can be intuitively observed that the above optimization problem only contains quadratic terms, and the closed solution can be obtained directly, and its solution can be calculated only once. Therefore, this problem can be directly solved by solving the following equation:

[0107]

[0108] Among them, D φ Contains Dh and D v , which is the Toeplitz matrix of the discrete gradient operator with forward difference. l represents the vector form of the illumination component L(N) to be optimized. Represents the initial illumination component Here, yes In vector form, is a unit matrix of appropriate size. Meanwhile, the operator Diag(x) uses vector X to reconstruct a diagonal matrix. is a symmetric positive definite Laplace matrix and can therefore be solved using techniques such as preconditioned conjugate gradients.

[0109] 3) Lighting regularization function and its solution

[0110] By local weighting weight W L，φ Guided, the above optimization problem can obtain a refined illumination component map L with structure preservation, weak texture compression, and piecewise smoothness characteristics that conform to the spatial illumination distribution. The inverse result (1. / L) can be used as the value of the exposure ratio matrix E, denoted as: E a .

[0111] Step S5, in online mode, high dynamic recovery of EFIS.

[0112] 1) Combine known conditions

[0113] The exposure ratio matrix estimated in step S4 and the dual parameters controlling SNF to realize INMF are denoted as E a , α and β.

[0114] 2) Establishment of EFIS high dynamic formula model

[0115] Combining the proposed INMF model, the estimated exposure ratio matrix and the dual-parameter control coefficient, the EFIS for LDR state in DSLs proposed in this invention is: j (x) Achieve online high dynamic recovery results It can be expressed as:

[0116]

[0117] Here, S j (x) represents the sequence form S of 18 coded stripe images collected online by DSLs j (x), j = 1,...18, the high dynamic recovery result is

[0118] In summary, the present invention proposes a high dynamic enhancement method for an auxiliary dual structured light system integrating offline-online dual-mode cooperation, which constructs a system nonlinear response function that conforms to the actual scene through optimization fitting in the "offline mode", that is, a Sigmoid nonlinear function with optimized parameters, such as Figure 7 and Figure 8 As shown; then, the alternating direction minimization and fast Fourier method are combined to realize the refined estimation of the initial illumination value of the low-exposure DSLs image collected in the current scene; finally, the Sigmoid function with known parameters and the refined estimated illumination map are combined to realize the nonlinear mapping of the low-exposure DSLs image sequence to the HDR image sequence through the ideal mapping formula in the "online mode", producing an enhanced result with high visibility and high signal-to-noise ratio, as shown Figure 4 The LIME algorithm has unreasonable assumptions about the components in the model, which directly generates artifacts and overexposure in local areas, thereby destroying the original coding information and indirectly causing the reconstruction result of the structured light method to be unable to restore the 3D morphology, as shown in Figure 1. Figure 5 Although the STAR algorithm can obtain better visual expression enhancement results, its initial unreasonable assumptions lead to unsatisfactory estimated illumination components, which results in unclear encoding details in the results, indirectly causing a large number of details to be lost in the reconstruction results of the structured light method, as shown in Figure 2. Figure 5Compared with the prior art, the present invention takes the nonlinear response function inside the camera into consideration, combines the nonlinear response function estimation and high dynamic formula of the "offline" and "online" dual modes, and realizes the high-quality restoration of the low-exposure EFIS image sequence (or can be understood as: high visibility, the encoded information is not destroyed, the signal-to-noise ratio is high, etc.).

[0119] It should be noted that, without violating the spirit and scope of the present invention, those skilled in the art may make appropriate changes or modifications to the above embodiments. For example, the Quasi-Newton optimization technique for fitting the real Debevec-CNRF of DSLs and obtaining parameter optimization in the present invention can be replaced by other techniques, and is not limited to the optimization techniques used in the present invention. For another example, the joint alternating direction minimization and fast Fourier solution method used in the refined estimation of the initial illumination value proposed in the present invention can be implemented by other solutions, such as QR fast decomposition, low-rank space conversion, etc.

[0120] In summary, compared with the prior art, the present invention has the following advantages:

[0121] 1) The present invention provides an efficient solution for offline parameter estimation and online nonlinear HDR mapping. For the processed low-exposure EFIS, this method has high online processing mapping efficiency, high visibility, and high-quality visibility with high signal-to-noise ratio.

[0122] 2) An “exposure stack” is defined and acquired in offline mode, which indirectly captures wide dynamic range imaging in real scenes.

[0123] 3) The Quasi-Newton optimization technique is used to fit the real Debevec-camera nonlinear response function (CNRF) of the dual structured light system and obtain the parameter-optimized SNF.

[0124] 4) A high dynamic recovery formula is constructed by combining the Sigmoid function with known parameters, the refined estimated illumination map and the ideal mapping conditions, thereby realizing the efficient mapping of the low-exposure EFIS to the HDR state EFIS in an online state.

[0125] 5) The present invention can not only realize nonlinear high-quality intensity mapping in "online mode", but also can be integrated into high-performance development platforms such as embedded systems to realize real-time processing, and has extremely high use value and a wide range of application scenarios.

[0126] 6) The present invention estimates INMF in an offline state and directly maps the EFIS collected by DSLs into a highly dynamic EFIS in an online mode (stage), thereby avoiding the degradation problem (such as low visibility and low signal-to-noise ratio) that occurs when capturing EFIS online, and solving the problem of being unable to accurately estimate depth and perform 3D reconstruction under a fixed low exposure setting, thereby hopefully improving the system's ability to reconstruct high-speed moving objects.

[0127] In summary, the present invention proposes a method for determining the nonlinear response function of the camera by changing the lighting conditions rather than the exposure time in the same scene, so as to achieve the result of converting the low dynamic range EFIS into the high dynamic range EFIS. At the same time, considering the inaccuracy of the existing theoretical model of image processing, the present invention introduces an intensity nonlinear mapping function to directly realize the high dynamic of the system encoding fringe pattern. For example, the parameter-optimized Sigmoid nonlinear function estimated in the offline state is used as INMF, and the captured low dynamic range EFIS is mapped to the high dynamic range EFIS signal required by the structured light algorithm in the "online mode". It has been experimentally verified that the present invention can achieve high-speed and high-quality reconstruction of the dual structured light system under low exposure settings.

[0128] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0129] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0130] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0131] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0132] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0133] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0134] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0135] The flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of an instruction, and the module, a program segment or a part of an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0136] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. An offline-online dual-mode collaborative strategy-assisted dual-structured light system HDR method, comprising the following steps: In offline mode, based on the ideal visual model, the intensity nonlinear mapping function INMF under the high dynamic range restoration model is defined; In offline mode, the camera nonlinear response function CNRF is calibrated to obtain the estimated camera nonlinear response function; In an offline mode, a Sigmoid nonlinear function with unknown dual parameters is established, and the dual parameters are estimated by fitting, and then the estimated dual parameters are used as common parameters to nonlinearly adjust the brightness mapping function of the coded fringe image sequence EFIS with a low dynamic range; In the online mode, the illumination matrix diagram of the coded fringe image sequence EFIS is estimated to obtain the estimated illumination component diagram and exposure ratio matrix; In an online mode, the estimated exposure ratio matrix and the dual parameters are used to obtain a high dynamic restoration result from a low dynamic range coded fringe image sequence EFIS.

2. The method according to claim 1, characterized in that The intensity nonlinear mapping function INMF under the high dynamic range restoration model is expressed as: Where x represents the pixel value, the symbol θ represents pixel-by-pixel division, R(x) represents the reflectance component of the actual scene, L(x) represents the illumination component, H(x) represents the image irradiance, and f is used to describe the nonlinear relationship between the image irradiance H(x) and the pixel value. is the intensity nonlinear mapping function, is the identity matrix, S(x) is used to generate backend imaging that pleases the user's vision, and j represents the sequence index of the encoded stripe image.

3. The method according to claim 2, characterized in that According to the following fitting formula, the estimated camera nonlinear response function f is obtained debevec : in: Among them, f debevec is the estimated camera nonlinear response function, α and β are unknown parameters to be determined, and Z represents the pixel intensity value of the input encoded image sequence.

4. The method according to claim 3, characterized in that The brightness mapping function of the low dynamic range coded fringe image sequence EFIS is adjusted according to the following formula: in, Represents the brightness mapping function of the low dynamic range coded fringe image sequence EFIS.

5. The method according to claim 4, characterized in that The illumination component map L is fitted by the following optimization function: in: Among them, λ L is the regularization coefficient, the operator |·| is the absolute value operation, is a local window with x as the center and a set radius, θ is a set constant, and a first-order difference filter Contains horizontal direction ▽ h and vertical direction ▽ v Operation, the estimated exposure ratio matrix E is obtained in an inversely proportional manner according to the illumination component L, is the initial estimated value of the illumination component L, h represents the horizontal direction, v represents the vertical direction, L(y) represents the illumination intensity at the y position in the neighborhood centered at x, L(x) represents the illumination component to be optimized, W L,φ Represents the structural weight.

6. The method according to claim 5, characterized in that The optimization function of the illumination component map L is solved by the following formula: in, yes In vector form, is the identity matrix. The operator Diag(x) uses the vector x to reconstruct a diagonal matrix. φ Contains D h and D v , is the Toeplitz matrix of the discrete gradient operator with forward differentiation, l is the vector form of L(x), yes in vector form.

7. The method according to claim 6, characterized in that For each pixel value x, the coded fringe image sequence EFIS collected online by the dual structured light system is used as the initial estimate of the illumination, expressed as 8. The method according to claim 1, characterized in that The high dynamic recovery results It is expressed as: Among them, S j (x) represents the sequence of multiple coded fringe images collected online by the dual structured light system, E a (x) represents the estimated exposure ratio matrix, and α and β represent the estimated dual parameters.

9. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images