HDR method of off-line and on-line dual-mode cooperative strategy assisted dual structured light system

Abstract

The application discloses an off-line and on-line dual-mode cooperative strategy assisted dual-structure light system HDR method. The method comprises the following steps: in the off-line mode, defining an intensity nonlinear mapping function INMF under a high dynamic range recovery model; in the off-line mode, calibrating a camera nonlinear response function CNRF to obtain an estimated camera nonlinear response function; in the off-line mode, establishing a Sigmoid nonlinear function with unknown double parameters, and fitting the double parameters to estimate the double parameters, and then using the estimated double parameters as common parameters to nonlinearly adjust the brightness mapping function of a low dynamic range encoded fringe image sequence EFIS; in the on-line mode, estimating an illumination matrix diagram of the encoded fringe image sequence EFIS to obtain an estimated illumination component diagram and an exposure proportion matrix; and in the on-line mode, using the estimated exposure proportion matrix and the double parameters to obtain a high dynamic recovery result from the low dynamic range encoded fringe image sequence EFIS. The application can realize high-speed and high-quality reconstruction of images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and more particularly, to an off-line and on-line dual-mode cooperative strategy assisted dual-structured light system HDR (high dynamic range image) method. BACKGROUND

[0002] Natural image enhancement techniques are generally divided into two categories: global enhancement and local enhancement. Global enhancement performs the same processing on all image pixels regardless of their spatial distribution. Representative schemes of this category include linear amplification, non-linear power-law enhancement, histogram normalization and its variants, etc. However, this kind of global enhancement technique can cause the loss of details in certain local regions, mainly because the global processing method cannot ensure that all local regions are well enhanced. Local enhancement techniques take the spatial distribution of pixels into account and can achieve better results. Classical schemes of this category include local histogram normalization, enhancement methods based on Retinex theory and its variants, etc. However, local enhancement techniques can fall into the risk of under / over enhancement, mainly because the non-linear response characteristics of the camera are ignored.

[0003] Structured light technology gradually plays an important role in the field of three-dimensional reconstruction with its non-contact, high precision and reliability. At present, this technology has shown great application value in many fields such as industrial automation detection, special effects production, medical diagnosis, virtual simulation, etc. Due to the limitations of hardware level and cost, the inherent structured light system often cannot balance the precision and efficiency. How to realize the efficient reconstruction of the inherent structured light system has been a problem concerned by the scientific research and industry. The most direct idea to solve this problem is to reduce the exposure of the imaging device and improve the sampling frequency. However, this way sacrifices the imaging brightness or poor visual performance. Therefore, the structured light system with low exposure setting needs a high-quality enhancement method as auxiliary imaging to recover the encoded information submerged in darkness.

[0004] In the prior art, patent application 201180008464.1A discloses a high dynamic range image generation and rendering scheme. The HDR image generation system performs motion analysis on a set of lower dynamic range images and derives relative exposure levels for the images based on information obtained in the motion analysis. When integrating LDR (low dynamic range) images to form a HDR image, using these relative exposure levels, the HDR image renderer tone maps sample values in the HDR image to corresponding lower dynamic range values and calculates local contrast values. Based on the local contrast, a residual signal is derived and based on the tone mapped sample values and the residual signal, sample values for the LDR images are calculated. User preference information can be used during various stages of HDR image generation or rendering.

[0005] Patent application 2013100084271A discloses a low-illumination image enhancement method. The method comprises: using a bright channel prior and a dark channel prior to obtain a bright channel image and a dark channel image of a low-illumination image; obtaining an adaptive atmospheric light image through the bright channel image; obtaining an adaptive transmission function image through the dark channel image and the adaptive atmospheric light image; and restoring a scene image according to the low-illumination image, the adaptive atmospheric light image and the adaptive transmission function image in the atmospheric scattering physical model. The scheme is characterized in that it is based on the atmospheric scattering physical model, can adaptively process various images taken in a night or a weakly-lit environment, and the enhanced image has ideal contrast and visual effect, and the overall enhancement effect is better than that of a traditional image enhancement method.

[0006] In the development process of augmented reality (AR) and virtual reality (VR) technologies (such as the metaverse), vision-based 3D reconstruction technology plays a crucial role. Whether it is active projection or passive projection coding mode, reconstructing the 3D surface profile of an object is essential for various vision-based 3D reconstruction systems, such as binocular stereo vision, laser scanning, time-of-flight, and structured light systems. Among them, structured light systems have been widely used in industrial and commercial fields due to their high precision and high-density 3D reconstruction capabilities in active projection coding mode.

[0007] Traditional structured light systems usually include a projector for projecting a coded fringe pattern and a camera for capturing coded fringe images. Ideally, a structured light system decodes the depth information of a target under high exposure settings through calibration parameters and accurately estimates the 3D point coordinates of the projected object surface using the triangulation principle. When performing high-speed reconstruction of a structured light system, the exposure time of the camera needs to be precisely synchronized with the exposure time of the projector to support high-frame-rate reconstruction. Notably, this high-frame-rate requirement involves low exposure time settings, but such settings can cause the coded information in the coded fringe image to be obscured in darkness, making it difficult to identify effective features and resulting in decreased 3D reconstruction quality, such as loss of local details and reduced precision. Obviously, how to solve the problem of underexposure recovery of coded fringe images under low exposure settings becomes a key factor affecting the quality of 3D reconstruction of structured light systems. To address this issue, the most direct method is to improve the dynamic range of the coded fringe image.

[0008] Currently, most high dynamic range images are generated based on different exposure times under the same lighting conditions, which means that each frame of high dynamic range video is composed of multiple image frames with different exposure amounts. This generation scheme places high requirements on the high sampling frequency of hardware devices, which often causes hardware devices to be difficult to respond, resulting in jitter in the generated video. In addition, ordinary cameras cannot quickly generate images with different exposures in "online mode".

[0009] Many computer vision algorithms assume that image intensity accurately records the radiance of the scene. However, camera manufacturers often improve the visual quality of captured images through nonlinear camera internal processing, such as white balance and demosaicing processing. These nonlinear processes come at the expense of irradiance and light intensity, resulting in the dynamic range of the image being compressed, and neglecting these processes can lead to a decrease in the performance of visual algorithms.

[0010] Traditional high dynamic imaging methods not only need to estimate CNRF, but also need to perform a tone mapping operation to map the high dynamic radiance domain image to a low dynamic range (LDR) intensity image.

[0011] In summary, the structured light system three-dimensional reconstruction method needs to cooperate with high-visibility coded imaging. Limited by the hardware system conditions, the existing structured light system often needs to set the camera unit to a low exposure state to realize high frame rate imaging, and then cooperate with the projection unit and its reconstruction algorithm to realize high frame rate three-dimensional reconstruction effect. However, the coded fringe images captured by the system under low exposure setting often have low visibility problems, which directly affects the discrimination ability of the structured light method which relies on salient coded features, and reduces the accuracy of its decoding and the completeness of its three-dimensional reconstruction. SUMMARY

[0012] The purpose of the present application is to overcome the defects of the prior art described above, and to provide an off-line-online dual-mode cooperative strategy assisted dual-structured light system HDR method. The method comprises the following steps:

[0013] In the offline mode, based on the ideal visual model, define the intensity nonlinear mapping function INMF under the high dynamic range recovery model;

[0014] In the offline mode, calibrate the camera nonlinear response function CNRF to obtain the estimated camera nonlinear response function;

[0015] In the offline mode, a Sigmoid nonlinear function with two unknown parameters is established, and the two parameters are fitted and estimated, and then the estimated two parameters are used as common parameters to nonlinearly adjust the brightness mapping function of the low dynamic range coded fringe image sequence EFIS;

[0016] In the online mode, the illumination matrix map of the coded fringe image sequence EFIS is estimated to obtain the estimated illumination component map and exposure ratio matrix;

[0017] In the online mode, the estimated exposure ratio matrix and the two parameters are used to obtain the high dynamic recovery result from the low dynamic range coded fringe image sequence EFIS.

[0018] Compared with the prior art, the advantages of the present application are that, for the EFIS (Encoded Fringe Image Sequence) underexposure problem caused by low exposure setting in high frame rate reconstruction of the double structured light system, the present application proposes an integrated offline-online dual-mode assisted double structured light system high dynamic enhancement method. First, in the "offline mode", the Sigmoid Nonlinear Function (SNF) optimized and solved as the Intensity Nonlinear Mapping Function (INMF) is used as the intensity nonlinear mapping function; then, in the "online mode", the INMF is used to nonlinearly map the captured low dynamic range (LDR) encoded fringe image sequence (EFIS) to the high dynamic range (HDR) EFIS required by the structured light algorithm. Finally, combined with the HDR EFIS signal, the double structured light system and its algorithm can easily realize high dense point cloud recovery and high speed three-dimensional reconstruction.

[0019] Other features and advantages of the present application will become apparent from the following detailed description of illustrative embodiments thereof, which proceeds with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the application and, together with the description, serve to explain the principles of the application.

[0021] Figure 1 is the overall process schematic diagram of the offline-online dual-mode coordinated strategy assisted double structured light system HDR method according to an embodiment of the present application;

[0022] Figure 2 is the schematic diagram of the exposure stack (left lens example) collected at an interval of 1000us per picture in the offline state according to an embodiment of the present application;

[0023] Figure 3 is the schematic diagram of the underexposed encoded sequence captured in the online state and its ×10 intensity amplification version according to an embodiment of the present application;

[0024] Figure 4 is the schematic diagram of the HDR enhanced version in the online state according to an embodiment of the present application;

[0025] Figure 5 is the effect comparison diagram of the three-dimensional reconstruction realized by the enhanced scheme assisted structured light algorithm according to an embodiment of the present application;

[0026] Figure 6 is a schematic diagram of three color channel response curves under a left lens and their average curve according to an embodiment of the present application;

[0027] Figure 7 is a schematic diagram of actual data (average response curve) and fitted curve according to an embodiment of the present application;

[0028] Figure 8 is a schematic diagram of a set of camera response functions of a DoRF dataset and a sigmoid function fitted curve of optimized parameters according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangements, numerical expressions, and numerical values of components and steps set forth in these embodiments are not limiting to the scope of the present application unless otherwise specifically stated.

[0030] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way limiting to the scope of the application or its applications or uses.

[0031] Techniques, methods, and apparatus known to those of ordinary skill in the relevant art can not be discussed in detail herein, but should be considered as part of the specification, where appropriate.

[0032] In all of the compositions and methods shown and discussed herein, any specific values should be interpreted as merely exemplary, and are not to be interpreted as a limitation. Thus, other examples of the exemplary embodiments can have different values.

[0033] It should be noted that like references and characters herein relate to like items throughout the figures, and once an item is defined in one figure, it need not be discussed further in subsequent figures.

[0034] Generally, the HDR method of the auxiliary dual structured light system provided by the present application is a dual-mode off-line and on-line collaborative strategy, which includes an off-line mode stage and an on-line mode stage.

[0035] For example, the steps in the off-line mode include:

[0036] 1) DSLs (Dual Structured Light System) image exposure stack acquisition and ideal visual-HDR model. First, the acquisition method of the DSLs image exposure stack is given. Then, starting from the classic imaging model and Mann theoretical mapping model, the INMF (Intensity Nonlinear Mapping Function) proposed by the present application is redefined, and the INMF under the HDR mode defined by the present application is further derived through the ideal visual exposure model.

[0037] 2) Camera non-linear response function analysis and calibration. Analysis of the intrinsic properties of the classical CNRF, combined with the off-line exposure stack under different exposure times, the Debevec method is used to estimate the CNRF of the current DSLs to obtain the real scene in the off-line mode.

[0038] 3) Sigmoid non-linear function establishment, optimization fitting and parameter estimation. Introducing the Sigmoid non-linear function controlled by two unknown parameters as the approximate curve of the average statistical dorf data, then, through the Quasi-Newton optimization technology, the real Debevec-CNRF fitting of DSLs is realized to obtain SNF, at the same time, the unknown parameters in the optimization objective function are estimated, and finally the SNF with known parameters is obtained and used for intensity non-linear mapping.

[0039] For example, the steps in the online mode include:

[0040] 4) EFIS light matrix map estimation. Jointly using the alternating direction minimization and the fast Fourier method, the proposed optimization objective function is solved step by step, and the fine estimation of the light initial value is realized.

[0041] 5) High dynamic recovery of EFIS. Jointly using the Sigmoid function with known parameters, the fine estimated light map and the ideal mapping condition, the high dynamic recovery formula is established, and the low exposure DSLs image sequence is mapped to the HDR image sequence in the online mode (stage).

[0042] Specifically, referring to Figure 1 As shown, the auxiliary dual structured light system HDR method provided by the offline-online dual mode cooperative strategy includes the following steps:

[0043] Step S1, in the offline mode, the DSLs image exposure stack is collected and the ideal visual-HDR model is determined.

[0044] 1) DSLs image exposure stack collection

[0045] For example, in the offline mode, the software triggers the DSLs left / right lenses to realize the collection of "1000us per interval" (200us-10000us left / right two groups of 22 pictures), and the two groups of image sequences can be approximately regarded as representing the "exposure stack" of the scene dynamic range, as shown in Figure 2 and Figure 3 Therefore, it can be used for the estimation or calibration of the response function under the DSLs dual lens.

[0046] 2) Ideal visual exposure-HDR model

[0047] a) Classical imaging model

[0048] According to the classical Retinex theory model, the actual light reaching the observer's eye, or can be understood as the image irradiance H(x) captured by the camera sensor, can be composed of the reflectance component R(x) and the illumination component L(x) of the actual scene:

[0049]

[0050] In the formula, the operator represents the pixel-based point multiplication. In theory, H(x) is the high dynamic scene imaging, and in practice, it is processed by CNRF f in almost most of the existing cameras:

[0051] S(x) = f(H) (2)

[0052] The back-end imaging S(x) for generating pleasing user vision is also the LDR image captured by DSLs under low exposure setting. Here, CNRF f describes the nonlinear relationship between image irradiance H(x) and pixel value S(x). When the exposure setting value of the camera changes, the irradiance H reaching the camera CMOS sensor will change linearly. However, considering the nonlinear processing process inside the camera, the image intensity S will not change linearly with the irradiance H, but nonlinearly. Therefore, the mapping function between different exposure images is also a nonlinear function, also known as the intensity nonlinear mapping function, namely INMF.

[0053] b) Mann theory mapping model

[0054] The INMF proposed by Mann is expressed as:

[0055]

[0056] Where is the INMF, and e is the exposure ratio. CNRF and INMF are both nonlinear functions describing the camera imaging processing, and the two constitute the camera response model.

[0057] Combining the above formulas, the following corresponding equation can be obtained:

[0058]

[0059] This equation is called the co-parameter equation, which describes the relationship between f and , and can also be used for mutual conversion between the two.

[0060] c) INMF definition of the present application

[0061] Unlike the above method, the present application proposes to define the INMF of the low dynamic EFIS collected by DSLs as:

[0062]

[0063] It is worth noting that, unlike the exposure ratio e in the foregoing, E in the present application is a matrix, representing the desired exposure ratio of each pixel within it. The sequence form number j = 1,... 18 of the encoded fringe pattern image.

[0064] d) Ideal visual exposure model

[0065] Under-exposed DSLs images are generally classified into globally under-exposed and locally under-exposed images. The former is mainly due to its low illumination, while the latter is mainly due to non-uniform illumination and the limited dynamic range of industrial cameras. Therefore, in an embodiment, the ideal exposure state is assumed to be: adjusting the illumination to be uniform and high intensity, which not only restores the information "submerged" in low brightness, but also enables the camera to fully record the scene encoding information, i.e. HDR. Accordingly, the ideal visual exposure model (or the enhanced image for HDR restoration) can be written as:

[0066]

[0067] wherein, represents the matrix in which all elements of the assumed illumination L are adjusted to 1, i.e.

[0068] e) INMF definition under the HDR restoration model

[0069] Jointly formula (1), (2), (5) and (6), the INMF result under the ideal HDR restoration can be rewritten as:

[0070]

[0071] wherein, the symbol represents pixel-by-pixel division. Next, how to construct the camera response function approximating the actual scene under the assumed function form becomes the key. The following steps S2 and S3 realize the calibration and SNF optimization fitting of the actual CNRF, respectively, and the exposure ratio matrix and INMF estimated by steps S4 and S5, respectively, further realize the high dynamic restoration of the EFIS captured by the DSLs under the low exposure setting.

[0072] Step S2, in offline mode, analyze and calibrate the camera nonlinear response function.

[0073] 1) CNRF intrinsic properties

[0074] As described by researcher Nayar, CNRFξ should possess three assumed properties: a) Consistency, meaning it is identical for all pixels on the camera sensor. In other words, each pixel of the camera responds consistently under the same lighting conditions, ignoring any minor differences that may exist between pixels; b) Normalizability, meaning its output can be scaled to a range of [0,1], which facilitates comparison and analysis between different cameras or different coded images; c) Monotonicity, meaning it is monotonically increasing, implying that pixel values ​​increase with increasing incident light intensity.

[0075] 2) CNRF Definition

[0076] Combining the three assumptions, in one embodiment of the present invention, the theoretical space Φ of the camera's nonlinear response function ξ is... ξ Defined as:

[0077]

[0078] Where (x, y) represent spatial coordinates. Clearly, equation (8) implies the intensity nonlinear mapping function. It has the same hypothetical properties as ξ.

[0079] 3) Debevec-CNRF estimation

[0080] In offline mode, the camera response function f is estimated from images acquired with adjusted exposure using the Debevec authors' method. debevec ,like Figure 6 As shown. This function can be fitted using the unknown parameters of the Sigmoid function described below. It is important to note that only by fitting a parameterized Sigmoid function can it be converted into a parameterized brightness mapping function; otherwise, using a non-linear f... debevec However, the function cannot derive INMF, which is directly used for brightness nonlinear mapping.

[0081] Step S3: In offline mode, establish the Sigmoid nonlinear function and perform optimization fitting and parameter estimation.

[0082] 1) Reasons for choosing Sigmoid

[0083] a) First, the Sigmoid curve distribution conforms to the intrinsic properties of CNRF;

[0084] b) Secondly, the sigmoid curve fitting result is infinitely close to the response curve calibrated by the real camera provided by Nayar, such as Figure 8 As shown.

[0085] In the traditional classic Debevec study, the authors proposed a method to recover high dynamic range radiance maps using multiple photos taken with traditional imaging devices at different exposure times. This method involves taking multiple photos of the same scene at different exposures and using these different exposures to recover the response function of the imaging process through an algorithm. When the response function is obtained, the multiple photos can be fused into a single high dynamic range radiance map, the pixel value of which is proportional to the true radiance value in the scene. However, this method is not directly applicable to the EFIS collected online by high dynamic DSLs, because: it is mainly used to recover the high dynamic range of the scene from multiple photos at different exposures, rather than directly enhancing the EFIS taken by DSLs at low exposure settings; this method is based on the same light, adjusting different exposure times, i.e. each frame of high dynamic range video is composed of several frames with different exposure amounts. This generation method requires a high sampling frequency of the hardware device, which often makes the hardware device difficult to respond, resulting in jitter in the generated video. It is impossible for ordinary industrial cameras to quickly produce different exposure images.

[0086] 2) Two-parameter (unknown) Sigmoid function establishment

[0087] In one embodiment of the present application, a two-parameter (unknown) Sigmoid function based on the human visual system is proposed as a model for fitting the camera response function, i.e. SNF:

[0088]

[0089] where Z represents the pixel intensity value of the input encoded image sequence, and a and β are the unknown parameters of the model. It should be noted that the reference CNRF f assumption is used to calculate f sigmoid as to calculate the f assumption characteristics that do not exist blur.

[0090] 3) Sigmoid function and Debevec method estimated CRF curve fitting (estimated at the same light, different time periods)

[0091] Sigmoid function with double (unknown) parameters and Debevec method estimated nonlinear response function f debevec fit as follows:

[0092]

[0093] 4) Optimization - fitting to estimate unknown double parameters of Sigmoid function

[0094] In one embodiment of the present application, the quasi-Newton optimization technique is used to solve the optimization problem in equation (10). In the experiment, the estimated values (a, b) = (0.03, 1.2) are provided after optimization.

[0095] It can be derived from equations (3) and (9) that the luminance mapping function of LDR EFIS is nonlinearly adjusted using the common parameters (a, b) which can be expressed as:

[0096]

[0097] This expression is the intensity nonlinear mapping function containing the double (known) parameters SNF. It is worth noting that the implementation of equation (11) is dominated by the exposure ratio e.

[0098] Step S4, in the online mode, estimate the EFIS illumination matrix map.

[0099] 1) EFIS illumination initial value estimation

[0100] As can be seen from equations (5) and (7) above, the exposure ratio matrix E estimated can be directly obtained in inverse proportion to the illumination component L. Considering that the exposure ratio matrix E is inversely proportional to the illumination component map, the solution of E can be achieved by estimating the illumination component map L. In other words, the global optimization solution proposed for the estimation of the illumination component L can be used as the estimated value of the exposure ratio matrix E. In order to simplify, for each pixel value x, the EFIS directly acquired by DSLs in the online mode is first used as the initial estimate of the illumination, i.e. L = S. The physical meaning of the above operation: maintain the encoding information consistent with the original input EFIS of the illumination component; the initial value of the illumination is consistent with the original input EFIS intensity distribution information.

[0101] 2) EFIS illumination MAP prior and constraint

[0102] In theory, estimating the refined illumination of the scene is a pathological problem, and the prior consistent with the actual illumination needs to be introduced to become the key to solving the problem. The present application assumes that the scene where the actual DSLs is located is a natural illumination with a segmented smoothness in space, which also conforms to the actual illumination distribution prior. In addition, a small window weighting matrix is designed as the structural weight guide of the optimization function below, which is represented as:

[0103]

[0104] where h represents the horizontal direction, v represents the vertical direction, and L(y) represents the illumination intensity at position y in the neighborhood centered at x. Here, the operator | · | is the absolute value operation, ​It is a local window centered at x with a radius of 3, and the small constant θ = 0.001 is used to avoid a denominator of 0. The first-order difference filter in the formula... Includes the horizontal direction and vertical direction Operation. The refined illumination component map can be solved using the following optimization function:

[0105]

[0106] in, Let L(x) represent the estimated initial illumination value, L(x) represent the illumination component to be optimized, and W represent the initial illumination value. L，φ This represents the structural weights. Here, the regularization coefficient λ... L This is used to balance the two terms in the optimization function. Intuitively, it can be observed that the above optimization problem only contains quadratic terms, and the closed-form solution can be directly obtained. Furthermore, the solution can be calculated in only one step. Therefore, this problem can be solved directly by solving the following equation:

[0107]

[0108] Among them, D φ Includes Dh and D v It is the Toeplitz matrix of the discrete gradient operator with forward difference. l represents the vector form of the illumination component L(N) to be optimized. Represents the initial illumination component In vector form. Here, yes The vector form, It is an identity matrix of appropriate size. Meanwhile, the operator Diag(x) reconstructs a diagonal matrix using the vector X. Because of the above formula... It is a symmetric positive definite Laplacian matrix, and therefore, it can be solved using techniques such as preprocessing conjugate gradients.

[0109] 3) Illumination regularization function and its solution

[0110] Through local weighting weight W L，φ Guided by the above optimization problem, a refined illumination component map L can be obtained that preserves structure, compresses weak texture, and exhibits piecewise smoothness characteristics consistent with spatial illumination distribution. The inverse of L (1. / L) can be used as the value of the exposure ratio matrix E, denoted as: E a .

[0111] Step S5, in online mode, performs highly dynamic recovery for EFIS.

[0112] 1) Combining known conditions

[0113] The exposure ratio matrix estimated according to step S4, the control SNF to realize the INMF double parameters are respectively denoted as E a , a and β.

[0114] 2) EFIS high dynamic formula model establishment

[0115] Combined with the proposed INMF model, the estimated exposure ratio matrix and the double parameter control coefficient, the EFIS proposed by the application for the LDR state in DSLs: S j (x) realizes online high dynamic recovery results Can be expressed as:

[0116]

[0117] Here, S j (x) represents the sequence form S j (x), j = 1,...18 of 18 encoded fringe images collected by DSLs online, and the high dynamic recovery result thereof is

[0118] In summary, the application proposes a high dynamic enhancement method of an auxiliary dual structured light system integrated with offline-online dual mode cooperation, which constructs a system nonlinear response function conforming to the actual scene through optimization fitting in the offline mode, that is, the parameter-optimized Sigmoid nonlinear function, as shown in Figure 7 And Figure 8 Then, the fine estimation of the initial value of illumination is realized on the low-exposure DSLs image collected by the current scene by combining the alternating direction minimization and the fast Fourier method; finally, combined with the Sigmoid function with known parameters and the fine estimation of the illumination map, the nonlinear mapping of the low-exposure DSLs image sequence to the HDR image sequence is realized through the ideal mapping formula in the online mode, and the enhanced result with high visibility and high signal-to-noise ratio is generated, as shown in Figure 4 The LIME algorithm has unreasonable assumptions about components in the assumptions of the model, directly generates problems such as artifacts and overexposure in local areas, thereby causing the original encoded information to be destroyed, and indirectly causing the reconstruction result of the structured light method to have the problem of being unable to recover the 3D topography, as shown in Figure 5 Although the STAR algorithm can obtain an enhanced result with better visual expression, the estimated illumination component is not ideal due to the initial unreasonable assumption, thereby the problem of the encoded details not being obvious in the result, and indirectly causing the reconstruction result of the structured light method to have the problem of a large amount of detail loss, as shown in Figure 5The present application takes into account the non-linear response function inside the camera, in combination with the non-linear response function estimation and high dynamic formula of the dual mode of "off-line" and "on-line", to realize the high-quality recovery (or can be understood as: high visibility, undamaged coding information, high signal-to-noise ratio, etc.) of the EFIS image sequence of low exposure.

[0119] It should be noted that the above-mentioned embodiments can be appropriately changed or modified by those skilled in the art without departing from the spirit and scope of the present application. For example, the Quasi-Newton optimization technique used in the present application for fitting the true Debevec-CNRF of DSLs and obtaining parameter-optimized SNF can be replaced by other techniques, and is not limited to the optimization technique used in the present application. For another example, the joint alternating direction minimization and fast Fourier solution method used in the illumination initial value refinement estimation proposed in the present application can be realized by other solution schemes, such as QR fast decomposition, low-rank space conversion, etc.

[0120] In summary, compared with the prior art, the present application has the following advantages:

[0121] 1) The present application provides an efficient scheme of off-line parameter estimation and on-line non-linear HDR mapping, which is efficient in on-line processing and mapping for low-exposure EFIS, and has high visibility, high signal-to-noise ratio and high-quality visibility.

[0122] 2) The "exposure stack" is defined and realized in the off-line mode, which indirectly captures the wide dynamic range imaging in the actual scene.

[0123] 3) The true Debevec-camera non-linear response function (CNRF) fitting of the dual structured light system is realized by the Quasi-Newton optimization technique, and the parameter-optimized SNF is obtained.

[0124] 4) The high dynamic recovery formula is constructed by combining the known parameter Sigmoid function, the refined estimated illumination map and the ideal mapping condition, and then the low-exposure EFIS is efficiently mapped to the HDR state EFIS in the on-line state.

[0125] 5) The present application not only realizes the non-linear high-quality intensity mapping in the "on-line mode", but also can be integrated into the embedded high-performance development platform to realize real-time processing, and has high use value and wide application scenarios.

[0126] 6) The application estimates the INMF in the offline state, and directly maps the EFIS collected by the DSLs in the online mode (phase) into high dynamic EFIS, avoiding the degradation problems (such as low visibility, low signal-to-noise ratio) that occur when capturing EFIS online, solving the problem that the depth cannot be accurately estimated and 3D reconstruction cannot be performed under fixed low exposure settings, and thus the reconstruction capability of the system for high-speed moving objects can be improved.

[0127] In summary, the application proposes a method for determining the nonlinear response function of the camera by changing the illumination condition instead of the exposure time under the same scene, realizing the conversion of the low dynamic range EFIS into the high dynamic range EFIS. Meanwhile, considering the inaccuracy of the existing image processing theoretical model, the application introduces the intensity nonlinear mapping function, directly realizes the high dynamic of the system coded fringe pattern, for example, the Sigmoid nonlinear function optimized by the estimated parameters in the offline state is used as the INMF, and the captured low dynamic range EFIS is mapped into the high dynamic range EFIS signal required by the structured light algorithm in the online mode. It is verified by experiments that, by using the application, high-speed and high-quality reconstruction of the dual structured light system can be realized under low exposure settings.

[0128] The application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions loaded thereon for causing a processor to implement various aspects of the application.

[0129] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch cards or punched tape, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0130] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0131] Computer readable program instructions for carrying out operations of the present application can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computing / processing device, partly on the user's computing / processing device, as a stand-alone software package, partly on the user's computing / processing device and partly on a remote computing / processing device or entirely on the remote computing / processing device or server. In the latter scenario, the remote computing / processing device can be connected to the user's computing / processing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing / processing device, for example, through the Internet using an Internet Service Provider. In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0132] The computer readable program instructions can also be loaded onto a computing / processing device, other programmable data processing apparatus, or other device to cause a series of operations to be performed on the computing / processing device, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computing / processing device, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0133] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer- readable storage medium having no data, programs, program modules, e.g., instructions for operation, or digital content stored thereon or therein for a short time or not at all. The computer readable storage medium can also have instructions stored thereon or therein which may

[0134] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0135] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0136] Having described various embodiments of the application, it is to be understood that the above description is meant to be illustrative only and not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art, without departing from the scope and spirit of the described embodiments. The selection of terms to be used in the description is intended to best explain the principles of the embodiments, the practical application, or technical improvement over the prior art, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the application is defined by the claims appended hereto.

Claims

1. A method for HDR of a dual structured light system assisted by an off-line-online dual-mode collaborative strategy, comprising the following steps: in the off-line mode, defining an intensity nonlinear mapping function INMF under a high dynamic range recovery model based on an ideal visual model; in the off-line mode, calibrating a camera nonlinear response function CNRF to obtain an estimated camera nonlinear response function; in the off-line mode, establishing a Sigmoid nonlinear function with unknown double parameters and fitting the double parameters to estimate, and then using the estimated double parameters as a common parameter to nonlinearly adjust a luminance mapping function of a low dynamic range encoded fringe image sequence EFIS; in the online mode, estimating an illumination matrix map of the encoded fringe image sequence EFIS to obtain an estimated illumination component map and an exposure ratio matrix; in the online mode, using the estimated exposure ratio matrix and the double parameters to obtain a high dynamic range recovery result from the low dynamic range encoded fringe image sequence EFIS; wherein the intensity nonlinear mapping function INMF under the high dynamic range recovery model is expressed as: wherein, denotes a pixel value, the symbol denotes a pixel-wise division, denotes a reflectance component of the actual scene, denotes an illumination component, denotes an image irradiance, for describing the image irradiance and the non-linear relationship between the pixel value, is an intensity non-linear mapping function, is an identity matrix, is a backend imaging for generating a flattering user vision, j denotes a sequence index of the encoded fringe image; where the estimated camera non-linear response function is obtained according to the following fitting equation : wherein the luminance mapping function of the low dynamic range encoded fringe image sequence EFIS is adjusted according to the following formula: wherein, is an estimated camera non-linear response function, and are unknown parameters to be solved, and Z represents the pixel intensity values of the input coded image sequence; wherein the illumination component map L is fitted by the following optimization function: wherein, represents a luminance mapping function of the encoded fringe image sequence EFIS of low dynamic range. wherein the exposure ratio matrix is expressed as: the optimization function of the illumination component map L is solved by the following formula: wherein, is a regularization coefficient, the operator is an absolute value operation, is a local window centered at and set to a radius size, is a set constant, a first order difference filter contains horizontal direction and vertical direction operations, the estimated exposure ratio matrix is obtained in an inverse proportion manner, is an initial estimate of the illumination component , the operator represents the horizontal direction, h represents the vertical direction, v represents is a neighborhood centered at x the illumination intensity at position y , the operator represents the illumination component to be optimized estimate, represents the structure weight.

2. The method of claim 1, wherein, The computer program is executed by a processor to implement the steps of the method according to any one of claims 1 to 4. where is a vector form of is the identity matrix, the operator is used to reconstruct a diagonal matrix from the vector contains and is a Toeplitz matrix of the discrete gradient operator with forward difference, is a vector form of is a vector form of​ 3. The method of claim 2, wherein, For each pixel value The initial estimate of the illumination, denoted as , is obtained using the sequence of encoded fringe images EFIS acquired on-line with the dual structured light system.

4. The method of claim 1, wherein, The high dynamic recovery result is expressed as: wherein, represents a sequence form of a plurality of coded fringe images collected by the dual structured light system in line, represents an estimated exposure ratio matrix, and represents an estimated dual parameter.

5. A computer readable storage medium having stored thereon a computer program, wherein, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 4.

6. A computer device comprising a memory and a processor, having stored on the memory a computer program capable of running on the processor, characterized in that, ​

Images