Method for determining one or more sets of exposure settings for a three-dimensional image acquisition process

Abstract

A method for determining one or more sets of exposure settings for a three-dimensional image acquisition process implemented by an imaging system, the three-dimensional image acquisition process comprising capturing one or more sets of image data by the image sensor with corresponding sets of exposure settings, wherein the one or more sets of image data can be used to generate one or more three-dimensional point clouds that determine the three-dimensional coordinates of points on the surface of one or more imaged objects, each set of exposure settings determining the value of one or more parameters of the imaging system, wherein the method comprises: determining one or more sets of candidate exposure settings; and selecting one or more sets of exposure settings that meet one or more optimization criteria from the sets of exposure settings.

Description

Technical Field

[0001] Embodiments described in this application are directed to a method for determining one or more exposure settings for a three-dimensional image acquisition process. Embodiments described in this application are also directed to a system for generating a three-dimensional image of an object. Background Art

[0002] Three-dimensional surface imaging (3D surface imaging) is a rapidly developing technology. The term "3D surface imaging" as used in this application is understood to refer to the process of generating a three-dimensional representation of the surface of an object by capturing spatial information in three dimensions (that is, capturing depth information in addition to the two-dimensional spatial information in traditional images or photographs). This three-dimensional representation can be visually displayed as a "three-dimensional image" on a screen, for example.

[0003] A variety of different techniques can be used to obtain the data needed to generate a three-dimensional image of an object's surface. Such techniques include, but are not limited to, structured light illumination, time-of-flight imaging, holography, stereo systems (active and passive), and laser line triangulation. In each of these techniques, the data can be captured in the form of a "point cloud." In this case, the intensity values ​​at different points in three-dimensional space are recorded, so that each point in the point cloud has its own set of (x, y, z) coordinates and a corresponding intensity value I.

[0004] Figure 1 The figure shows an exemplary storage method that can be used when storing point cloud data in a memory. As shown in the figure, the point cloud data is stored in the form of a two-dimensional point cloud matrix 101 with N rows and M columns. Each element of the matrix includes a data tuple, which includes {x ij ,y ij ,z ij} coordinates, where i = {1, 2...N}, j = {1, 2,...M}, and N and M are both integer values. The data tuple also includes the intensity value I of the point ij The intensity values ​​together with the corresponding spatial coordinates determine the outer surface geometry of the object. By representing each point in three dimensions, a three-dimensional representation of the object can be obtained.

[0005] In some cases, one or more two-dimensional images of an object may be acquired by a two-dimensional sensor array, and point cloud data may be calculated based on these two-dimensional images. In such cases, the point cloud matrix elements may be mapped to pixel elements of the sensor array. For example, for a given matrix element, the index symbols i and j may represent the position of the corresponding pixel element within the sensor array. Thus, {x ij ,y ij ,z ij} coordinates can determine the position of a point in space, which corresponds to the position of the corresponding pixel in one or more two-dimensional images.

[0006] As with all forms of imaging, the signal-to-noise ratio (SNR) of the resulting three-dimensional image depends in part on the dynamic range of the sensor used in capturing the image data. If the signal strength received from different points on an object can vary significantly, for example because the object's surface contains several bright and dark areas, a balance must be struck between: (1) maximizing the illumination intensity and / or sensor exposure time to ensure that sufficient light is received from the darker areas of the object; and (2) minimizing the illumination intensity and / or sensor exposure time to avoid saturating the sensor with signals from the brighter areas of the object. One solution that has been proposed to address this problem is to apply the concept of high dynamic range (HDR) imaging to three-dimensional imaging. HDR imaging is a well-established technique in the field of digital imaging for increasing the photometric dynamic range. The technique involves capturing several images of the same scene at different exposure times; and generating a single HDR image of the scene by post-processing the image data. Images captured with longer exposure times are able to capture details in the darker areas of the scene that are not discernible in images captured with shorter exposure times due to insufficient signal reaching the camera. At the same time, images captured with shorter exposure times are able to capture details in brighter areas of the scene that cannot be resolved in images captured with longer exposure times due to camera saturation. By post-processing such images with a suitable HDR algorithm, a single high-definition image can be obtained in which detailed elements in both bright and dark areas are visible.

[0007] Generally speaking, the principles of HDR imaging can be applied to 3D imaging in the same way as traditional 2D imaging. Figure 2 An example of this implementation is schematically shown. Similar to the two-dimensional HDR imaging method, multiple sets of image data can be captured at different exposure settings, but the difference is that each set of image data in this case includes three-dimensional image data rather than two-dimensional image data. That is, each set of image data specifies the three-dimensional position of each point in the viewed scene. Each set of image data can be stored in the form of a point cloud matrix 201a, 201b, 201c, respectively. Subsequently, the individual point cloud matrices can be merged into a single point cloud matrix 203 that can be used to present a three-dimensional HDR image. However, compared with two-dimensional HDR imaging, three-dimensional HDR imaging does have additional challenges. One of the problems is that compared with two-dimensional imaging, in order to obtain the three-dimensional coordinates of each point, additional calculation steps must be added, wherein the impact of any deficiency in the SNR of the captured image data will be amplified by the additional calculation steps, thereby seriously affecting the calculation accuracy of the three-dimensional coordinates of each point in the output image. In addition, it is difficult to identify the appropriate exposure setting when capturing each set of image data used to generate the final three-dimensional image.

[0008] Therefore, there is a need to provide improved three-dimensional HDR image generation technology. Summary of the invention

[0009] According to a first aspect of the present invention, there is provided a method for determining one or more sets of exposure settings for a three-dimensional image acquisition process performed by an imaging system, the imaging system comprising an image sensor, the three-dimensional image acquisition process comprising capturing one or more sets of image data by the image sensor at respective sets of exposure settings, wherein the one or more sets of image data can be used to generate one or more three-dimensional point clouds determining the three-dimensional coordinates of points on the surface of one or more imaged objects, each set of exposure settings specifying the value of one or more parameters of the imaging system that have an impact on the amount of signal reaching the image sensor, the method comprising:

[0010] (1) identifying one or more sets of candidate exposure settings using image data captured by the image sensor;

[0011] (2) For each set of candidate exposure settings:

[0012] determining an amount of signal that different pixels of the image sensor may receive when the set of candidate exposure settings is used to capture a set of image data used by the three-dimensional image acquisition process;

[0013] determining, based on the amount of signals that the different pixels may receive, whether each pixel will be a well-exposed pixel when the set of candidate exposure settings is used, wherein a well-exposed pixel is a pixel having an associated quality parameter value greater than a threshold, the quality parameter value of the pixel reflecting a degree of uncertainty in a three-dimensional coordinate of a point in a point cloud associated with the pixel when a set of image data captured with the set of candidate exposure settings is generated;

[0014] determining an exposure cost, wherein the exposure cost is derived from values ​​of the one or more parameters of the set of candidate exposure settings; and

[0015] (3) selecting one or more sets of exposure settings for the three-dimensional image acquisition process from the one or more sets of candidate exposure settings, wherein the selection satisfies one or more optimization criteria, wherein the one or more optimization criteria are criteria defined in the following aspects:

[0016] (a) the number of pixels in a group N, wherein a pixel belongs to the group N if the pixel is determined to be a well-exposed pixel for at least one selected set of exposure settings; and

[0017] (b) Exposure cost of the selected one or more sets of exposure settings.

[0018] In some embodiments, each set of image data in the one or more sets of image data can be used to generate a corresponding three-dimensional point cloud. In such embodiments, each set of image data can include image data captured with a corresponding set of exposure settings, which is then combined with previously captured image data to form a new set of image data that can be used to generate a three-dimensional point cloud.

[0019] In some embodiments, the different pixels include a subset of all pixels of the image sensor. In some embodiments, all of the different pixels belong to a predetermined area of ​​the image sensor. In some embodiments, the different pixels include all pixels in the image sensor.

[0020] In some embodiments, for each set of candidate exposure settings, the method includes:

[0021] identifying one or more sets of alternative candidate exposure settings for which the one or more parameters of the imaging system have different values ​​but the same expected amount of received signal from the image sensor; and

[0022] For each set of alternative candidate exposure settings, determining an exposure cost, wherein the exposure cost is derived from the one or more parameters of the set of alternative candidate exposure settings,

[0023] The one or more sets of alternative candidate exposure settings are selectable for use in the three-dimensional image acquisition process.

[0024] The one or more sets of candidate exposure settings may be selected to ensure that a ratio of the number of pixels in the set N to the exposure cost of the selected one or more sets of exposure settings meets a criterion.

[0025] The one or more sets of candidate exposure settings may be selected to ensure:

[0026] (a) the number of pixels in the group N meets the first criterion; and

[0027] (b) The exposure costs of the selected one or more sets of exposure settings meet the second criterion.

[0028] The first criterion may be to maximize the number of pixels belonging to the group N. The first criterion may be to ensure that the number of pixels belonging to the group N is greater than a threshold.

[0029] The second criterion may be to minimize the total exposure cost. The second criterion may be to ensure that the total exposure cost of each selected set of exposure settings is less than a threshold.

[0030] In some embodiments, for one or more groups of exposure settings in each group of exposure settings, the step of determining the amount of signal that different pixels of the image sensor may receive when the group of candidate exposure settings is used to capture a set of image data includes: capturing a set of image data with the group of candidate exposure settings.

[0031] Each set of image data captured at the one or more sets of candidate exposure settings may be used to identify one or more other sets of candidate exposure settings.

[0032] In some embodiments, steps (1) to (3) are repeated one or more times, wherein for each iteration:

[0033] selecting a set of exposure settings from the one or more sets of candidate exposure settings identified at that iteration; and

[0034] The optimization criteria include: a first criterion for maximizing the number of pixels in the group N; and a second criterion for ensuring that the sum of the exposure cost of a group of exposure settings selected in the current iteration and the corresponding exposure costs of each group of exposure settings selected in all previous iterations is less than a threshold.

[0035] For each iteration, the selected set of exposure settings may be used to capture a set of imaging data with the imaging system,

[0036] For each iteration after the second iteration, each set of candidate exposure settings for the current iteration is determined using the set of image data captured in the previous iteration.

[0037] Determining whether each pixel is a well-exposed pixel at a set of candidate exposure settings may include determining a probability that each pixel is well-exposed, the probability being determined based on the amount of signal received by the pixel in a previous iteration of the method.

[0038] The exposure cost of each set of exposure settings may be a function of the exposure time used by the set of settings.

[0039] Identifying one or more sets of candidate exposure settings may include determining, for one or more pixels of the image sensor, a series of exposure times that would make the pixel likely to be a well-exposed pixel.

[0040] The value of the quality parameter associated with a pixel may be determined based on the amount of ambient light in the imaged scene.

[0041] In some embodiments, each set of exposure settings includes one or more of the following:

[0042] exposure time of the image sensor;

[0043] the size of the aperture in the optical path between the object and the sensor;

[0044] The intensity of light used to illuminate the object; and

[0045] The strength of the neutral density filter placed in the light path between the object and the sensor.

[0046] The imaging system may be an optical imaging system including one or more light sensors. The imaging system may include one or more light sources for illuminating the imaged object.

[0047] The image data in each set of image data may include one or more two-dimensional images of the object captured by the sensor.

[0048] The imaging system may be an imaging system that obtains each set of image data through structured illumination.

[0049] Each set of image data may include a series of two-dimensional images of the object captured by the light sensor, each two-dimensional image in the series of two-dimensional images being captured with a different illumination pattern.

[0050] Each set of image data may include a series of Gray code images and a series of phase-shifted images.

[0051] Each set of image data may include color information.

[0052] According to a second aspect of the present invention, there is provided a method for generating a three-dimensional image of one or more objects using an imaging system including an image sensor, the method comprising:

[0053] capturing one or more sets of image data by the image sensor at respective exposure settings, the sets of image data being usable to generate one or more three-dimensional point clouds defining three-dimensional coordinates of points on the surfaces of the one or more objects, each set of exposure settings specifying values ​​of one or more parameters of the imaging system that affect the amount of signal reaching the image sensor; and

[0054] constructing a three-dimensional point cloud using data from one or more sets of image data from each set of captured image data,

[0055] Therein, the exposure settings used in capturing each set of image data are determined by the method according to the first aspect of the present invention.

[0056] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium comprising computer executable code, which, when executed by a computer, enables the computer to implement the method of the first aspect of the present invention.

[0057] According to a fifth aspect of the present invention, there is provided an imaging system for implementing a three-dimensional image acquisition process by capturing one or more sets of image data with one or more sets of exposure settings, wherein the one or more sets of image data can be used to generate one or more three-dimensional point clouds for determining the three-dimensional coordinates of points on the surface of one or more imaged objects, the imaging system comprising an image sensor for capturing the one or more sets of image data, the imaging system for determining the one or more sets of exposure settings for the three-dimensional image acquisition process by implementing the method according to the first aspect of the present invention.

[0058] The embodiments described in the present application provide a means for obtaining a three-dimensional HDR image of an object. The three-dimensional HDR image is obtained by capturing multiple sets of image data. The image data can be obtained by one of a variety of techniques known in the art. Such techniques may, for example, include structured light illumination, time-of-flight imaging, holographic technology, stereo imaging (including active and passive), and laser line triangulation. Each set of image data can be used to calculate a corresponding input point cloud, which specifies the three-dimensional coordinates of a point on the surface of the object and its intensity. The information of each set of image data can be combined or merged to provide a single output point cloud, which can be used to present a three-dimensional image of the object.

[0059] The sets of image data may be acquired with multiple sets of different exposure settings so that the exposure of the imaging system for each set of image data is different. Here, "exposure settings" refers to physical parameters that can be adjusted to change the exposure mode of the imaging system. In the case of using an optical imaging system, a set of exposure settings may include parameters that have a direct impact on the amount of incident light on the image sensor. Such parameters include, for example, one or more of the following: the integration time of the camera / light sensor; the aperture size of the camera / light sensor; and the light intensity used to illuminate the scene or object. The set of exposure settings may also include the strength of a neutral gray filter placed in the light path between the object and the camera / light sensor. The strength of the neutral gray filter may be different each time a new set of image data is acquired to change the amount of light reaching the camera or light sensor.

[0060] It will be appreciated that in addition to the parameters that affect the amount of light incident on the sensor, other parameters of the system may be varied between different exposure operations. Such parameters may include the sensitivity of the camera or light sensor. The sensitivity may be varied, for example, by adjusting the gain applied to the device.

[0061] In some embodiments, only one of the above exposure settings may be changed for each exposure, while the other exposure settings remain unchanged each time a new set of image data is collected. In other embodiments, two or more of the above settings may be changed each time a new set of image data is captured.

[0062] When image data is acquired at a greater exposure, the signal detected from the darker areas of the object or scene can be maximized. In this way, the SNR of points in such darker areas can be improved, thereby improving the accuracy of calculating the spatial coordinates of these points when the corresponding set of image data is converted into a point cloud. In contrast, when the exposure is reduced by adjusting the exposure setting, the signal captured from the brighter areas of the object or scene can be maximized without causing saturation of the camera or light sensor. Therefore, by reducing the exposure, a second set of image data with an improved SNR for the points in the brighter areas can be obtained, and a second point cloud with an improved accuracy of calculating the spatial coordinates of the points in the brighter areas can be generated. Subsequently, the image data at these different exposures can be combined to ensure that the SNR of both the brighter and darker areas of the object in the output three-dimensional image is improved.

[0063] In order to determine how to best combine the data at each exposure, in the embodiment described in the present application, an additional “quality parameter” is used to evaluate each point in each set of image data. Figure 3 FIG. 3 is an example of a point cloud matrix 301 used in one embodiment. Figure 1 Compared with the traditional point cloud matrix, Figure 3 Each element of the point cloud matrix contains an additional value q ij The quality parameter value q ij Reflects the corresponding point {x ij ,y ij ,z ij} coordinates. In some embodiments, the quality parameter value can achieve the corresponding point {x ij ,y ij ,z ij The best estimate of the expected error in the value of .

[0064] Depending on the specific technique used when acquiring the image data, the quality parameter can be determined in one of a number of different ways. In one example, which is further detailed below, a structured illumination method is used to acquire image data for calculating an output point cloud. In this case, the value of the quality parameter for a given point on the surface of an object can be obtained, for example, based on the contrast observed when the light and dark stripes of a structured illumination pattern are projected onto the point. That is, the quality parameter can be related to or derived from the difference between the amount of light detected at the point when illuminated by the light stripes of the illumination pattern and the amount of light detected at the point when illuminated by the dark stripes of the illumination pattern. In some embodiments, a series of sinusoidally modulated intensity patterns can be projected onto the object, and these successively projected intensity patterns can be phase-shifted relative to each other. Therein, the quality parameter can be related to or derived from the intensity measured at each point when the intensity patterns are projected onto the object.

[0065] In another example, when performing three-dimensional imaging with a time-of-flight system, the amplitude of the recovered signal at a given point can be compared with a measure of the ambient light of the system to obtain a quality parameter value at that point. In yet another example, when using a stereoscopic imaging system, the quality parameter can be obtained by comparing and matching features identified from an image captured by a first camera with features identified from an image captured by a second camera; and determining a match quality score. The score can be calculated, for example, as the sum of absolute differences (SAD):

[0066]

[0067] Where (r,c) is the image pixel coordinate, I1, I2 is the stereo image pair, (x,y) is the disparity to be evaluated by SAD, and (A,B) is the matching array dimension (in pixels). The disparity calculation value (x′,y′) of pixel (r,c) is calculated according to (x′,y′)=argmin x,y SAD(r,c,x,y) calculation. Based on the calculated SAD value, it can be further interpreted as a measure of the uncertainty of the three-dimensional coordinates of each point in the point cloud reconstructed from the stereo image. If necessary, the SAD can be normalized, for example, for regional image intensity to improve performance.

[0068] Regardless of the specific imaging method used to obtain the image data, the quality parameter can be used to determine the extent to which the data of each exposure is taken into account when obtaining the coordinate values ​​of the points in the final three-dimensional image. In some embodiments, each set of image data can be used to construct a corresponding input point cloud matrix. Then, for a given element {i, j} in the point cloud matrix used to construct the final three-dimensional image, a set of coordinate values ​​{x ij ,y ij ,z ij} can be obtained by adding the corresponding sets of coordinate values ​​{x ij ,y ij ,z ij The weight value applied to each matrix may depend on the quality parameter value q of the position in the matrix. ij In this way, {x ij ,y ij ,z ij The} values ​​are adjusted in a direction so that the sets of image data used to obtain these values ​​have a higher SNR at the points of interest.

[0069] In another example, a set of coordinate values ​​{x ij ,y ij ,z ij} can be selected as {x in a single input point cloud matrix among the input point cloud matrices ij ,y ij ,z ij The input point cloud matrix where the selected value is located may be an input point cloud matrix having the highest q value at the point compared with other input point cloud matrices.

[0070] In another example, for a given element {i, j} of the output point cloud matrix, a set of coordinate values ​​of the point {x ij ,y ij ,z ij} can be obtained by converting each set of coordinate values ​​{x ij ,y ij ,z ij That is, the value x of the output point cloud matrix is ij It can include the corresponding value x of point {i,j} in each input point cloud matrix ij The average value of the output point cloud matrix y ij It can include the corresponding value y of point {i,j} in each input point cloud matrix ij The average value of the output point cloud matrix is ​​z ij It can include the corresponding value z of point {i,j} in each input point cloud matrix ij The average value of . A threshold criterion may be applied in the averaging operation, wherein only the values ​​associated with q values ​​above the threshold are used to calculate the average value. Thus, for example, if the value of an element of one of the input point cloud matrices {x ij ,y ij ,z ij} is lower than the threshold, then when calculating the value {x ij ,y ij ,z ij}, the value of the element {x ij ,y ij ,z ij}.

[0071] It is understandable that the threshold criterion may also be applied to the other situations mentioned above. For example, in calculating the value {x ij ,y ij ,z ij}, if the coordinate values ​​{x ij ,y ij ,z ij} weighting method, then for each group of coordinate values ​​{x ij ,y ij ,z ij}, a zero weight can be applied to it.

[0072] It is understood that when constructing the point cloud of the final three-dimensional image, it is not necessary to always construct the corresponding point cloud for each set of acquired image data separately. On the contrary, in some embodiments, using a suitable algorithm, the quality parameter value of a given pixel can be obtained by taking into account the actual signal received by the pixel during the exposure process, without actually constructing a point cloud based on the corresponding data. When generating the point cloud of the final three-dimensional image, a single set of merged image data can be generated by merging the sets of image data obtained from each exposure. In this way, the value of each pixel in the set of merged data can be based on the value of the pixel in each set of acquired image data, and there is a trend towards each set of image data having a higher associated quality parameter value for the pixel. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0074] Figure 1 Shown is an example of a traditional point cloud matrix;

[0075] Figure 2 Shown is a method of merging or combining multiple point cloud matrices into a single point cloud matrix;

[0076] Figure 3 Shown is an example of a point cloud matrix of an embodiment of the present application;

[0077] Figure 4 A flowchart of the steps of the implementation method described in this application;

[0078] Figure 5A is a schematic diagram of a structured illumination imaging system in one embodiment;

[0079] Figure 5B for Figure 5A A simplified diagram of the geometric structure of the structured illumination imaging system;

[0080] Figure 6 Shown is an example of how the standard deviation of GCPS values ​​varies with signal amplitude in an embodiment in which three-dimensional spatial information of an object is restored by combining Gray code technology and phase shift technology;

[0081] Figure 7 FIG. 5 is a schematic diagram showing a method of obtaining a plurality of input point cloud matrices using the imaging system of FIG. 5 and generating a single output point cloud matrix using the input point cloud matrices;

[0082] Figure 8 Shown is an example of a point cloud matrix of an embodiment of the present application;

[0083] Fig. 9Shown is a graph of the variation of depth noise at a point in a three-dimensional image with the contrast obtained in a series of images used to generate the three-dimensional image;

[0084] Fig.10 is a histogram of the number of well-exposed pixels in the image at different candidate exposure times;

[0085] Fig.11 is a histogram of the number of well-exposed pixels in the image at different candidate exposure times;

[0086] Fig.12 Shown is a series of images representing the number of well-exposed pixels obtained from the accumulated exposure of the image sensor;

[0087] Fig.13 Shown is pseudo code for executing an algorithm according to one embodiment. DETAILED DESCRIPTION

[0088] Figure 4 Flowchart of the steps performed in the embodiments described in the present application. In the first step S401, a plurality of sets of three-dimensional image data are collected by an imaging system. Each set of three-dimensional image data can be used to calculate a corresponding input point cloud of three-dimensional coordinates of different points on the surface of the imaged object and a surface intensity or brightness value at each point.

[0089] In step S402, a quality parameter value of the data associated with each point in each corresponding input point cloud is evaluated. As described above, the quality parameter value includes an uncertainty measurement result of the three-dimensional coordinates of each point. The quality parameter can be calculated as a function of the intensity capture value calculated for the spatial coordinates of each point in the corresponding input point cloud.

[0090] In step S403, a single set of output three-dimensional image data is calculated based on the image data contained in each set of acquired image data. Consistent with each set of acquired image data, the set of output image data determines the three-dimensional coordinate values ​​of different points on the surface of the imaged object and the surface intensity or brightness at each point. The three-dimensional coordinate values ​​are calculated by weighting the three-dimensional coordinate values ​​specified in the corresponding input point cloud according to the corresponding quality parameter values. In this way, the set of output image data can be used to present a three-dimensional image of the object (step S404).

[0091] Below, refer to Figures 5 to Figure 7 , describing an exemplary implementation of acquiring three-dimensional image data using a structured light illumination method.

[0092] refer to Figure 5A, which is a schematic diagram of a system suitable for capturing a three-dimensional image of an object 501 by structured light illumination. The system includes a projector 503 and a camera 505. The projector is used to project a spatially varying two-dimensional illumination pattern onto the object 40. The pattern itself includes a series of light and dark stripes 507, 509. The pattern can be generated, for example, by a spatial light modulator. The camera 505 is used to capture a two-dimensional image of the object when illuminated by the projector.

[0093] Figure 5B The geometric structure diagram of the system is shown in Figure 1. The camera and the projector are separated by a distance B. Point A on the object is at a distance D from the camera and forms an angle θ with the camera. c , and forms an angle θ with the projector p Due to the angle between the camera and the projector, any change in the topology of the object's surface will cause the light and dark stripe pattern detected by the camera to deform. It is not difficult to see that since the deformation of the pattern encodes information about the three-dimensional surface of the object, it can be used to deduce the surface topology of the object. The three-dimensional information can be recovered by sequentially capturing images of the object illuminated by different illumination patterns and comparing the measured intensity of each pixel between the obtained series of images.

[0094] In this embodiment, the three-dimensional information is obtained using phase shifting technology. Phase shifting is a well-known technique in which a series of sinusoidally modulated intensity patterns are projected onto an object, each pattern being phase shifted relative to the previous pattern. A two-dimensional image of the illuminated object is captured after each change in the intensity pattern. Changes in the topology of the object surface will cause the intensity pattern at different points on the surface measured by the camera to produce phase changes. By comparing the light intensity of the same pixel between a series of two-dimensional images, the phase of each point can be calculated, which in turn can be used to obtain the depth information of the object. The data is output as a two-dimensional array, each element of which is mapped to a corresponding pixel of the camera and determines the three-dimensional spatial coordinates of the point corresponding to the pixel.

[0095] It is understood that in addition to phase shifting, other techniques may be used to recover the three-dimensional spatial information. For example, in some embodiments, Gray code technology may be used, or Gray code and phase shifting technology may be used in combination. The precise algorithm used to decode the three-dimensional spatial information from the series of two-dimensional images varies depending on the specific illumination pattern and the way these patterns change between the series of images. For further information on algorithms for recovering depth information using the above and other techniques, see the paper "Structured Light Projection for Accurate Determination of Three-Dimensional Shapes" (O. Skotheim, F. Couweleers, 12th International Conference on Experimental Mechanics (ICEM12), August 29-September 2, 2004, Polytechnic University of Bari, Italy). In each of these techniques, the calculation of the three-dimensional spatial information of the object takes into account the fact that the intensity of each point of the object exposed to the light and dark areas of the illumination pattern changes as the pattern changes.

[0096] In one example of using Gray code and phase shifting to recover the three-dimensional spatial information, a projector projects a series of Gray code patterns and phase shift patterns onto an object (for further details of such methods, see, for example, Giovanna Sansoni, Matteo Carocci, Roberto Rodella, "Three-dimensional vision based on the combined use of Gray code and phase shift light protection: analysis and compensation of systematic errors" (Applied Optics, Vol. 38, pp. 6565-6573, 1999)). For each point on the object, the following two corresponding pixel positions can be defined: (1) projector pixel coordinates, i.e., the pixel position in the projector that emits the light incident on that point on the object; and (2) camera pixel coordinates, i.e., the pixel position in the camera that captures the light reflected from that point on the object. Using a suitable algorithm and taking into account the relative position of the camera and the projector (this relative position is directly determined by standard calibration measurements), the image captured by the camera can be processed to determine the corresponding projector pixel coordinates for each camera pixel. In fact, the projector pixels that are within the "field of view" of a particular camera pixel can be determined. Furthermore, by combining the image data received from the Gray code pattern and the phase shift pattern, the projector pixel coordinates may be determined at a higher resolution than the resolution of the projector pixels themselves.

[0097] The above scheme can be understood as follows: First, by selecting a number (N) of Gray code patterns and setting the number of sinusoidal fringes in the phase shift pattern to 2 N , so that each fringe is aligned with the binary transitions of the series of N Gray code patterns. The resulting Gray code words GC(i,j) and phase values ​​φ(i,j) can be combined to form a set of "GCPS" values ​​that describe the absolute fringe position at each field of view position. These GCPS values ​​can in turn be used to calculate the absolute fringe position by scaling the code min / max values ​​to the width of the projector image (w p) and height (h p ) to determine the projector pixel coordinates. In fact, by calculating the phase of the sinusoidal pattern for each camera pixel, the "fringe displacement" can be measured.

[0098] Then, we can define:

[0099]

[0100] Among them, GC v (i,j) is the Gray code measurement result, φ v (i, j) is the phase stepping measurement result, both measurements use vertical stripes (same as above, index symbols i, j represent pixel elements of the image sensor). From the above formula, it can be seen that the projector raw sub-pixel column for each camera image pixel can then be calculated:

[0101]

[0102] Among them, α max and α min are the maximum and minimum values ​​of the vertical stripe GCPS code. Similarly, when the GCPS value is obtained with horizontal stripes, it can be defined as:

[0103]

[0104] Then, according to the β(i,j) equation, the projector raw sub-pixel row for each camera image pixel can be calculated:

[0105]

[0106] Among them, β max and β min are the maximum and minimum values ​​of the horizontal stripe GCPS code.

[0107] After obtaining the projector sub-pixel column and projector sub-pixel row coordinates P c (i,j),P r (i, j), these values ​​can be used to obtain the {x, y, z} coordinates of the point on the object to be imaged. Specifically, for a given camera pixel p that is to receive light from a point on projector g, the coordinates {x, y, z} on the object can be derived using known triangulation methods similar to those used in stereo vision, taking into account conditions such as lens parameters and the distance between the camera and the projector. ij ,y ij ,z ij} is the estimated value E of the point position.

[0108] The uncertainty of each GCPS value is greatly affected by the amplitude of the recovered signal, but is less affected by the ambient light. Experiments performed by the inventors show that the uncertainty of the GCPS value is generally relatively constant before the amplitude of the received signal drops to a certain level, but after it drops to a certain level, it increases almost exponentially. This shows that the measured amplitude and ambient light can be converted into an expected GCPS value measurement uncertainty through a pre-established model. Figure 6 An example of this is shown, which shows the standard deviation (std) of the GCPS values ​​as a function of intensity. A new position estimate E' can be obtained by using the same calculation as above for obtaining the position estimate E, but with the projector pixel position fixed to g + Δg, where Δg is the deviation from the standard deviation of the GCPS value at the detected signal amplitude. Subsequently, the estimated standard deviation ΔE at the position measurement E can be derived by assuming ΔE = |(|E – E'|)|. The estimated ΔE can then be used to determine the quality parameter.

[0109] Obviously, no matter which structured illumination algorithm is used, in order to achieve high-precision calculation of three-dimensional spatial information, it is necessary to measure the intensity variation of each point on the object at the maximum signal-to-noise ratio, which in turn requires that the intensity contrast of the specific point when exposed to the alternating light and dark regions of the illumination pattern is as high as possible. In line with the above, the contrast of a specific point can be limited by the need to accommodate a large range of intensity variations over the entire surface of the object. If the surface of the object itself includes multiple light and dark regions, it may not be possible to maximize the signal recovered from the darker areas of the object surface without causing saturation effects when the brighter areas of the object are exposed to the brighter stripes of the illumination pattern, given the limited dynamic range of the camera. Therefore, it may not be possible to optimize the intensity contrast at every point on the surface with a single exposure (in this context, an "exposure" should be understood as a series of two-dimensional images that can be used to calculate the corresponding point cloud).

[0110] In order to solve the above problems, in the embodiment described in the present application, multiple exposure shots are performed with different settings. Figure 7As shown in the schematic diagram. First, a first series of two-dimensional images is captured with a first set of exposure settings. In this example, the exposure settings are changed by changing the relative size of the camera aperture, but it is understandable that the exposure settings can also be changed by adjusting one or more other parameters of the imaging system. Circle 701 represents the relative size of the camera aperture used to capture the first series of images, and patterns 703a, 703b, and 703c represent the illumination patterns projected onto the object when capturing the corresponding images in the first series of images. It can be seen that each illumination pattern 703a, 703b, and 703c includes a sinusoidal modulated intensity pattern that projects a series of alternating light and dark stripes onto the object. Directly below each pattern is a waveform schematic diagram representing the phase of the pattern in the field of view, wherein the dark stripes in the pattern correspond to the troughs and the bright stripes correspond to the peaks. It can be seen that the illumination patterns in the successive images are phase-shifted with each other, and the positions of the light and dark stripes in each pattern are shifted relative to each other. Among them, the depth information of the corresponding point is calculated by comparing the light intensity of the same pixel in the series of two-dimensional images with a suitable algorithm. Then, the three-dimensional space information and the intensity value of each point are stored in the first point cloud matrix 705 .

[0111] As mentioned above Figure 3 As shown, in addition to the three-dimensional coordinates {x, y, z} and the intensity value I, each element in the point cloud matrix 705 also includes a quality parameter value q. In this embodiment, the q value is determined based on the standard deviation of the intensity measured when the corresponding camera pixel In(i, j) is projected onto the object under three different illumination patterns 703a, 703b, 703c. In another embodiment, the q value is determined based on the difference between the maximum intensity value and the minimum intensity value of the pixel in a series of images. In this way, the quality parameter can measure the contrast of each pixel in the series of images.

[0112] In the next step of the method, as shown by circle 707, the exposure setting is adjusted by expanding the camera aperture size to increase the amount of light entering the camera from the object. Subsequently, a second series of two-dimensional images is captured, wherein, again, the illumination pattern 709a, 709b, 709c of each image is different from one another. The second series of two-dimensional images is used to calculate a second input three-dimensional image point cloud 711, which records the depth information of each element as well as the intensity value and the quality parameter value q. Since the exposure of the first series of images is different from the exposure of the second series of images, the contrast measured by a given pixel element {i, j} may be different in the two series of images. For example, the difference between the maximum intensity level and the minimum intensity level measured by a given pixel element {i, j} is different in the two sets of images. Therefore, the q value recorded by the first point cloud matrix 705 and the q value recorded by the second point cloud matrix 711 may be different from each other.

[0113] In yet another step, as indicated by circle 713, the exposure setting is further adjusted by enlarging the camera aperture size. Subsequently, a third series of two-dimensional images is captured by the camera while the illumination patterns 715a, 715b, 715c are projected onto the object. The third series of two-dimensional images is used to calculate a third input three-dimensional image point cloud matrix 717, which similarly records the depth information of each element as well as the intensity value and the quality parameter value q. Consistent with the first and second point cloud matrices, due to the different exposure amounts of the second and third series of images, the contrast detected by the same pixel element {i, j} in the second and third series of images may be different. Therefore, the q value recorded by the third point cloud matrix 717 may be neither the same as the q value of the first point cloud matrix 705 nor the same as the q value of the second point cloud matrix 711.

[0114] After calculating the point cloud matrix for each series of images, the method further obtains a single output point cloud matrix 719 by aggregating the data of each point cloud matrix, which can then be used to render a three-dimensional representation of the object.

[0115] It is understandable that although Figure 7 The example includes three series of input images, but this is for example only, and the output point cloud matrix can be obtained based on any number (N) of captured point cloud calculations, where N ≥ 2. Similarly, although the above embodiment changes the exposure by increasing the aperture size of the camera used to capture each series of images, it is easy to understand that this approach is not the only way to change the exposure. Other ways to change the exposure include, for example, increasing the illumination intensity, increasing the camera integration time, increasing the camera sensitivity or gain, or changing the intensity of the neutral gray filter set in the optical path between the object and the camera.

[0116] As mentioned above, although Figure 7 The example includes the step of generating a point cloud for each set of collected image data, but it is understood that this is for example only, and in some embodiments, the generation of the output point cloud does not need to construct a point cloud for each set of image data separately. In such a case, the associated quality parameter value of each pixel in each set of image data can be derived from the signal detection level of these pixels and the amount of ambient light incident on the image sensor.

[0117] By capturing multiple sets of image data at multiple different exposure settings and fusing the data in each set of data into a single output three-dimensional image, the embodiments described in the present application can help compensate for the limitations of the camera's dynamic range, thereby simultaneously increasing the signal-to-noise ratio of bright and dark points on the object surface. In this way, the embodiments described herein can help ensure that useful data is obtained from areas that are either omitted in the final image or covered by noise when using existing three-dimensional surface imaging methods, and can ensure higher object surface topography mapping accuracy than such existing three-dimensional surface imaging methods.

[0118] In the above embodiments, it is assumed that the camera sensor images in grayscale mode, that is, each pixel measures a single intensity value relative to the total light level incident on the sensor. However, it will be appreciated that the embodiments described herein are equally applicable to color imaging situations. For example, in some embodiments, the camera sensor may include an RGB sensor that decomposes incident light into three color channels of red, green, and blue using a Bayer Mask. In other embodiments, the camera sensor may include a three-CCD device that uses three separate CCD sensors to collect red, blue, and green light signals, respectively. In this case, the point cloud matrix may be obtained in the same manner as in the above embodiments, but within each matrix element, the intensity value I ij Decomposed into three color intensity values ​​r ij , g ij , b ij . Figure 8 An example of such a point cloud matrix is ​​shown.

[0119] As described above, the data used to present the final three-dimensional image can be obtained, for example, by changing one or more exposure settings including illumination intensity, camera integration time (exposure time), increasing camera sensitivity or gain, or changing the intensity of a neutral gray filter placed in the optical path between the object and the camera. It can be seen that for any exposure, there are a large number of different settings combinations available. Some of these combinations can achieve better solutions than other combinations. For example, in some cases, it may be necessary to select a set of exposure settings that can minimize the exposure time, while in other cases, there may be additional or different considerations, such as the need to keep the aperture size unchanged to avoid changes in the depth of field, which imposes other constraints on the parameters that need to be adjusted by the imaging system and the amount of adjustment of such parameters. In general, when acquiring images, it is generally necessary to strike a balance between achieving an acceptable image SNR (especially the depth value noise at each point on the surface of the imaged object) and one or more exposure requirements, such as (1) total acquisition duration, (2) illumination intensity required for acquisition, (3) aperture size, etc.

[0120] It should be noted that 3D measurement systems using active illumination (e.g. structured light, laser triangulation, time-of-flight) differ from conventional cameras in that the amount of ambient light can significantly affect the system's dynamic range when it is desired to achieve maximum noise levels. This means that traditional automatic exposure algorithms cannot be used directly - such algorithms are generally optimized only to achieve a sufficiently large total signal level, while active 3D measurement systems must avoid saturation while enforcing a corrected ratio of active emission to ambient light. At the same time, manually determining a good exposure setting is a complex task, as it requires the user to fully understand the camera model and its parameter settings.

[0121] Therefore, it is necessary to provide a means for determining which exposure settings to adjust and to what extent to optimize data quality for a given scene or object. Figure 5B As described above, for any given pixel, the improvement in noise begins to fade when the contrast rises to a certain point. The embodiments described in the present application can use this phenomenon to determine a suitable set of exposure settings for one or more imaging constraints (such as maximum total exposure time, maximum aperture size, etc.).

[0122] More specifically, a target to be achieved can be specified in terms of a quality parameter value for pixels in the final 3D image, where the target needs to be achieved under one or more imaging constraints. The goal of this approach is to try to optimize the final 3D image in terms of noise while imposing one or more constraints ("costs") such as a maximum total exposure time or total illumination power.

[0123] First, we can define an exposure cost function E with one or more exposure settings of the system as variables: Cost , where the exposure setting is the setting that determines the amount of light incident on the camera:

[0124] E Cost =f1(exposure time)+f2(aperture size)+f3(neutral gray filter strength)+f4(lighting intensity)+…

[0125] Among them, the function {f1,f2,…f n} determines how the exposure cost changes with each parameter respectively. These functions can be defined by the user and effectively determine the "side effect" of the user adjusting each parameter. For example, if a 3D image needs to be captured in a very short time, the user can assign a higher cost to the exposure time. For another example, if the user has no time constraints but wants to keep the overall power consumption to a minimum, he or she can assign a higher cost to the lighting intensity. Cost The values ​​can be used to provide constraints when determining the optimal exposure setting for a given acquisition operation.

[0126] In addition, the exposure value E can be defined value , which represents the amount of light reaching the sensor, and includes both ambient light and light from the sensor system itself:

[0127] E value = e1 (exposure time) · e2 (aperture size) · e3 (neutral gray filter strength) · e4 (lighting intensity) · …

[0128] E value This is used to incorporate the various effects that affect the amount of signal received by the camera and therefore the quality of each individual pixel within the system. value It can also be extended to return two exposure values: ambient light (E ambient )Exposure value; Projection light (E amplitude )Exposure value.

[0129] In addition, the function {e1,e2,…e n}, where the relationship between each pair of functions determines the degree to which adjusting one parameter changes the amount of incident light to the camera relative to adjusting the other parameter. For example, doubling the exposure time is equivalent to doubling the aperture size in terms of the effect of increasing the amount of incident light to the camera. For another example, doubling the aperture size may be equivalent to reducing the neutral gray filter strength by a factor of four in terms of the effect of increasing the amount of incident light to the camera. The functions {e1, e2, …e n} can be defined to take such relationships into account.

[0130] Function {e1, e2, …e n} and the relationship between them can be determined empirically offline by experimentation. The usefulness of knowing the relationship between such functions is that changes in the value of one parameter can be translated into changes in the values ​​of the other parameters. For example, when seeking to achieve a particular E value value, if it is known that the E value The exposure time change of the value can be converted into the aperture size change. The effect of these two changes on E value The advantage of this approach, as will be seen below, is that in the acquisition operation, only one parameter (usually exposure time) needs to be of interest, making it easier to determine the exposure setting. The user can then convert the exposure time change into other parameter values ​​according to their specific requirements (e.g., if they want to minimize the total exposure time, or conversely, if they want to minimize the aperture size, etc.).

[0131] In the following, two methods based on E are given. value , E Cost and q minExample of a method for determining exposure settings using the value q min is the minimum acceptable quality parameter value.

[0132] The goal of the first example is to determine the maximum value of each image that can be enlarged so that the final image satisfies the quality parameter value q(p)>q min The total exposure cost of a series of exposure operations is less than the predetermined maximum cost. That is, for a series of exposure operations consisting of n exposures, each exposure i is calculated at a corresponding exposure cost of The goal of this example is to capture images with a set of exposure settings. Under the condition of min A series of exposure operations that can maximize the total number of pixels that satisfy q(p)>q min The number of pixels and A series of exposure operations with a ratio of .

[0133] The goal of the second example is to determine a set of exposure settings for each image that can minimize the total exposure cost in the series of exposure operations, while at the same time ensuring that the quality parameter value q(p)>q is satisfied in the final three-dimensional image. min The number of pixels reaches the threshold.

[0134] In the following, different strategies that can meet the above objectives are discussed. In each strategy, the above objectives can be limited to a specific two-dimensional or three-dimensional region of a given scene.

[0135] For many 3D measurement systems, there is a general relationship between the received contrast / signal c and the noise ( This relationship. Contrast represents the amplitude / signal level of active illumination used by the camera system. In addition, ambient light also affects the amount of light incident on the camera. Contrast is generally related to the amount of light collected. For example, doubling the exposure time will likely double the contrast (and ambient light). Reference Fig. 9 It can be seen that when the contrast exceeds a certain value (such as 50), there is an approximate "plateau" section, in which the measurement noise no longer improves significantly with the increase of contrast. In addition, in many applications, there is a fixed limit of tolerable noise level (such as 0.15mm). Fig. 9 In the exposure case shown, for example, an exposure with a contrast ratio greater than 40 is considered to be "sufficient". In addition, it can be seen that the minimum tolerable contrast ratio also depends on the amount of ambient light.

[0136] Reference again Figure 5B As shown, the measurement point σ of the structured light system SLThe distance uncertainty can be calculated according to the following formula:

[0137]

[0138] Where D is the distance from the measurement point, B is the baseline between the camera and the projector, and θ c and θ p are the camera angle and projector angle respectively. FOV is the projector field of view, φ max is the number of projected sine waves, A is the amplitude of the observed sine wave, and C is the received fixed light signal, i.e., the DC level of the ambient lighting + the emitted sine wave (Bouquet, G. et al., “Design tools for 3D cameras based on TOF and SL”, Optics Express, Vol. 25, pp. 27758-27769, 2017, the content of which is cited herein).

[0139] However, at the same time, there is also the problem of sensor saturation. max Defined as the maximum signal level that the sensor can withstand without saturation, then A+C>S max , and σ SL The degradation rate is faster. Among them, when the sine wave can only be partially recovered, the quality will be reduced, and the quality will be further reduced when only the Gray code part of the code can be decoded. Once the sensor is saturated, all information is lost. It can be understood that when capturing data at multiple exposure settings, it is not necessary to always capture Gray code and phase images / sine waves at the same time. Since Gray code is more tolerant to saturation, Gray code can be captured at one of the exposure settings and used in combination with phase images captured at multiple exposure settings. This can avoid multiple repetitions of Gray code capture, saving time.

[0140] To take into account the possibility of saturation, the above equation can be rewritten as:

[0141]

[0142] It should be noted that in most cases, B and FOV can be considered constants.

[0143] As mentioned above, the exposure cost E of a specific image acquisition operation is Cost is expressed as a function of different parameters that have an impact on the amount of incident light on the camera. For simplicity, in the following, it is assumed that all other parameters except the exposure time remain unchanged. Thus, E Cost =Exposure time length t.

[0144] Accordingly, the following equation can be obtained:

[0145] A′(p)=tA *(p)

[0146] C′(p)=tC * (p)

[0147] Where A′(p) represents the amplitude measurement value at point p in the image, C′(p) represents the ambient light measurement value at point p, and A * (p) is the amplitude at point p that is independent of exposure time, C * (p) is the ambient light at point p that is independent of exposure time.

[0148] A * and C * It can be considered as a normalized value normalized to some time unit. * and C * It can be expressed as the following formula:

[0149] A * (p) = v + (p)-v - (p)

[0150] C * (p) = v - (p)

[0151] Among them, for each pixel p, v + (p) is the pixel signal level when the system starts active illumination (i.e., when the structured light projector is turned on), v - (p) is the pixel signal level when the system illumination is turned off.

[0152] In this way, the noise prediction value of the system is expressed as:

[0153]

[0154] When considering the case of each pixel p, we can get:

[0155]

[0156] σ SL The expression can be simplified to:

[0157]

[0158] To establish exposure time values, quality parameters can be defined Thus, the required q>q min It can be rewritten as SL <σ max This expression represents the shortest time t required for the corresponding pixel to be fully exposed at each point on the object. min It can be seen that the required t minDepends not only on the ambient light, but also on the distance D from the object and the angle θ c and θ p The position within the scene determined thereby. Depending on the specific implementation, such variables may represent constants or be used to reflect actual per-pixel data.

[0159] Given the over-saturation (where σ SL >>σ max ), there is also a time value t max , which determines the maximum time for which a pixel can be exposed without saturation. In a practical scenario, this means that for some pixels, due to σ max >σ SL the required signal levels A and C will cause the sensor to saturate, and thus σ max >σ SL cannot be satisfied. This situation can occur, for example, when the sensor system is used outdoors in sunlight or in the presence of strong light sources. In such cases, the built-in illumination will not be able to overpower the ambient light.

[0160] The t min and t max values for each pixel can be determined in various ways. In a first example, these two values are predicted from the captured image as follows.

[0161] Assume that the image I(t0) captured with exposure time t0 contains a pixel p0 with v + (p0) and v - (p0), and assume that p0 is neither fully over-saturated nor overly under-saturated. Then, the t min and t max for this pixel can be predicted using the following equations:

[0162]

[0163] At the same time, since

[0164]

[0165] we get

[0166]

[0167] If the captured pixel is over-saturated, no other prediction can be obtained except t max <t0. If the captured pixel is overly under-saturated (e.g., the contrast is close to zero), no other prediction can be obtained except t min >t0. However, for the case of overly under-saturated pixels, there are also various (where σ SL (p0)>σ maxAt the same time) allow t min and t max The situation for making predictions.

[0168] In addition, it is interesting to note that there is also t min >t max In real-world situations, this usually means that the system's active illumination is too weak to overcome the ambient illumination and achieve a sufficiently high image quality. This may only be the case for part of the scene. For example, the surface normal of the imaged object and its specular properties may be unfavorable to the acquisition system.

[0169] In the first example, different exposure settings i Init ={E1,E2,…E n} Capture a set of candidate images I init ={I1,I2,…I n It should be noted that since only the exposure time is considered in this case, E init ={t1,t2,…t n Then, determine which part of the images (i.e., which part of the candidate exposure time) satisfies the total time less than T max This constraint (here, it can be understood that in this case, due to E Cost It is expressed only in terms of exposure time, so T max The value actually corresponds to ) while producing the best results in maximizing the number of pixels with an acceptable noise level in the final 3D image.

[0170] In this example, in order to quickly evaluate the candidate exposure image I Init To obtain the number of pixels with good exposure results at an exposure time of , the following histogram-based method can be used. Init Given one or more images in , calculate t for each pixel min and t max Assumption I Init Cover the entire dynamic range of the scene (usually about 7 stops), so that t can be calculated for most pixels min and t max For each pixel p, there may be a good t min and t max Several images of estimated values These values ​​are obtained from Select some I in the calculation method, or calculate it by Internal min and t maxAfter that, create a two-dimensional histogram H. The histogram represents [xΔ t ,(x+1)Δ t ] is a k×k matrix of exposure times within this range, where:

[0171]

[0172] T * is the maximum time bar considered.

[0173] In addition, each H(x,y) value is defined as min Located in row x and t max The number of pixels in column y:

[0174] H(x,y)={p i |t min (p)∈[xΔ t ,(x+1)Δ t ], t max (p)∈[yΔ t ,(y+1)Δ t ]

[0175] Through this histogram, we can estimate the exposure time t∈[0,T * ]Number of pixels with good exposure results during exposure good :

[0176]

[0177] Among them, idx(t) is the number of rows / columns corresponding to t:

[0178]

[0179] It should be noted that this formula is applicable to [0,T * ], not limited to I init The exposure time in .

[0180] Fig.10 An example of such a histogram is shown in . The exposure time is in milliseconds, and Δ t =5ms. The x-axis represents t min ∈(0,500), the y-axis represents t max ∈(0,500). The region labeled R1 contains the values ​​of H that should be added to obtain N given t′=105 ms (represented by the points on the diagonal). good That is, region R1 contains t min (p)∈(0,t′) and tmax (p)∈(t′,T * ) of all pixels p.

[0181] For a given set of exposure times E′={t0, t1, …} in ascending order, we can calculate:

[0182]

[0183] The formula is shown in Fig.11 Since rectangles R1 and R2 overlap, the above formula avoids counting the histogram bins twice.

[0184] In order to further speed up the calculation process, the cumulative histogram H can also be used c :

[0185]

[0186] Among them, the following formula can be used to quickly calculate the N of the entire set of exposures. good (i.e., calculating the cumulative sum of pixels with an acceptable noise level in at least one captured image):

[0187]

[0188] This method can quickly calculate N with only the sum of four values. good (t).

[0189] Using the above N good The calculation method can determine a set of exposure times that meets the first goal mentioned above, that is, when the total exposure time of the corresponding image is less than T max Under the condition of min A set of exposure times for the number of pixels.

[0190] In one embodiment, the following "greedy" algorithm may be used:

[0191] set up:

[0192] While E opt Total time <T max ,but:

[0193] For the set of exposures E opt ∪E, find the maximum increase that satisfies q(p)>q min The number of pixels E∈E init (This can be done quickly using the histogram method described above);

[0194] Add E to E opt , and from Einit Remove E from it.

[0195] Fig.12 The results after each round of calculation of the above algorithm are shown, where each row represents one round of the above while loop. After each step, G is calculated, where G is E opt a binary image in which all pixels with good exposure results in the contained exposure images are true. Since initially no pixels are known to have an acceptable q value, the initial image of G (image 1) is all black. Subsequently, the first exposure image of the scene with the most pixels having good exposure results (q > q min ) is selected (see image 2), so that G is updated (image 3). In the next round, based on the updated image G, and a new E is selected as the image with the most other pixels having good exposure results, so that G is further updated.

[0196] When the cost is defined as a function of multiple imaging parameters (not just exposure time), the above algorithm runs until the total cost of the selected set of exposures exceeds E costmax , rather than until the total exposure time exceeds T max . More specifically, after determining a set of candidate exposure settings, the E value value of this set of exposure settings can be determined, and this value can be used to convert / generate other set(s) of candidate exposure settings with the same E value value. This can be achieved by taking into account the above function {e1, e2, … e n}. For example, a set of candidate exposure settings with an exposure time of t1 and an aperture size of a1 can be identified, and then converted to a second set of exposure settings with an exposure time of t2 and an aperture size of a2, where t2 > t1 and a2 < a1. These two mutually replaceable sets of exposure settings should have a similar effect on the amount of light collected from the bright and dark regions of the object being imaged. However, depending on whether the user pays more attention to minimizing the total exposure time as much as possible or minimizing the aperture size as much as possible (i.e., depending on the function f1 or f2 in E Cost ), these two mutually replaceable sets of exposure settings can have very different costs.

[0197] The above greedy algorithm can be very easily modified to meet the requirements of the above second objective, that is, to determine a set of exposure settings for each image that maximally reduces the total exposure cost of the corresponding series of exposure operations while ensuring that the number of pixels satisfying the quality parameter value q(p) > q min reaches the threshold in the finally obtained three-dimensional image.

[0198] Let

[0199] While Eopt In the case where q(p)>q min The total number of pixels is less than N min :

[0200] For the set of exposures E opt ∪E, find the maximum increase that satisfies q(p)>q min The number of pixels E∈E init (This can be done quickly using the histogram method described above);

[0201] Add E to E opt , and from E init Eliminate E.

[0202] It is understood that when selecting a set of optimal exposure times, it is not necessary to limit the selection to only the initial series of captured images I. init The above histogram representation allows the use of the exposure time corresponding to each time bin in the histogram, rather than just the exposure time t i ∈E init For a given H(x,y), a set of optimal exposure times can be quickly calculated as follows (here, we return to the time when the total exposure time of each image is less than T max Under the condition of maximum increase, the final image satisfies the quality parameter value q(p)>q min The first target of the number of pixels).

[0203] Below, assume is the discrete exposure time corresponding to the row / column of the histogram H, and let To achieve pixel-free satisfaction The maximum time of exposure (the lower limit of acceptable exposure time). Similarly, To achieve pixel-free satisfaction The shortest exposure time (the upper limit of acceptable exposure time).

[0204] In addition, there are is a corresponding set of acceptable discrete exposure times, where The sorting order of each time is ascending.

[0205] To determine a set of optimal exposure times, the following process is performed:

[0206] -Identification The total time is less than T max All possible subsets of This can be achieved by: Here, since it is meaningless when the number of aperture stops between exposures is less than, for example, 1 / 3 stop, it is assumed that n is relatively small.max All subsets of All time Map to an integer k, where k = 1, 2, ..., n. Then, for each integer k, list all possible different segmentation results, such as:

[0207] 1={1}

[0208] 2={2}

[0209] 3={3},{1+2}

[0210] 4={4},{3+1}

[0211] 5 = {5}, {4+1}, {3+2}

[0212] 6 = {6}, {5+1}, {4+2}, {3+2+1}

[0213] When using the cost-based approach, The total cost is less than All possible subsets of Identification is the basis of the algorithm.

[0214] For every integer k, the above partitioning produces The total time of a group is A subset of . In this way, the total time ≤ T max All subsets of <k max The result of integer segmentation, where k max The closest to T max of The segmentation result can be generated either from the bottom up or recursively. For example, in the case of Python code, the subsets can be generated recursively by the following series of commands:

[0215]

[0216] The bottom-up method simply uses dynamic programming. In addition, the result table of all such segmentations up to a certain larger k value can be calculated in advance.

[0217] -After obtaining each subset After that, we can calculate And select the one with the largest N good Subset of values It should be noted that the N of each subset can be quickly calculated by the above cumulative histogram method. good value.

[0218] The context of the examples described above is that a series of initial images are captured first and then the optimal exposure settings for the image acquisition are determined. Therefore, in the above examples, each set of candidate exposure settings is determined offline, so that each set of exposure settings to be used in the three-dimensional image acquisition process is determined before the corresponding set of image data is acquired. In the following, a method for selecting the optimal exposure settings online will be described. Therein, it is assumed that one or more initial exposures have been performed and the purpose of the method is to predict the subsequent optimal exposure. Therefore, in this online method, the three-dimensional image acquisition process is performed in an iterative manner, wherein each iteration determines a set of optimal exposure settings to be used next time, and then a set of image data is captured with this set of exposure settings. In each iteration of the method, one or more new sets of candidate exposure settings are considered and a single set of candidate exposure settings is selected from them for use in acquiring the next set of image data.

[0219] Consider an image I(t) taken with an exposure time t. For each pixel p i ∈I(t), the pixel signal level when the system starts active illumination (structured light projector is turned on) is recorded as v + (p i ), and the pixel signal level when the system turns off the lighting is recorded as v - (p i ).

[0220] The constraints are as follows (the numbers given below are based on the camera sensor being an 8-bit sensor with 256 intensity levels):

[0221] -v bad Maximum number of "black" underexposed pixels (e.g. 10). <v bad There are pixels whose value cannot be guaranteed to increase by increasing the exposure time. Such pixels include, for example, pixels in shadow areas.

[0222] -v min : The minimum acceptable pixel value (such as 50). v∈[v bad ,v min ] pixels are able to increase their value as the exposure time increases.

[0223] -v max : Maximum acceptable pixel value (e.g. 230). v>v max The pixels are overexposed pixels and their appropriate exposure time cannot be calculated.

[0224] -σ max : Maximum measurement uncertainty that can achieve acceptable 3D quality. The σ for each pixel is calculated as follows:

[0225]

[0226] Given p with σ(p) and time t, σ′(p) can be estimated as follows from t′=αt:

[0227]

[0228] The algorithm is as follows:

[0229] -Acquire images with initial exposure time

[0230] - Calculate a set of candidate exposure times that may be used for the next exposure

[0231] -For each candidate exposure time, calculate the expected number of well-exposed pixels that can be obtained

[0232] -Choose the exposure time with the maximum expected number of well-exposed pixels

[0233] - Repeat the above process until the condition is met (for example, the maximum allowed time is reached)

[0234] Given a set of completed exposures, the probability model can be used to calculate the number of well-exposed pixels. Consider an image I(t) taken with exposure time t, where each pixel p i ∈I(t) can belong to one of the following categories:

[0235] - Case 1 (cannot be improved): the pixel is correctly exposed, but σ(p) is too high, i.e.:

[0236] v - (p)∈(v min ,v max ),v + (p)∈(v min ,v max ),σ(p)>σ max

[0237] Since it is not possible to obtain suitable measurement results for pixels of this category by increasing the exposure time, such pixels are excluded from consideration.

[0238] - Case 2 (acceptable quality): The pixel is correctly exposed and the quality of the 3D measurement result at this point is acceptable, i.e.:

[0239] v + (p)∈(v min ,v max ),σ(p)<σ max

[0240] Such pixels can be excluded from further consideration.

[0241] - Scenario 3 (overexposure): This scenario can actually be divided into the following two possibilities:

[0242] (a) Even when the projector is turned off, the pixel is still overexposed, i.e., v - (p)>v max In this case, there is no other measure except to reduce the ambient light. Although it is possible to increase α′ for each exposure level 2α The probability of , but the upper limit is α′ 2α .

[0243] Pr(σ(p)<σ max |t>t i )=0,

[0244]

[0245] (b) The pixel is overexposed when the projector is on, but correctly exposed when the projector is off, i.e., v + (p)>v max And v - (p) <v min In this case, the goal should be to reduce the exposure time. Same as above:

[0246] Pr(σ(p)<σ max |t>t i )=0

[0247] Assume that when p is correctly exposed, p has a certain probability α 2b Satisfy|v + (p)-v - (p)|=R, then it means that if:

[0248]

[0249] Then v + (p) = v max The probability is α 2b .

[0250] So, we can get:

[0251]

[0252]

[0253] Then, it is estimated that while keeping σ(p)<σ max At the same time, t' can be reduced. Let t" = β"t', then:

[0254] if Then σ″(p)>σ max

[0255] So, we get:

[0256] Pr(σ(p)<σ max |t∈(β″t i ,β′t i ))=α 2b ,

[0257] Pr(σ(p)<σ max |t<β″t i )=0

[0258] - Situation 4 (underexposure): There are three possibilities for this situation:

[0259] (a) The pixel is still underexposed with the projector turned on, i.e., v + (p) <v bad In this case, the exposure time needs to be increased. Similarly, although the α of each exposure level can be increased 3α The probability of , but the upper limit is α′ 3α .

[0260] Pr(σ(p)<σ max |t <t i )=0,

[0261]

[0262]

[0263] (b) The pixel is still underexposed with the projector on, i.e.: σ(p)>σ max , but v + (p)∈(v bad ,v min ). You can maintain v + (p) <v max At the same time, increase the exposure time to v + (p), so that tσ(p)<σ max . The probability is estimated as follows:

[0264] Pr(σ(p)<σ max |t <t i )=0,

[0265]

[0266]

[0267]

[0268] (c) The pixel is underexposed with the projector off, i.e.: v - (p) <v min ,v + (p)∈(vmin ,v max ),σ(p)>σ max . This situation is similar to the above situation 4(b).

[0269] Fig.13 The overall framework of the above online method is given, which contains examples of possible algorithms for selecting candidate times and estimating the expected number of pixels with acceptable quality parameter values. The algorithm uses a set of current exposure times E = {(t i ,I(t i )} and the corresponding image, and use ComputeExpectedNum to estimate that the previous exposures E at a given exposure time t meet σ(p)<σ max The algorithm uses the above probability model and can estimate α by machine learning techniques. i The method estimates the probability of each pixel falling into one of the above categories:

[0270] - Given a value with I(t i ) of a certain t i

[0271] - Calculate the candidate time in a logarithmic coordinate system, each t i 1 / 6 step length, 1 step length <t min and>t max

[0272] -For each candidate time, calculate the expected number of well-exposed pixels (using the method described above)

[0273] -Choose the time for the maximum expected number of well-exposed pixels.

[0274] As described above, when the cost is defined as a function of multiple imaging parameters (rather than just exposure time), the algorithm runs until the total cost for a set of exposures exceeds Until the total exposure time exceeds T max As another alternative, the algorithm terminates when the ratio of the expected increase in the number of new well-exposed pixels (the return value of ComputeExpectedNum) to the exposure cost of the selected exposure falls below a threshold.

[0275] It is understandable that in the above example, the exposure setting that changes in each image acquisition operation is limited to the exposure time, and other parameters (aperture size, illumination intensity, neutral gray filter intensity, etc.) remain unchanged between image acquisition operations. However, as mentioned above, based on the change in exposure time, it is possible to infer the extent to which other parameters need to change in order to achieve the same signal-to-noise ratio. This purpose can be achieved by considering the correlation function {e, e2, …, e n For example, after determining a set of exposure times {t1, t2, …, t n}, we can minimize the corresponding cost {f1,f2,…,f n}, while considering the corresponding correlation functions {e1,e2,…,e n}, the set of exposure times is converted into a set of aperture sizes {a1, a2, ..., a n} or a combination of these parameters {{t1,a1},{t2,a2}…,{t n ,a n Therefore, although the focus of the algorithm described in this application is on exposure time, the determination results of different exposure times can be used as an intermediary variable to determine other settings that have an impact on the amount of incident light on the camera.

[0276] When adjusting the projector brightness, it mainly affects the amplitude of the received signal, but has a smaller impact on the ambient light intensity. This can be easily incorporated into the above "greedy" algorithm by simply including the images captured at different projector brightness in the above group (e.g., a set of different discrete brightness values).

[0277] Furthermore, it will be appreciated that, although the specific examples described herein relate to structured illumination systems, the methods described herein can be easily extended to other forms of 3D imaging by taking into account how the signal-to-noise ratio of the final image varies with the received light signal. For example, for an active time-of-flight system, the depth noise σ TOF The following relationship exists:

[0278]

[0279] Among them, N ph is the total received signal level (the sum of amplitude A and ambient light C), τ response is the time response of the system, c is the speed of light, and m is the number of sampling times implemented. When considering the influence of dark noise and environmental noise, is replaced by the signal-to-noise ratio (SNR). In this case, τ response Typically a constant (depending on component characteristics).

[0280] It will be appreciated that the above algorithm can be easily restricted to act only on the valuable region of the image. The region can be a specified two-dimensional (specified in pixel coordinates) or three-dimensional (specified in XYZ world coordinates) region. Subsequently, pixels determined to be outside the specified two-dimensional or three-dimensional target region can be excluded from further consideration by the algorithm.

[0281] In summary, the embodiments described herein provide a means for presenting a high SNR three-dimensional image of a scene or object. Large variations in the amount of light at different points in a scene can be accommodated by performing multiple image acquisitions at different exposure settings, and by merging the data from multiple sets of image data into a single point cloud that maximizes the signal-to-noise ratio at each point. In addition, embodiments of the present application provide a means for determining a set of exposure settings for acquiring each image in a manner that maximizes the signal-to-noise ratio of the final three-dimensional image while satisfying one or more constraints such as time, depth of focus, and illumination power.

[0282] It is understood that the implementation form of the technical solution and the operations described in this specification can be implemented in digital electronic circuits or computer software, firmware or hardware including the structure disclosed in this specification and its equivalent structure, or in a combination of one or more thereof. The implementation form of the technical solution described in this specification can be implemented by one or more computer programs, that is, by one or more computer program instruction modules encoded in a computer storage medium and executed by a data processing device or used to control the operation of a data processing device. As an alternative or additional solution, the program instructions can be encoded in an artificially generated propagable signal, such as a machine-generated electrical, optical or electromagnetic signal, and encode information for transmission to a suitable receiving device for execution by a data processing device. The computer storage medium can be a computer-readable storage device, a computer-readable storage matrix, a random or serial access storage array or device, or a combination of one or more thereof, or can be included therein. In addition, although the computer storage medium itself is not a propagable signal, the computer storage medium can be used as the origin or destination of the computer program instructions encoded in the artificially generated propagable signal. Additionally, computer storage media may be one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices), or may be contained therein.

[0283] Although certain embodiments have been described above, these embodiments are given by way of example only and are not intended to limit the scope of the invention. On the contrary, the novel methods, novel devices, and novel systems described herein may be implemented in a variety of forms, and various deletions, substitutions, or changes may be made to the methods and systems described herein without departing from the spirit of the invention. The appended claims and their equivalents are intended to cover such forms or modifications that fall within the spirit and scope of the invention.

Claims

1. A method for determining one or more sets of exposure settings for use in a three-dimensional high dynamic range image acquisition process performed by an imaging system, characterized in that: The imaging system comprises an image sensor, the 3D high dynamic range image acquisition process comprises the image sensor acquiring one or more sets of image data with respective sets of exposure settings, wherein the one or more sets of image data can be used to generate one or more 3D point clouds, the one or more 3D point clouds defining the 3D coordinates of one or more points on the surface of an imaged object, each set of exposure settings specifying the value of one or more parameters of the imaging system that affect the amount of signal reaching the image sensor, the method comprising: (1) identifying one or more sets of candidate exposure settings using image data captured by the image sensor; (2) For each set of candidate exposure settings: determining an amount of signal that different pixels of the image sensor may receive when the set of candidate exposure settings is used to capture a set of image data used by the three-dimensional high dynamic range image acquisition process; Determining, based on the amount of signals that the different pixels may receive, whether each pixel will be a well-exposed pixel when the set of candidate exposure settings is used, wherein a well-exposed pixel is a pixel whose associated quality parameter value is greater than a threshold, the quality parameter value of the pixel reflecting a degree of uncertainty in a three-dimensional coordinate of a point in the point cloud associated with the pixel when the set of image data captured with the set of candidate exposure settings is used to generate a point cloud associated with the pixel; determining an exposure cost, wherein the exposure cost is derived from values ​​of the one or more parameters of the set of candidate exposure settings; and (3) selecting one or more sets of exposure settings for the three-dimensional high dynamic range image acquisition process from the one or more sets of candidate exposure settings, wherein the selection is used to satisfy one or more optimization criteria, wherein the one or more optimization criteria are criteria defined in the following aspects: (a) the number of pixels in a group N, wherein a pixel belongs to the group N if the pixel can be determined as a well-exposed pixel under at least one selected set of exposure settings; and (b) Exposure cost of the selected one or more sets of exposure settings.

2. The method according to claim 1, characterized in that For each set of candidate exposure settings, the method comprises: identifying one or more sets of alternative candidate exposure settings for which the one or more parameters of the imaging system have different values ​​but the same expected amount of received signal from the image sensor; and For each set of alternative candidate exposure settings, determining an exposure cost, wherein the exposure cost is derived from the values ​​of the one or more parameters of the set of alternative candidate exposure settings, The one or more sets of alternative candidate exposure settings are selectable for use in the three-dimensional high dynamic range image acquisition process.

3. The method according to claim 1 or 2, characterized in that: The one or more sets of candidate exposure settings are selected to ensure that a ratio of the number of pixels in the set N to the exposure cost of the selected one or more sets of exposure settings meets a criterion.

4. The method according to claim 1 or 2, characterized in that: The one or more sets of candidate exposure settings are selected to ensure: (a) the number of pixels in the group N meets the first criterion; and (b) The exposure costs of the selected one or more sets of exposure settings meet a second criterion.

5. The method according to claim 4, characterized in that The first criterion is to maximize the number of pixels belonging to the group N.

6. The method according to claim 4, characterized in that The first criterion is to ensure that the number of pixels belonging to the group N is greater than a threshold.

7. The method according to claim 4, characterized in that The second criterion is to minimize the sum of the exposure costs.

8. The method according to claim 4, characterized in that The second criterion is to ensure that the sum of the exposure costs of each set of exposure settings selected is less than a threshold.

9. The method according to claim 1, characterized in that: For one or more of the sets of exposure settings, determining the amount of signal that different pixels of the image sensor may receive when the set of candidate exposure settings is used to capture a set of image data includes capturing a set of image data with the set of candidate exposure settings.

10. The method according to claim 9, characterized in that One or more sets of other candidate exposure settings are determined using each set of image data captured with the one or more sets of candidate exposure settings.

11. The method according to claim 4, characterized in that Steps (1) to (3) are repeated through one or more iterations, wherein for each iteration: selecting a set of exposure settings from the one or more sets of candidate exposure settings determined in the iteration; and The first criterion is to maximize the number of pixels in the group N, and the second criterion is that the sum of the exposure cost of the group of exposure settings selected in the current iteration and the corresponding exposure costs of each group of exposure settings selected in all previous iterations is less than a threshold.

12. The method according to claim 11, characterized in that For each iteration, capturing a set of imaging data with the imaging system using the selected set of exposure settings, For each iteration after the second iteration, sets of candidate exposure settings for the current iteration are determined using a set of image data captured in the previous iteration.

13. The method according to claim 12, characterized in that The step of determining whether each pixel is a well-exposed pixel at a set of candidate exposure settings includes determining a probability that each pixel is well-exposed, the probability being determined based on the amount of signal received by the corresponding pixel in a previous iteration of the method.

14. The method according to claim 1, characterized in that The exposure cost of each set of exposure settings is a function of the exposure time used by the set of settings.

15. The method according to claim 14, characterized in that The step of determining one or more sets of candidate exposure settings includes: for one or more pixels of the image sensor, determining an exposure time range that makes the pixel likely to be a well-exposed pixel.

16. The method according to claim 1, characterized in that The quality parameter value associated with a pixel is determined based on the amount of ambient light of the imaged scene.

17. The method according to claim 1, characterized in that Each set of exposure settings includes one or more of the following: exposure time of the image sensor; the size of an aperture in the optical path between the object and the sensor; The intensity of light used to illuminate the object; and The strength of a neutral density filter placed in the light path between the object and the sensor.

18. The method according to claim 1, characterized in that The imaging system is an optical imaging system including one or more light sensors.

19. The method according to claim 18, characterized in that The imaging system includes one or more light sources for illuminating the imaged object.

20. The method according to claim 1, characterized in that The image data in each set of image data includes one or more two-dimensional images of the object captured by the sensor.

21. The method according to claim 1, characterized in that The imaging system is an imaging system that obtains each set of image data through structured illumination.

22. The method according to claim 21, characterized in that Each set of image data includes a series of two-dimensional images of the object acquired by the light sensor, each two-dimensional image in the series of two-dimensional images being acquired with a different illumination pattern.

23. The method according to claim 22, characterized in that Each set of image data includes a series of Gray code images and a series of phase shift images.

24. The method according to claim 1, characterized in that Each set of image data includes color information.

25. A method of generating a three-dimensional high dynamic range image of one or more objects using an imaging system including an image sensor, characterized in that: The method includes: capturing, by the image sensor, one or more sets of image data at respective sets of exposure settings, each set of the image data being usable to generate one or more three-dimensional point clouds defining three-dimensional coordinates of points on the surfaces of the one or more objects, each set of exposure settings specifying values ​​of one or more parameters of the imaging system that affect an amount of signal reaching the image sensor; and constructing a three-dimensional point cloud using data from one or more sets of image data in each of the captured sets of image data, Wherein the exposure settings used in capturing each set of image data are determined by a method according to any one of claims 1-24.

26. A computer-readable storage medium, characterized in that: The method comprises computer executable code which, when executed by a computer, causes the computer to implement the method according to any one of claims 1 to 24.

27. An imaging system for implementing a three-dimensional image acquisition process by capturing one or more sets of image data with one or more sets of exposure settings, characterized in that: The one or more sets of image data can be used to generate one or more three-dimensional point clouds that determine the three-dimensional coordinates of points on the surface of one or more imaged objects. The imaging system includes an image sensor for capturing the one or more sets of image data. The imaging system is used to determine the one or more sets of exposure settings for the three-dimensional high dynamic range image acquisition process by implementing the method described in any one of claims 1 to 25.

Images