Reflectivity Sensing Using a Time-of-Flight Camera

Abstract

Embodiments of the present disclosure relate to reflectivity sensing using a time-of-flight camera. Techniques for obtaining reflectivity measurements using a time-of-flight (ToF) measurement device. An example method includes: obtaining distance measurements using one or more pixels in a ToF sensor; obtaining intensity measurements corresponding to the distance measurements using one or more pixels; and calculating a reflectivity value based on the distance measurements, the intensity measurements, and a reflectivity calibration factor. In some embodiments, the calibration distance is obtained as follows: by measuring a reference distance to a calibration surface using a reference pixel in the ToF sensor and obtaining the calibration distance from the measured reference distance. In some embodiments, the reflectivity calibration factor is obtained as follows: by obtaining a reference intensity corresponding to the reference distance using the reference pixel and calculating the reflectivity calibration factor as a function of the reference intensity and the viewing angles of one or more pixels.

Description

Technical Field

[0001] The present disclosure generally relates to time-of-flight (ToF) measurements, and more particularly to techniques for obtaining reflectivity measurements using a ToF sensor. Background Art

[0002] In optical sensing applications, depth measurement, i.e., measurement of the distance to individual features of one or more objects in the field of view of an image sensor, can be performed as a so-called time-of-flight (ToF) measurement, which is a distance measurement determined using the speed of light and an image / pixel sensor. The distance to an object of interest is typically calculated per pixel, and once calculated, this distance can be used for depth detection, pose recognition, object detection, etc. The per-pixel distances are combined to create a depth map, which provides a three-dimensional image. ToF measurement techniques are increasingly appearing in portable electronic devices (e.g., cellular phones and "smart" devices).

[0003] Many conventional TOF measurement schemes require multiple sequential exposures, also known as replicas. Each exposure requires amplitude modulation of the light generated from a light source using a modulation signal that is at a corresponding phase relative to a reference signal, which is applied to the pixels that demodulate the light reflected from one or more objects of interest. For different exposures, the phase is different. For example, one scheme requires four separate exposures where the phase of the modulation signal relative to the reference signal is at 0°, 90°, 180°, and 270° respectively. The measurement information from the four exposures is collected and compared to determine the depth map. For high-precision measurements with an extended unambiguous range, even more exposures can be performed, e.g., up to nine separate raw measurements. This conventional scheme and several variations and supporting hardware are described in detail in the co-pending U.S. Patent Application Serial No. 16 / 176,817, entitled "Image Sensor with Interleaved Hold for Single-Readout Depth Measurement", filed on October 31, 2018, the entire content of which is incorporated herein by reference for the purpose of providing context for the present disclosure. Summary of the Invention

[0004] Described herein are techniques for measuring the reflectivity of one or more imaged objects for each pixel of a time-of-flight (ToF) sensor. In this context, reflectivity is the efficiency of an object or material surface in reflecting radiant energy and can be understood as the percentage of light incident on the object or material surface that is reflected back to the ToF sensor. Using the calibration and measurement techniques described below, a ToF sensor can be used to generate a reflectivity map as well as a depth map. For example, this can be used to classify the imaged material and / or provide additional data for object and face recognition. Reflectivity data can also be used, for example, to detect skin color or detect sweat on a person's skin.

[0005] The embodiments described in detail below include example methods for reflectivity measurement implemented in a time-of-flight (ToF) measurement device. The example method includes the steps of: obtaining distance measurements using one or more pixels in the ToF sensor; and obtaining intensity measurements corresponding to the distance measurements using the same one or more pixels. The method also includes calculating a reflectivity value for one or more pixels based on the distance measurements, the intensity measurements, and a reflectivity calibration factor. These steps can be performed for each pixel or pixel group among several (or many) pixels or pixel groups using a reflectivity calibration factor specific to the corresponding pixel or corresponding pixel group.

[0006] In some embodiments, calculating the reflectivity value includes: multiplying the intensity measurement by the reflectivity calibration factor and multiplying by a factor proportional to the square of the distance measurement. In some embodiments, for example, the reflectivity value for each pixel i,j is calculated according to R i，j = c i，j I i，j (d i，j / d calib ) 2 where R i，j is the calculated reflectivity value for pixel i,j; c i，j is the reflectivity calibration factor for pixel i,j; I i，j is the measured intensity for pixel i,j; d i，j is the distance measurement for pixel i,j; and d calib is the reflectivity calibration distance.

[0007] In some embodiments, a ToF measurement device obtains a calibration distance as follows: by measuring a reference distance to a calibration surface, for example, using a reference pixel in the ToF sensor, and obtaining the calibration distance from the measured reference distance, where the reference pixel has a viewing angle aligned with the optical axis of the ToF sensor. In some embodiments, the ToF measurement device calculates a reflectivity calibration factor for each respective pixel or respective pixel group as a function of the viewing angle of the pixel or pixel group and the reference reflectivity, or as a function of the distance to the calibration surface measured by the pixel or pixel group and the reference reflectivity, or as a function of both of the above and the reference reflectivity.

[0008] For example, in some embodiments, for each pixel or each pixel group, the reflectivity calibration factor is obtained by: measuring the reflection intensity from the calibration surface for the pixel or pixel group; scaling the measured intensity based on the viewing angle of the pixel or pixel group relative to the optical axis of the ToF sensor; and calculating the reflectivity calibration factor for the pixel or pixel group based on the scaled measured intensity and the reference reflectivity. In some embodiments of these embodiments, the ToF measurement device measures the distance to the calibration surface for each pixel or each pixel group and calculates the scaled measured intensity for each pixel or each pixel group according to where: L i，j is the scaled measured intensity for pixel i,j; I i，j is the measured intensity for pixel i,j; dc i，j is the measured distance to the calibration surface for pixel i,j; β i，j is the viewing angle of pixel i,j relative to the optical axis of the ToF sensor; and d calib is the calibration distance.

[0009] The reflectivity values calculated according to the techniques disclosed herein can be used, for example, to normalize pixel values or groups of pixel values in a two-dimensional image. This can be used for machine learning purposes, such as allowing intensity values to be mapped to a fixed scale that is independent of distance. As another example, the reflectivity values calculated according to these techniques can be used to form a reflectivity image, or to form a three-dimensional point cloud that includes distance measurements and corresponding reflectivity values.

[0010] Also illustrated in the drawings and described below is an apparatus corresponding to the methods outlined above and detailed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a diagram illustrating a system for time-of-flight (ToF) measurement according to some embodiments of the embodiments described herein.

[0012] Figure 2An example photon mixing device (PMD) is illustrated.

[0013] Figure 3 is a diagram illustrating the principle of phase measurement according to the time-of-flight (TOF) technique.

[0014] Figure 4 illustrates the principle of an example calibration process according to some embodiments described herein.

[0015] Figure 5 is a flowchart illustrating an example method for measuring reflectance using a ToF sensor according to some embodiments.

[0016] Figure 6 is a block diagram illustrating components of an example ToF measurement device configured to perform reflectance measurements according to some embodiments. DETAILED DESCRIPTION

[0017] The present invention will now be described with reference to the accompanying drawings, in which like reference numerals are used throughout to refer to like elements, and in which the structures and devices illustrated are not necessarily drawn to scale. In the present disclosure, the terms “image” and “image sensor” are not limited to images or sensors involving visible light, but encompass the use of visible light and other electromagnetic radiation. Thus, the term “light” as used herein broadly means and refers to visible light, as well as infrared and ultraviolet radiation.

[0018] Figure 1 illustrates the basic principle of well-known continuous wave (CW) time-of-flight (TOF) measurement. A light source 110, such as a light emitting diode (LED) or a vertical cavity surface emitting laser (VCSEL), is modulated using an electrical signal (e.g., a radio frequency sine wave of, for example, 300 MHz) such that the light source 110 emits an amplitude-modulated optical signal towards a target scene 120. The optical signal travels at the speed of light c, the optical signal is reflected from one or more objects in the scene 120, and returns to reach a pixel array 135 in a TOF sensor 130, which has a time-of-flight to and from the target scene 120, thereby imposing a phase shift on the optical signal received at the pixel array 135 relative to the originally transmitted optical signal.

[0019] A modulation signal 137 is used to modulate the emitted light or its phase-shifted form, and the modulation signal 137 is also provided as a reference signal to the pixels in the pixel array 135 to correlate with the modulation signal superimposed on the reflected optical signal—in effect, the reflected optical signal is demodulated by each pixel in the pixel array 135.

[0020] Although the structure and design of the light-sensing pixels can vary, in some cases, each pixel in the pixel array 135 can be a photon mixing device or PMD. Figure 2 The basic structure of an example PMD is illustrated, which includes readout diodes A and B and modulation gates A and B. A reference signal is applied differentially across modulation gates A and B, thereby creating an electric potential gradient across the p-substrate, while the incident light is received at the photogates / diodes. A differential sensor signal is generated across readout diodes A and B. The sensor signal from the pixel can be integrated over a period of time to determine phase measurement information.

[0021] The voltage difference at the Read-A and Read-B nodes of the PMD corresponds to the following correlation: the correlation between the modulated optical signal detected by the photosensitive diode structure in the illustrated device and the reference signal, which is applied between the Mod-A and Mod-B nodes of the device. Thus, as discussed further below, the PMD (and other photosensitive pixel structures) demodulates the modulated optical signal reflected from the target scene 120, thereby generating a pixel signal value (in this case, the voltage difference between Read-A and Read-B), which indicates the distance traveled by the reflected optical signal.

[0022] Although the modulation signal can take any of a variety of forms, the principle behind this correlation / demodulation is most easily understood in the case where the modulation signal is a sine signal. If the modulation amplitude is "a" and the phase shift is The modulation signal g(t) and the received signal s(t) are given as:

[0023] m(t) = cos(ωt), and

[0024]

[0025] Then the correlation of the received signal with the reference signal gives:

[0026]

[0027] which is a function of the phase difference between the two signals. It should be understood that with a periodic modulation signal, this correlation can be performed over an extended period (e.g., several cycles of the modulation signal) to improve the signal-to-noise ratio of the resulting measurement.

[0028] The phase difference between the emitted optical signal and the reflection of the received optical signal can be extracted by an N-phase shift technique, and this phase difference is proportional to the distance traveled by the optical signal. This requires sampling the correlation function at N different points, which is performed, for example, by performing correlation relative to the modulation signal g(t) using N different phase shifts of a reference signal. At least two measurements are required to calculate the phase shift and thus determine the distance traveled. This is typically done using four different phase shifts at 0 degrees, 90 degrees, 180 degrees, and 270 degrees, as this allows for the simple elimination of systematic offsets in the correlation results. This can be seen in Figure 3 where Figure 3 it is shown how the correlations A0 and A1 at 0 degrees and 90 degrees respectively correspond to a first phase vector having an "ideal" component and a systematic component, the "ideal" component corresponding to the actual difference traveled by the optical signal, and the systematic component reflecting systematic errors in the measured values and readings. Similarly, the correlations A2 and A3 at 180 degrees and 270 degrees respectively correspond to a second phase vector pointing in the opposite direction, the second phase vector having an "ideal" component exactly opposite and the same systematic component. In the figure, the ideal component is represented by a vector extending from the origin to the circle, while the systematic error component is represented by a smaller vector. The actual phase can then be calculated as follows

[0029]

[0030] (It will be understood that the above expression is only valid when A2 - A0 is greater than zero. The well-known "atan2(y,x)" function can be used to provide the correct transformation to the phase vector for all cases, including when A2 - A0 is less than zero.)

[0031] Based on this phase, the distance or "depth" to the target scene 120 can be calculated as follows:

[0032]

[0033] where f mod is the frequency of the modulation signal. It should be understood that due to "phase wraping", since it is not possible to distinguish from a single distance calculation whether the distance traveled is less than a single wavelength or multiple wavelengths of the modulation waveform, this distance calculation has ambiguous results. Various techniques for resolving this ambiguity are well known, for example, by incorporating amplitude information obtained from the reflected optical signal, and / or by repeating the measurement using different modulation frequencies, but a detailed discussion of these techniques is not necessary for a full understanding of the technology of the present disclosure and is therefore outside the scope of the present disclosure.

[0034] In addition to the distance / depth measurements described above, intensity measurements for each pixel in the pixels of a ToF camera, or more generally a ToF sensor, can also be obtained from the correlations described above. More specifically, the light intensity is the amplitude of the autocorrelation function. For the four-phase measurements discussed above, the light intensity can be calculated as:

[0035] I = sqrt((A1 - A3) 2 - (A2 - A0) 2 ) / 2.

[0036] These intensity measurements can then be used to measure the amount of light reflected from an object. Due to the modulated illumination signal used to perform the distance measurement, other light sources (e.g., sunlight or background illumination) do not affect these intensity measurements. Instead, the external influence on the intensity measurement for a given pixel is the reflectivity of the object or material imaged by the pixel, and the distance from the ToF illumination unit to the imaged object or material and back to the pixel. The internal influences on the intensity measurement include: the illumination intensity in the direction of the imaged object or material, the sensitivity of the pixel, and the measurement time.

[0037] Described herein are techniques for compensating for the influence of the distance to an object such that reflectivity values for the imaged object or material can be obtained from the intensity measurements for the pixels. These techniques can also be used to compensate for variations in illumination intensity across the field of view, as well as variations in pixel sensitivity.

[0038] The reflectivity from a surface depends on the surface angle α, which is the angle between the surface of the object at the point viewed by a given pixel and the ray extending from the object to the pixel at that point. (It will be understood that a given pixel has a viewing area in the shape of a cone, and the intersection of the cone with the imaged object creates a viewing region, not a point. Thus, the "point" on the object viewed by a given pixel as referred to herein should be understood to refer to the point at or near the center of that region). The reflectivity measurements obtained by the techniques described herein will generally indicate the proportion of light reflected back from the object or material towards the pixel of interest that has been corrected for distance. If the surface angle α can be calculated as follows: for example, by evaluating the distance measurements of the pixel and several surrounding pixels, or by fitting a three-dimensional (3D) model of the object to the ToF distance data obtained from the ToF sensor, then the reflectivity value R for α = 0 can also be derived for each pixel, i.e., the reflectivity that would be observed at a given point on the object or material if that point were viewed directly (i.e., α = 0) rather than at an angle. This reflectivity value can be expressed as R(α = 0), which can be used to derive material properties (e.g., skin color, skin perspiration) or the type of material.

[0039] In either case, i.e., when a reflectance value is obtained that compensates only for distance or only for R (α = 0), the reflectance provides a unique way to sense the representation of an object. It provides additional information for object recognition applications such as face recognition and pose detection. In many of these applications, the dependence on the angle α is immaterial since the object classification model can learn from many different angles anyway.

[0040] The first step to enable the measurement of reflectance is camera calibration. As proposed above, each pixel in a ToF sensor can have variations in sensitivity. Additionally, the illumination unit that illuminates the object(s) or material of interest may not produce a uniform light distribution. Also, a reference needs to be obtained to have a scale for the sensed reflectance.

[0041] One process for calibration is to acquire a single intensity and depth image of a planar, diffuse calibration surface that is arranged such that the optical axis of the ToF sensor is perpendicular to the calibration surface. This can be done, for example, during the calibration of each ToF camera during manufacturing. Alternatively, this can be performed for a specifically designed ToF camera where the calibration result is used for similar cameras.

[0042] One step of this calibration process is to transform the intensity measurement I for each pixel i,j i，j to a virtual sphere S, where the intensity measurement I i，j is obtained from imaging the planar calibration surface and the center of the virtual sphere S is at the optical center of the ToF sensor. It will be understood that intensity measurements, depth measurements, etc. can be performed on a per-pixel basis or for a small group of adjacent pixels. For simplicity, the following discussion will refer to individual pixels arranged in a rectangular array such that they can be conveniently indexed using the indices i and j. However, it should be understood that in this discussion, a group of adjacent pixels can generally be substituted for each pixel and these pixels or pixel groups can be arranged in a configuration other than a rectangular array. Thus, the reference to "pixel i,j" can be understood to refer to any pixel or any group of pixels in the ToF sensor.

[0043] To transform each intensity measurement I i，j to the virtual sphere, the intensity that would be reflected from the sphere is calculated for each intensity measurement I i，jObtained from the imaging of the calibration surface. Since the center of the sphere is located at the optical center of the camera, and since it is assumed that the calibration surface and the virtual sphere have a uniform reflectivity, when the virtual sphere is sensed under the assumption of uniform illumination, perfect pixels should produce the same value. Of course, pixel sensitivity and illumination will vary. Therefore, the second step of the calibration process is to establish a correction factor or reflectivity calibration factor c for each pixel i，j r.

[0044] Transforming this value onto the sphere requires compensation for the different orientations of the planar calibration surface relative to the virtual sphere, and also requires compensation for the different distances from the ToF to the calibration surface, as well as the different distances from the ToF sensor to the virtual sphere. This can be seen in Figure 4 as follows Figure 4 illustrates a ToF sensor 110, which is arranged such that its optical axis points directly at a planar, diffuse calibration surface 120. For simplicity, it is assumed that the illumination source is at the optical focus of the ToF sensor 110, and the illumination source emits modulated light towards the calibration surface 120; the same assumption is made for the pixels of the ToF sensor 110. In Figure 4 the distance from the optical focus of the ToF sensor 110 to the planar calibration surface 120 is denoted as d calib and is the radius of a virtual sphere 130, which has its center at the optical focus of the ToF sensor 110 and the virtual sphere 130 touches the calibration surface 120. In some embodiments, at least one pixel on the ToF sensor 110 can view the calibration surface 120 directly along the optical axis of the ToF sensor - this pixel can be regarded as a "reference pixel". Other pixels will have various viewing angles β i，j relative to the optical axis. In Figure 4 the distance from each of the other pixels to the calibration surface 120 is denoted as d i，j . Given the above assumptions, it can be shown that if the calibration surface 120 and the ToF sensor 110 are oriented such that the optical axis of the sensor is exactly perpendicular to the calibration surface 120, then:

[0045] d i，j = dc alib / cosβ i，j .

[0046] Therefore, if the viewing angle β of each pixel is known i，j and the exact geometry of the calibration setup is known, then the distance d i，j can be calculated. Alternatively, of course, these distances can also be simply measured using the distance measurement capabilities of the ToF sensor 110.

[0047] UsingFigure 4 For the calibration settings shown, each pixel in the pixels of the ToF sensor 110 can measure the intensity I of light as follows i，j , which is reflected back towards the ToF sensor 110 by the calibration surface from the area viewed by the pixel. This intensity L i，j can be transformed into a "virtual" intensity measurement value, which represents the intensity that would be observed if the calibration surface 120 in the plane had the shape of a virtual sphere 130 - in other words, the measured intensity L i，j can be transformed onto the virtual sphere 130. This requires two compensations. First, the difference in distance, i.e., from d i，j to d calib can be resolved by applying the inverse square law. Second, assuming that the calibration surface 120 is diffuse, then the different angles, i.e., from β i，j to 0 can be resolved using Lambert's cosine law. (It can be shown that the angle β between the surface normal vectors n1 and n2 of the calibration surface 120 and the virtual sphere 130 is the same as the viewing angle β i，j for the pixel of interest (i.e., the angle between the optical axis of the ToF sensor and the center of the pixel's field of view).) The result is that the "virtual" intensity on the virtual sphere 130 can be expressed as:

[0048]

[0049] Given the assumptions discussed above, i.e., it can alternatively be expressed as:

[0050]

[0051] Similarly, under these same assumptions, the virtual intensity can be obtained from the distance alone according to the following formula:

[0052]

[0053] Note that the optical axis of the ToF sensor corresponds to the Figure 4 distance d in calib , and the optical axis of the ToF sensor must be perpendicular to the calibration surface 120 for the latter expression to be accurate.

[0054] In the above expressions, assuming the case of a diffuse material, the different angles are compensated using Lambert's cosine law, at which the ToF sensor 110 views the calibration surface 120. Alternatively, a custom function can be used, which can be provided in the data sheet of the reference material. As Figure 4 seen in, the angle β can be calculated from the normal vector of the surface.

[0055] Note that in the above expression, the virtual intensity L i，j is calculated relative to a virtual sphere 130 having a radius d calib where the radius d calib is equal to the distance from the ToF sensor 110 to the calibration surface 120. The virtual intensity is defined relative to the distance d calib . It should be understood that the virtual intensity can be calculated for a virtual sphere of any arbitrary radius using the inverse square law, regardless of the size of the calibration setting. Thus, for example, the virtual intensity can be transformed to a virtual sphere having a radius of "1" to simplify subsequent calculations. If the distance from the ToF sensor 110 to the calibration surface 120 is denoted as d axis and the distance d axis should be understood to represent any calibration distance that represents the radius of the virtual sphere which does not need to touch the calibration surface 120, then the virtual intensity on the virtual sphere having a radius d calib is:

[0056]

[0057] which simplifies to equation (1a):

[0058]

[0059] Thus it can be seen that d calib , the radius of the virtual sphere onto which the intensity is projected, can be the same as the distance from the ToF sensor 110 to the calibration surface 120, but the two do not necessarily need to be the same.

[0060] If the illumination from the light source at the ToF sensor 110 is perfectly uniform and if the pixels in the ToF sensor have perfectly uniform sensitivity, the virtual intensity L i，j can be expected to be the same. Of course, neither of these assumptions may be true. To account for these irregularities and transform the intensity values to reflectance values, a reference calibration factor c i，j can be calculated based on a reference reflectance R ref as:

[0061]

[0062] To obtain the "true" reflectance value, R ref should be the actual value of the reflectance of the calibration surface 120 at β = 0 (in which case the calculated reflectance as discussed below can be expected to fall within the range of 0 to 1). However, it will be understood that this reference reflectance R ref can be set to any arbitrary value to transform the intensity measurement values to the desired scale.

[0063] When the calibration factor for a given pixel i,j has been obtained, subsequent intensity measurements obtained from that pixel can be converted to reflectance values. In other words, multiplying the subsequent intensity measurement for pixel i,j by the calibration factor and using the inverse square law to correct to the calibration distance will yield the reflectance value for pixel i,j, which represents the percentage of light reflected back to the pixel from the imaged object or material:

[0064]

[0065] It should be understood that the term is a constant that depends on the calibration settings and not on any properties of the imaged object. Thus, if "true" reflectance values are not required, this term can be ignored, where the reflectance value is obtained by: multiplying the intensity measurement I i，j by the reflectance calibration factor c i，j and multiplying by any factor proportional to the square of the distance measurement d i，j . This will produce a reflectance value proportional to the "true" reflectance; appropriate selection of the reflectance calibration factor allows the result to be normalized to any desired scale.

[0066] Thus, it can be seen that the distance and intensity measurements to the object or material of the instance and the calibration factor can be used to obtain the calculated reflectance value for the imaged object. As in normal depth sensing, the ToF sensor captures a depth image and an amplitude image. The depth image may require capturing, for example, 4 or 8 raw images or exposures. The amplitude / intensity image can be calculated from these same 4 or 8 raw images. The distance from the pixel array to the surface being imaged is invariant to intensity. However, the distance between the illumination source and the surface needs to be compensated. This is done using the inverse square law as seen in equation (3). The reflectance calibration factor is used to convert the intensity measurement to a reflectance value and to correct for non-uniformities in illumination and pixel sensitivity.

[0067] The calculated reflectance values can be used in any of a number of image processing tasks. For example, the calculated reflectance values can be used to normalize corresponding pixel values or groups of pixel values in a two-dimensional image. The calculated reflectance values can be used to form a reflectance image or reflectance map, or used as an additional attribute for each point of a point cloud, such that the point cloud includes distance measurements and corresponding reflectance values. The reflectance values can use ToF distance measurements for mapping, and be associated with an RGB image. The reflectance values can be combined with depth and used as an input for enhanced object recognition. The reflectance values can be used, for example, to detect sweat on a human skin - the detected sweat can be used, for example, as an input to a lie detector algorithm, or to adjust a heating unit or an air conditioning unit. The surface of interest can be sensed at different angles using the techniques described herein, in order to sample the bidirectional reflectance distribution function (BRDF) of the surface. Since this functionality allows a rendering system to have realistic shadows, this functionality is valuable for 3D scanning. The ToF system may also have multiple different illumination units (e.g., red, blue, and green). Using the techniques described herein, taking images while each of these illumination units is active allows the true color of an object to be measured independent of the background illumination. Again, this can be valuable for 3D scanning if the scanned scene is placed in a virtual environment.

[0068] The calibration process using a planar calibration surface was described above, which includes the transformation of intensity values to a virtual sphere, the intensity values being obtained by imaging the surface. Instead, an actual sphere (or a portion of a sphere) can be used. Further, whether a planar or spherical calibration surface is used, the calibration can be performed individually for each pixel in the ToF sensor or for each group of pixels in the ToF sensor, but this is not necessarily required. The calibration can be performed, for example, for a subset of pixels or a group of pixels in such a way that the calibration factors thus obtained are then interpolated to those pixels or groups of pixels between the calibrated pixels. In this way, pixel-specific calibration factors can be determined for each pixel or each group of pixels, whether or not each pixel or each group of pixels is individually calibrated. For example, in some instances, the calibration surface can include a pattern of reflective and non-reflective (i.e., black) portions. If the exact dimensions of the pattern are known to the calibration unit, the alignment of the calibration surface relative to the ToF sensor can be evaluated and adjustments to the calibration process made accordingly. Although the reflectance calibration of pixels viewing the non-reflective portion cannot be directly calibrated, the neighboring pixels viewing the reflective portion can, and the reflectance calibration factors from those neighboring pixels can be interpolated or extrapolated to the pixels viewing the non-reflective portion of the calibration surface.

[0069] It should be further understood that if a particular model of ToF camera or other ToF measurement device has well-controlled manufacturing characteristics, this may be a situation where calibration does not need to be performed for each unit. Thus, for example, calibration can be performed using a reference device, where the calibration factors obtained from the reference device are pre-loaded into similar devices. This calibration can be updated from time to time, for example, using samples from the manufacturing line, to address drifts in the manufacturing process. Of course, since certain applications may be more tolerant of relatively inaccurate measurements, the feasibility of using this approach, as well as the possible frequency of updates, depends entirely on the desired accuracy for the reflectivity measurements.

[0070] In view of the detailed discussion and examples provided above, it will be understood that Figure 5 FIG. illustrates an example method in accordance with the techniques described herein for calibrating and using a ToF sensor for reflectivity measurements. As described above, calibration may not be performed for all pixels of a given ToF sensor, or even for a given sensor at all. Thus, Figure 5 the process shown can be divided into a calibration portion and a reflectivity measurement portion, and it will be understood that any given device or use instance may involve only one portion or the other or both.

[0071] The illustrated method begins with a calibration process as shown in blocks 510 - 540. As shown in block 510, the process begins by measuring the intensity of the reflection from a calibration surface for each of a plurality of pixels or for each of a group of pixels. As shown in block 520, the process may further include: measuring the distance to the calibration surface for each pixel or each group of pixels, and it should be understood that the distance measurement and the intensity measurement may be performed simultaneously, for example, in cases where both the distance and intensity values are obtained from multiple exposures or multiple phase images of the calibration surface.

[0072] As shown in block 530, the calibration process continues by scaling the measured intensity for each pixel or each group of pixels. This is done by projecting the measured intensity onto, for example, a virtual sphere, and thus correcting for distance and angle. This can be performed as a function of the viewing angle for the corresponding pixel or corresponding group of pixels, as a function of the measured distance to the calibration surface for one or more pixels, or as a function of both of the above. Examples of such scaling operations are shown in equations (1a), (1b), and (1c) above.

[0073] As shown in block 540, the calibration process continues to calculate a reflectivity calibration factor for each pixel or group of pixels based on the reference reflectivity and the scaled measured intensity for the pixel or group of pixels. As discussed above, in some embodiments, the reference reflectivity may represent the actual reflectivity of the calibration surface at zero viewing angle, but may alternatively represent any factor used to scale the intensity to a reflectivity value. In some embodiments, this may be done according to equation (2) above. Note that although Figure 5 (and the above equation) show the scaling of the intensity measurement value and the calculation of the reflectivity calibration factor as two different operations, in some embodiments, the reflectivity calibration factor may be calculated in one step, where the scaling of the intensity measurement value is inherent in the calculation.

[0074] Given the calibration factor for the pixels of the ToF sensor, the reflectivity measurement can be performed. An example measurement process is shown at Figure 5 blocks 550-570 in. Again, it should be understood that the measurement process can be performed separately and independently of the calibration process shown in blocks 510-540, for example, in the case where the reflectivity calibration factor is obtained using a different device and simply provided to the ToF measurement device that performs the reflectivity measurement.

[0075] As shown in block 550, the reflectivity measurement process begins by obtaining a distance measurement to the object or material of interest using one or more pixels in the ToF sensor. As shown in block 560, the method further includes: obtaining an intensity measurement corresponding to the distance measurement using one or more pixels. Again, it will be understood that the intensity measurement value and the distance measurement value can be obtained simultaneously, for example, using multiple-phase exposure of the object of interest.

[0076] As shown in block 570, the method further includes: calculating a reflectivity value for the pixel based on the distance measurement value, the intensity measurement value, and the reflectivity calibration factor. The reflectivity calibration factor may be pixel-specific, for example, and may have been obtained using the steps shown in blocks 510-540, but it may also be provided separately, or interpolated from one or more reflectivity calibration factors obtained according to the above techniques. For example, the calculation may include: multiplying the intensity measurement value by the reflectivity calibration factor and by a factor proportional to the square of the distance measurement value. In some embodiments, as discussed above, the reflectivity value is calculated according to equation (3) above.

[0077] The steps shown in 550-570 can be performed for each of several (or many) pixels (e.g., for all pixels in a ToF sensor), so the result provides a reflectivity map of one or more imaged objects. As discussed above, these reflectivity values can be used, for example, in combination with a depth map and / or an RGB color map, and / or in association with a 3D point cloud. As indicated above, in various applications, these reflectivity values can be used to enhance object recognition or characterization.

[0078] Figure 6 FIG. illustrates an example ToF measurement device 600 according to several embodiments of the presently disclosed devices and systems. The ToF measurement device 600 can be utilized to detect an object (e.g., as shown in the target scene 602), and to determine the distance to the detected object. The ToF measurement 600 can also be configured to determine the reflectivity values of one or more detected objects. The ToF measurement device 600 can be a continuous wave TOF system, such as a TOF system based on a photon modulation device (PMD). The ToF measurement device 600 can further be configured to superimpose data on the emitted light pulses according to the techniques described herein for reception by a remote device.

[0079] The illustrated ToF measurement device 600 includes a light source 624 configured to amplitude modulate a light beam with a modulation signal and to emit the amplitude-modulated light toward the scene 602. The amplitude modulation can be based on a reference signal generated by a reference signal generator 608. The reference signal can be, for example, a radio frequency (RF) signal in the MHz range, although other modulation frequencies can be used. The emitted light can include light having a varying wavelength range, such as light in the visible spectrum or infrared radiation. The emitted light reflects from one or more objects in the scene and returns to the sensor 604.

[0080] The illustrated ToF measurement device 600 further includes a sensor 604 including a plurality of pixels configured to generate a plurality of corresponding pixel signal values in response to the received light 614, where each pixel is configured to obtain the corresponding pixel signal value of the pixel by demodulating the received light using the reference signal 622. As Figure 6 seen, the received light 602 can reflect from the target scene 602. As discussed above, while several suitable pixel configurations are possible, one suitable pixel design is the PMD described above.

[0081] The number of pixels, rows, and columns can vary from one embodiment to another and is selected based on factors including desired resolution, intensity, etc. In one example, these sensor characteristics are selected based on the object to be detected and the expected distance to the object. Thus, for example, the pixel resolution of the pixels in sensor 604 can vary from one embodiment to another. Small objects require a higher resolution for detection. For example, finger detection requires a resolution of <5mm per pixel at a distance or range of about 0.5 meters. Medium-sized objects (such as hand detection) require a resolution of <20mm per pixel at a range of about 1.5 meters. Larger-sized objects (such as a human body) require a resolution of <60mm per pixel at about 2.5 meters. It should be understood that the above examples are provided for illustrative purposes only and variations may occur, including other objects, resolutions, and distances for detection. Some examples of suitable resolutions include VGA - 640x400 pixels, CIF - 352x288 pixels, QQ-VGA - 160x120 pixels, etc.

[0082] The ToF measurement device 600 also includes a reference signal generator 608, which in some embodiments can be configured to generate a reference signal 622 and provide the reference signal 622 to a plurality of pixels in the sensor 604. The reference signal 622 has a selectable phase relative to the phase of the modulation signal that is applied to the light transmitted towards the target scene 602. The image processing system 600 further includes an analog-to-digital converter (ADC) circuit 606, which can include one or more ADCs operatively coupled to a plurality of pixels in the sensor 604, where the ADC circuit 606 provides digital phase or distance measurements to a reflectivity map generator 610, which can include separate processing circuitry and / or digital logic and / or can be a functional unit implemented using the same circuitry as the composition control circuitry 612.

[0083] The illustrated ToF measurement device 600 also includes control circuitry 612, which can include, for example, a processor, a controller, etc. and / or other digital logic. In several embodiments, the control circuitry 612 is configured to cause the image processing system 600 to perform the above in connection with Figure 5The method described above. Thus, for example, the control circuitry 612 can be configured to control the light source 624, the sensor 604, and the reflectance map generator 610 to obtain distance measurements to an object or material of interest using one or more pixels in the sensor, and to obtain intensity measurements corresponding to the distance measurements using one or more pixels. Again, it will be understood that the intensity measurements and the distance measurements can be obtained simultaneously, for example, using multiple-phase exposures of the object of interest. The control circuitry 612 can further be configured to control the reflectance map generator to calculate reflectance values for the pixels based on the distance measurements, the intensity measurements, and a reflectance calibration factor. The reflectance calibration factor can be pixel-specific, for example, and the reflectance calibration factor can be obtained during a calibration process performed using the ToF measurement device 600, or can be provided to the ToF measurement 600. In some embodiments, as discussed above, the reflectance values are calculated according to equation (3) above.

[0084] These operations performed by the control circuitry 612 and the reflectance map generator 610 can be carried out for each of a number (or many) of pixels (e.g., for all pixels in the sensor 604), so that the result provides a reflectance map of one or more imaged objects. As discussed above, these reflectance values can be used, for example, in combination with a depth map and / or an RGB color map, and / or in association with a 3D point cloud. As indicated above, in various applications, these reflectance values can be used to enhance object recognition or characterization.

[0085] In some embodiments, the ToF measurement device 600 may further be configured to perform a calibration process. According to some of these embodiments, the control circuitry 612 is configured to control the light source 624, the sensor 604, and the reflectivity map generator 610 to measure the reflected intensity from a calibration surface for each pixel or each group of pixels among a plurality of pixels or a plurality of pixel groups, and to measure the distance to the calibration surface for each pixel or each group of pixels - again, it will be understood that the distance measurement and the intensity measurement may be performed simultaneously, for example, in cases where both the distance and intensity values are obtained from multiple exposures or multiple phase images of the calibration surface. The control circuitry 612 may further be configured to control the reflectivity map generator 610 to scale the measured intensity value for each pixel or each group of pixels, so as to project the measured intensity onto a virtual sphere. This may be performed as a function of the viewing angle for the corresponding pixel or corresponding group of pixels, as a function of the measured distance to the calibration surface for one or more pixels, or as a function of both of the above. Examples of such scaling operations are shown in equations (1a), (1b), and (1c) above. The control circuitry 612 in these embodiments may further be configured to control the reflectivity map generator 610 to calculate a reflectivity calibration factor for each pixel or each group of pixels as a function of a reference reflectivity and the scaled measured intensity for the pixel or pixel group. As discussed above, in some embodiments, the reference reflectivity may represent the actual reflectivity of the calibration surface at zero viewing angle, but may alternatively represent any factor used to scale the intensity to a reflectivity value. In some embodiments, this may be done according to equation (2) above. Note that although Figure 5 (and the above equations) show the scaling of the intensity measurement value and the calculation of the reflectivity calibration factor as two different operations, in some embodiments, the reflectivity calibration factor may be calculated in one step, where the scaling of the intensity measurement is inherent in the calculation.

[0086] In view of the detailed discussion above, it will be understood that the inventive subject matter described herein may be implemented as a method, apparatus, or article of manufacture using standard programming and / or engineering techniques to produce software, firmware, hardware, or any combination thereof, to control a computer to implement the disclosed subject matter. The term "article of manufacture" as used herein is intended to cover a computer program accessible from any computer-readable device, carrier, or medium. Of course, those skilled in the art will recognize that many modifications may be made to this configuration without departing from the scope or spirit of the claimed subject matter.

[0087] In particular, with respect to the various functions performed by the above-described components or structures (assemblies, devices, circuits, systems, etc.) and the terms used to describe such components (including references to "component"), unless otherwise specified, it is intended to correspond to any component or structure that performs the specified function (e.g., is functionally equivalent), even if such any component or structure is not structurally equivalent to the structure that performs the function disclosed in the exemplary implementations of the present invention illustrated herein. Additionally, although a particular feature of the present invention may be disclosed only with respect to one of several implementations, such feature may be combined with one or more other features of other implementations that are desirable or advantageous for any given or particular application. Further, to the extent that the terms "including / includes", "having / has / with" or variants thereof are used in the detailed description and claims, these terms are intended to be inclusive in a manner similar to the term "comprising".

Claims

1. A method in a time-of-flight (ToF) measurement device, the method comprising: Using one or more pixels in a ToF sensor to obtain distance measurement values; Using the one or more pixels to obtain intensity measurement values corresponding to the distance measurement values; And Calculating reflectance values of the one or more pixels based on the distance measurement values, the intensity measurement values, and a reflectance calibration factor, Wherein the reflectance calibration factor uses Lambert's cosine law to compensate for at least one of the viewing angles of the one or more pixels, and uses the inverse-square law to compensate for the difference between the distance measured by the one or more pixels to a planar, diffuse, calibration surface and a distance calibration value.

2. The method according to claim 1, wherein the reflectance calibration factor is specific to the one or more pixels in the ToF sensor.

3. The method according to claim 1, wherein the calculation comprises: Multiplying the intensity measurement values by the reflectance calibration factor and by a factor proportional to the square of the distance measurement values.

4. The method according to claim 1, wherein the reflectance value is calculated according to: R i，j = c i，j I i,j (d i，j / d calib ) 2 , where R i，j is the calculated reflectivity value, c i，j is the reflectivity calibration factor, I i，j is the intensity measurement value, d i，j is the distance measurement value, and d calib is the calibration distance for the reflectivity calibration factor.

5. The method according to claim 1 further comprises: Using the calculated reflectance value to normalize pixel values or groups of pixel values in a two-dimensional image.

6. The method according to claim 1, wherein the method comprises: Using the ToF sensor to obtain a plurality of distance measurement values, wherein each distance measurement value is obtained using a corresponding pixel or group of pixels in the ToF sensor; Using the corresponding pixel or group of pixels to obtain intensity measurement values corresponding to each of the distance measurement values; And Calculating reflectance values corresponding to each intensity measurement value based on the intensity measurement values, the corresponding distance measurement values, and corresponding reflectance calibration factors.

7. The method according to claim 6, wherein the corresponding calibration factors are each specific to a corresponding pixel or group of pixels.

8. The method according to claim 6, wherein for each respective pixel or each respective pixel group, the calculation comprises: Multiplying the intensity measurement values for the corresponding pixel or group of pixels by the reflectance calibration factor for the corresponding pixel or group of pixels and by a factor proportional to the square of the distance measurement value for the corresponding pixel or group of pixels.

9. The method according to claim 6, wherein the reflectance value for each pixel i,j is calculated according to: R i，j = c i，j I i，j (d i，j / d calib ) 2 , where: R i，j is the calculated reflectance value for pixel i,j; c i，j is the reflectivity calibration factor for pixel i, j; I i,j is the measured intensity for pixel i, j; d i,j is the distance measurement value for pixel i, j; And d calib is the calibration distance.

10. The method according to claim 6 further comprises: Using the calculated reflectance value to normalize pixel values or groups of pixel values in a two-dimensional image.

11. The method according to claim 6, wherein the method further comprises: Forming a reflectance image from the calculated reflectance values.

12. The method according to claim 6, wherein the method further comprises: Forming a three-dimensional point cloud that includes the distance measurement values and the corresponding reflectance values.

13. The method according to claim 6, wherein the method comprises: Obtaining the calibration distance by: Using a reference pixel in the ToF sensor to measure a reference distance to a calibration surface and obtaining the calibration distance from the measured reference distance.

14. The method according to claim 6, wherein the method comprises: Obtaining the reflectance calibration factor by: Calculate the reflectivity calibration factor for each pixel or each pixel group of the corresponding pixels or pixel groups as a function of the viewing angle and the reference reflectivity of the pixel or pixel group, or as a function of the distance to the calibration surface and the reference reflectivity measured by the pixel or pixel group, or as a function of both the viewing angle and the distance to the calibration surface and the reference reflectivity for the pixel or pixel group.

15. The method according to claim 14, wherein the function is R ref / cosβ i，j function, where R ref is the reference reflectivity, and β i，j is the viewing angle of the corresponding pixel or the corresponding pixel group relative to the optical axis of the ToF sensor.

16. The method according to claim 6, wherein the method comprises: For each pixel or each pixel group, obtain the reflectivity calibration factor as follows: Use the pixel or pixel group to measure the reflected intensity from the calibration surface; Scale the measured intensity based on the viewing angle of the pixel or pixel group relative to the optical axis of the ToF sensor; Calculate the reflectivity calibration factor for the pixel or pixel group as a function of the scaled measured intensity and the reference reflectivity.

17. The method according to claim 16, wherein scaling the measured intensity comprises: According to L i,j = I i,j / (cosβ i,j ) 3 to calculate the scaled measured said intensity, where: L i,j is the scaled measured intensity for pixel i, j; I i,j is the measured intensity for pixel i, j; and β i,j is the viewing angle of pixel i, j with respect to the optical axis of the ToF sensor.

18. The method according to claim 16, wherein the method further comprises: Measure the distance to the calibration surface for each pixel or each group of pixels, and wherein scaling the measured intensity comprises: according to calculate the scaled measured intensity, wherein: L i,j is the scaled measured intensity for pixel i, j; I i,j is the measured intensity for pixel i, j; dc i，j is the measured distance to the calibration surface for pixel i,j; β i，j is the viewing angle of pixel i, j relative to the optical axis of the ToF sensor; and d calib is the calibration distance.

19. A time-of-flight (ToF) measurement device, comprising: A light source configured to emit a series of light pulses formed by amplitude modulating light using a modulation signal; A ToF sensor including a plurality of pixels configured to generate corresponding pixel signal values in response to received light, wherein each pixel is configured to obtain the corresponding pixel signal value of each pixel by demodulating the received light using a reference signal; A reference signal generator configured to generate the reference signal and provide the reference signal to the plurality of pixels; And A control circuit system configured to control the light source, the ToF sensor, and the reflectivity map generator to: Use one or more pixels in the ToF sensor to obtain a distance measurement value; Use the one or more pixels to obtain an intensity measurement value corresponding to the distance measurement value; And Calculate a reflectivity value for the one or more pixels based on the distance measurement value, the intensity measurement value, and the reflectivity calibration factor, wherein the reflectivity calibration factor compensates for at least one of the viewing angles of the one or more pixels using Lambert's cosine law and compensates for the difference between the distance to a planar, diffuse, calibration surface measured by the one or more pixels and a distance calibration value using the inverse-square law.

20. The ToF measurement device according to claim 19, wherein the reflectivity calibration factor is specific to the one or more pixels in the ToF sensor.

21. The ToF measurement device according to claim 19, wherein the calculation includes: Multiply the intensity measurement value by the reflectivity calibration factor and by a factor proportional to the square of the distance measurement value.

22. The ToF measurement device according to claim 19, wherein the reflectivity value is calculated as follows: R i，j = c i,j I i,j (d i，j / d calib ) 2 where R i，j is the calculated reflectivity value, c i，j is the reflectivity calibration factor, I i，j is the intensity measurement value, d i，j is the distance measurement value, and d calib is the calibration distance for the reflectivity calibration factor.

23. The ToF measurement device according to claim 19, wherein the control circuit system is configured to control the light source, the ToF sensor, and the reflectivity map generator to: Use the ToF sensor to obtain a plurality of distance measurements, where each distance measurement is obtained using a corresponding pixel or corresponding pixel group in the ToF sensor; Use the corresponding pixel or corresponding pixel group to obtain an intensity measurement corresponding to each of the distance measurements; And Based on the intensity measurement, the corresponding distance measurement, and the corresponding reflectivity calibration factor, calculate a reflectivity value corresponding to each intensity measurement.

24. The ToF measurement device according to claim 23, wherein the corresponding calibration factors are each specific to a corresponding pixel or corresponding pixel group.

25. The ToF measurement device according to claim 23, wherein for each corresponding pixel or each corresponding pixel group, the reflectivity value is calculated by: multiplying the intensity measurement for the corresponding pixel or corresponding pixel group by the reflectivity calibration factor for the corresponding pixel or corresponding pixel group, and multiplying by a factor proportional to the square of the distance measurement for the corresponding pixel or corresponding pixel group.

26. The ToF measurement device according to claim 23, wherein the reflectivity value for each pixel i,j is calculated according to: R i，j = c i,j I i,j (d i，j / d calib ) 2 where: R i，j is the calculated reflectance value for pixel i, j; c i，j is the reflectivity calibration factor for pixel i, j; I i，j is the measured intensity for pixels i, j; d i，j is the distance measurement value for pixels i, j; And d calib is the calibration distance.

27. The ToF measurement device according to claim 23, wherein the control circuitry is configured to control the light source, the sensor, and the reflectivity map generator to measure a reference distance to a calibration surface using a reference pixel in the ToF sensor and obtain the calibration distance from the measured reference distance.

28. The ToF measurement device according to claim 23, wherein the control circuitry is configured to control the light source, the sensor, and the reflectivity map generator to obtain the reflectivity calibration factor by: Calculating the reflectivity calibration factor for each pixel or each pixel group in the corresponding pixel or corresponding pixel group as a function of the viewing angle and reference reflectivity of the pixel or pixel group, or as a function of the distance to the calibration surface measured by the pixel or pixel group and the reference reflectivity, or as a function of both the viewing angle and the distance to the calibration surface of the pixel or pixel group and the reference reflectivity.

29. The ToF measurement device according to claim 28, wherein the function is R ref / cosβ i，j function, where R ref is the reference reflectivity, and β i，j is the viewing angle of the corresponding pixel or the corresponding pixel group with respect to the optical axis of the ToF sensor.

30. The ToF measurement device according to claim 23, wherein the control circuitry is configured to control the light source, the sensor, and the reflectivity map generator to obtain the reflectivity calibration factor for each pixel or each pixel group by: Using the pixel or pixel group to measure the reflected intensity from the calibration surface; Scaling the measured intensity based on the viewing angle of the pixel or pixel group relative to the optical axis of the ToF sensor; Calculating the reflectivity calibration factor for the pixel or pixel group as a function of the scaled measured intensity and the reference reflectivity.

31. The ToF measurement device according to claim 30, wherein scaling the measured intensity includes: According to L i，j = I i，j / (cosβ i，j ) 3 to calculate the scaled measured said intensity, where: L i,j is the scaled measured intensity for pixel i, j; I i，j is the measured intensity for pixel i, j; and β i,j is the viewing angle of pixel i,j with respect to the optical axis of the ToF sensor.

32. The ToF measurement device according to claim 30, wherein the control circuitry is configured to control the light source, the sensor, and the reflectivity map generator to measure the distance to the calibration surface for each pixel or each group of pixels, and is configured to scale the measured intensity by calculating the scaled measured intensity, where: L i,j is the scaled measured intensity for pixel i,j; I i，j is the measured intensity for pixel i, j; dc i，j is the measured distance to the calibration surface for pixel i, j; β i，j is the viewing angle of pixel i,j with respect to the optical axis of the ToF sensor; And d calib is the calibration distance.

Images