Method and apparatus for compensating for stray light caused by an object in a scene

Abstract

The present disclosure relates to a method and apparatus for compensating for stray light caused by an object in a scene. A method for compensating for stray light caused by an object in a scene sensed by a time-of-flight camera is provided. The method includes receiving an image of the scene from the time-of-flight camera. Additionally, the method includes controlling the time-of-flight camera to illuminate using a code-modulated signal to capture a reference image of the scene such that the measurement range of the time-of-flight camera is limited to a distance range around the object. The method additionally includes modifying the image of the scene or an image derived therefrom using the reference image to obtain a compensated image of the scene. The method includes outputting the compensated image.

Description

Technical Field

[0001] This disclosure relates to error correction for time-of-flight (ToF) sensing. In particular, examples relate to a method and apparatus for compensating for stray light caused by an object in a scene sensed by a ToF camera. Background Art

[0002] Highly reflective objects (e.g., road signs) can cause the ToF camera to make incorrect depth measurements. Reflections from highly reflective objects can cause incorrect measurement results due to the unwanted mixing of stray light from the highly reflective object with light reflected from other objects in the scene sensed by the ToF camera. Summary of the Invention

[0003] Accordingly, it may be desirable to compensate for stray light from objects in a scene.

[0004] This need can be met by the subject matter of the appended claims.

[0005] One example relates to a method for compensating for stray light caused by an object in a scene sensed by a ToF camera. The method includes receiving an image of the scene from the ToF camera. Additionally, the method includes controlling the ToF camera to illuminate using a code-modulated signal to capture a reference image of the scene such that the measurement range of the ToF camera is limited to a distance range around the object. The method further includes modifying the image of the scene or an image derived therefrom using the reference image to obtain a compensated image of the scene. The method includes outputting the compensated image.

[0006] Another example relates to an apparatus for compensating for stray light caused by an object in a scene sensed by a ToF camera. The apparatus includes an input circuit configured to receive an image of the scene from the ToF camera. Additionally, the apparatus includes a processing circuit configured to control the ToF camera to illuminate using a code-modulated signal to capture a reference image of the scene such that the measurement range of the ToF camera is limited to a range around the object. The processing circuit is further configured to modify the image of the scene or an image derived therefrom using the reference image to obtain a compensated image of the scene. The apparatus includes an output circuit configured to output the compensated image. Brief Description of the Drawings

[0007] Some examples of the apparatus and / or method will be described below by way of example only and with reference to the drawings, in which

[0008] Figure 1 shows a flowchart of an example of a method for compensating for stray light caused by an object in a scene sensed by a ToF camera;

[0009] Figure 2 shows an example of an intensity image of a scene including a highly reflective object;

[0010] Figure 3 An example of a depth image showing a scene including a highly reflective object is shown;

[0011] Figure 4 An example of a phasor caused by stray light from a highly reflective object in the scene is shown;

[0012] Figure 5 An example of a phasor caused by light reflected by a target object in the scene is shown;

[0013] Figure 6 An example of a phasor caused by the superposition of stray light from a highly reflective object and light reflected by a target object is shown;

[0014] Figure 7 An example of an autocorrelation function for code-modulated illumination signals is shown; and

[0015] Figure 8 An example of a device for compensating for stray light caused by an object in a scene sensed by a ToF camera is shown. Detailed Description

[0016] Various examples will now be described more fully with reference to the accompanying drawings which show some examples. In the drawings, for clarity, the thickness of lines, layers, and / or regions may be exaggerated.

[0017] Thus, although other examples can have various modifications and alternative forms, some specific examples thereof are shown in the drawings and will subsequently be described in detail. However, this detailed description does not limit other examples to the specific forms described. Other examples may cover all modifications, equivalent forms, and alternative forms falling within the scope of the present disclosure. Throughout the description of the drawings, the same or similar reference numerals refer to similar or like elements, which may be implemented identically to or in a modified form of each other while providing the same or similar functions to each other.

[0018] It will be understood that when an element is referred to as being "connected" or "coupled" to another element, these elements can be directly connected or coupled or via one or more intermediate elements. If two elements A and B are combined using "or", it should be understood that all possible combinations are disclosed, i.e., only A, only B, and A and B (if not otherwise explicitly or implicitly defined). Alternative wordings for the same combination are "at least one of A and B" or "A and / or B". Necessary modifications apply equally to combinations of more than two elements.

[0019] The terms used herein for the purpose of describing particular examples are not intended to limit other examples. Whenever singular forms such as "a", "an", and "the" are used and only a single element is neither explicitly nor implicitly defined as mandatory, other examples may also implement the same functionality using multiple elements. Similarly, when a functionality is subsequently described as being implemented using multiple elements, other examples may implement the same functionality using a single element or processing entity. It should also be understood that the terms "comprises", "comprising", "includes", and / or "including", when used, specify the presence of the stated features, wholes, steps, operations, processes, acts, elements, and / or components, but do not preclude the presence or addition of one or more other features, wholes, steps, operations, processes, acts, elements, components, and / or any group thereof.

[0020] Unless otherwise defined, all terms (including technical and scientific terms) used herein are to be construed in accordance with their ordinary meaning in the field of the examples to which they pertain.

[0021] Figure 1 A flowchart of a method 100 for compensating for light reflections such as stray light from an object in a scene sensed by a ToF camera is shown. Method 100 includes receiving 102 an image of the scene from the ToF camera. The image of the scene can be any type of image that can be provided by the ToF camera, such as a raw image or an image derived from the raw image. In some examples, the image of the scene can be an image that is based on one of the images listed above and corrected for one or more errors of the ToF camera. The image of the scene can be an image that is a combination of one of the images listed above and another image (e.g., a combination of two raw or depth images captured using modulation signals of different frequencies).

[0022] Figure 2 and 3 Two exemplary images 200 and 300 of a scene sensed by a ToF camera are shown. Figure 2 An intensity image 200 of the scene is shown, and Figure 3 a corresponding depth image 300 of the scene is shown. Before describing the remainder of method 100 in detail, for teaching purposes, the following paragraphs will introduce some basic knowledge of ToF depth measurement.

[0023] The ToF camera includes an illumination element for illuminating a scene with modulated light (e.g., infrared light). The illumination element generates modulated light (e.g., by controlling one or more light-emitting diodes LEDs or one or more laser diodes based on a modulation signal) based on an (electrical) modulated radio frequency signal such as a continuous wave modulation signal. Objects in the scene illuminated by the modulated light reflect at least a portion of the modulated light back to the light capture element of the ToF camera (e.g., including optics, an image sensor, and driver electronics). In other words, the light capture element receives the reflected light from the object.

[0024] The image sensor of the light capture element is pixelated, and each pixel measures a portion of the reflected light. Thus, an (electrical) measurement signal is generated based on the reflected light from the scene. For example, each pixel may include a photon mixer device (PMD) for measuring the reflected light.

[0025] Depending on the distance d between the ToF camera and the object obj , i.e., depending on the depth, the reflected light exhibits a delay with respect to the emission of the modulated light. Thus, the measurement signal experiences a distance-dependent (depth-dependent) phase shift with respect to the modulated radio frequency signal.

[0026] The modulation signal and the measurement signal of the corresponding pixel are correlated according to an (auto)correlation function in order to obtain a correlation value L for each pixel. The correlation function mimics the phase-distance function that describes the distance d measured by each pixel of the ToF camera obj and the relationship between the phase values.

[0027] The output of the correlation function is the correlation value L for each pixel. Then, the determined correlation values L are combined into an original image (sometimes also referred to as a "phase image"). That is, the original image includes a plurality of pixels, and each pixel represents the corresponding correlation value L.

[0028] To sample the correlation function , multiple original images are generated. The phase offset δ between the modulation signal and the measurement signal used for correlation varies between individual original images. In other words, different phase offsets δ are used to correlate the modulation signal and the measurement signal in order to obtain individual original images.

[0029] Sampling the same object at the same distance and exhibiting the same reflectivity enables sampling of the correlation function . For example, phase offsets of 0°, 90°, 180°, and 270° can be used to generate four original images, each original image including the correlation values L 0° , L 90° , L 180°and L 270° of a plurality of pixels.

[0030] For the phase correlation value L 0° , the correlation function is related to the phase shift between the measurement signal and the modulation signal that is shifted by the distance of the pixels from the zero value of the independent variable of the function

[0031] Using four correlation values L obtained by sampling the correlation function 0° , L 90° , L 180° and L 270° , the phase shift (phase angle) can be determined as follows

[0032]

[0033] Taking into account the speed of light c and the modulation frequency f of the emitted light p (i.e., the modulation frequency of the modulation signal), the distance d obj (i.e., the depth) to the object can be calculated as follows:

[0034]

[0035] In the Figure 2 and 3 captured scenes, a highly reflective object in the form of a road sign 210 is arranged in front of a white wall in a non-limiting manner. As can be seen from the intensity image 200 shown in Figure 2 , the intensity of the light reflected by the ToF camera from the road sign 210 is very high due to the high reflectivity of the road sign 210. Additionally, it can be seen that the road sign is surrounded by a corona with enhanced light intensity compared to the rest of the intensity image 200. The corona is caused by the (strong) stray light of the highly reflective road sign 210. In the Figure 3 shown depth image 300, the stray light of the highly reflective road sign 210 creates a corona of incorrect depth measurement values around the road sign 210. In particular, the stray light of the highly reflective road sign 210 causes incorrect depth measurements of the white wall in the background near the road sign 210. Although the influence of the stray light on Figure 3 the upper right corner of the white wall shown in Figure 3 is not significant, the influence of the stray light on

[0036] This is evident from Figures 4 to 6This will become more apparent in the exemplary phasors shown below. A phasor (also known as a phase vector) is a complex number that represents a ToF measurement. The phasor includes an angle that describes the (depth / distance-dependent) phase shift of the measurement signal measured by the light capture element of the ToF camera based on the reflected light from the scene relative to the modulated radio frequency signal used by the illumination element of the ToF camera to illuminate the scene. In addition, the phasor includes a length that describes the intensity of the measurement signal (i.e., the intensity of the reflected light from the scene). Since the angle of the phase depends on the depth / distance, sensing objects at different distances relative to the ToF camera causes the phasor to rotate.

[0037] A phasor image including phasors representing individual pixels of the light capture element of the ToF camera can be derived from one or more raw images of the ToF camera. Thus, the phasor image can be an example of an image derived from the raw images of the ToF camera.

[0038] Figure 4 Shown is a phasor 400 caused by stray light from a highly reflective road sign 210 shown by Figure 2 and 3 . The phasor 400 exhibits an intensity I 1 and an angle α 1 . The phasor 400 represents an error component for the ToF measurement for the area around the highly reflective road sign 210 because it represents the stray light from the highly reflective road sign 210.

[0039] Figure 5 Shown is a phasor 500 caused only by a white wall in the background of the highly reflective road sign 210. The phasor 500 exhibits an intensity I 2 and an angle α 2 . The phasor 500 represents a desired component for the ToF measurement because it represents the reflected light from the white wall without being affected by the stray light from the highly reflective road sign 210.

[0040] Figure 6 Shown is a phasor 600 measured by irradiating with a continuous wave signal by the ToF camera. The phasor 600 exhibits an intensity I 3 and an angle α 3 . As can be seen from Figure 6 , the phasor 600 is a superposition of the phasor 400 and the phasor 500. In other words, the phasor 600 is a vector addition of the phasor 400 caused by stray light and the correct phasor 500 for the white wall.

[0041] Similar to the description of the road sign 210 above, other objects such as windshields, traffic mirrors, or license plates may cause strong stray light and can thus be understood as highly reflective objects.

[0042] However, stray light is not only caused by highly reflective objects in the scene sensed by the ToF camera. In general, any object that reflects "too much" light can cause the aforementioned stray light artefacts. For example, an object (such as a hand or a face) that is (very) close to the ToF camera may reflect a lot of light back to the ToF camera, even though it is not highly reflective. That is, any object (regardless of whether it is highly reflective) can cause stray light, resulting in incorrect ToF measurements. For example, if the ToF camera is used in a facial recognition or gesture recognition application (e.g., sensing the hand of a vehicle driver to determine user input), any other sensed body part of the sensed face or user can be close to the ToF camera, such that the user's face or other body part causes stray light captured by the ToF camera.

[0043] Referring again to Figure 1 , method 100 may allow compensation for stray light caused by objects (e.g., highly reflective objects or objects close to the ToF camera) in the scene sensed by the ToF camera. Method 100 includes controlling 104 the ToF camera to capture a reference image of the scene by irradiating with a code-modulated signal (instead of a continuous wave signal used to capture an image of the scene), such that the measurement range of the ToF camera is limited to a distance range around the object that causes stray light.

[0044] Code modulation is used to fix the measurement range of the ToF camera to the area around the object in order to characterize the object. For code modulation, irradiation is performed using a code-modulated signal instead of a continuous wave modulation signal. In the code-modulated signal, the pulse sequence is varied. In other words, while the continuous wave modulation signal exhibits a series of alternating high and low pulses of equal length (duration), the code-modulated signal exhibits pulses of varying length. For example, a Kasami code sequence or an m-sequence can be used for the code-modulated signal.

[0045] The result of the code-modulated signal used for irradiation is that the correlation function only differs from a constant value with respect to the reflected light emitted from a certain distance range by the ToF camera. In other words, only the light reflected from the object within a certain distance range causes the value of the correlation function to differ from the constant value. In terms of a mathematical expression, this can be represented as follows:

[0046]

[0047] where c(d) represents the correlation function, d represents the distance from the object reflecting the light to the ToF camera, a represents the constant value, f(d) represents the distance correlation function, d min represents the minimum distance from the object reflecting the light to the ToF camera at which the correlation function is sensitive to the reflected light, and d max represents the maximum distance from the object reflecting the light to the ToF camera at which the correlation function is sensitive to the reflected light.

[0048] In other words, compared with a continuous wave modulation signal, for a code modulation signal, the correlation range of the correlation function is limited. The correlation range of the correlation function that is sensitive to the light reflected by an object sensed by a ToF camera defines the measurement range of the ToF camera. That is to say, the measurement range of the ToF camera corresponds to the correlation range of the correlation function that outputs a distance-related output value for the correlation function.

[0049] Capturing a scene by irradiating with a code modulation signal limits the measurement range of the ToF camera to the distance range around an object that causes stray light, and the stray light emitted from the object in the scene can be characterized. For example, a reference image of the scene generated by irradiating with a code modulation signal can allow the derivation of an unwanted phasor 400 caused by the stray light of the road sign 210.

[0050] The pixels of the reference image indicate (represent) reference values that are substantially exclusively related to the light reflection (stray light) from the object. In other words, the values indicated by the pixels of the reference image are caused only (substantially) by the light reflection (stray light) from the object, rather than by the light reflection of any other object such as an object in the surrounding environment of the object. The reference image can be, for example, a reference original image or an image derived therefrom. For example, the pixels of the reference phasor image only indicate phasors related to the light reflection (stray light) from an object such as Figure 4 the exemplary phasor 400 shown and caused by it. Therefore, the reference image allows compensation for the influence of the stray light from the object in the image of the scene.

[0051] When modifying an image of the 106 scene or an image derived therefrom, the value indicated by the pixels of the image of the scene or the image derived therefrom is modified by a reference value indicated by the pixels of the reference image. For example, the image of the scene or the image derived therefrom can be modified pixel by pixel using the reference image. In other words, the value indicated by the pixels of the image of the scene or the image derived therefrom is modified by the reference value indicated by the pixels located at the same or corresponding pixel positions in the reference image. For example, the reference value can be subtracted from the value indicated by the pixels of the image of the scene or the image derived therefrom. For example, the reference value can indicate the phase shift between the code modulation signal used for illumination and the measurement signal generated by the ToF camera based on the reflected light from the scene. The reference value can indicate a phasor, for example. In other words, the error phase caused by the stray light of the object can be corrected by subtracting the stray light phasor from the measured phasor. This can be done by subtracting the phase value of the coded modulation image (e.g., the original pixel output value) from the value of the initial continuous wave image. The result is a corrected target object that is not affected by the strong reflection of the object. However, the proposed concept is not limited to subtracting the reference value from the value indicated by the pixels of the image of the scene or the image derived therefrom. In general, the pixels of the image of the scene or the image derived therefrom can be modified / regulated / changed in any suitable way based on the pixels of the reference image.

[0052] Method 100 additionally includes outputting 108 a compensated image of the scene. Similar to what was described above for the received image of the scene, the compensated image of the scene can be, for example, the original image or an image derived from the original image, such as a phase angle image (including pixels representing the phase shift / phase angle derived from one or more correlation values of the corresponding pixels of the light capture element), an image derived from the phase angle image, a phasor image, an image derived from the phasor image, an intensity image, an image derived from the intensity image, a depth image, or an image derived from the depth image.

[0053] The compensated image of the scene is corrected for the effect of light reflection (stray light) from the object. Thus, in the compensated image of the scene, at least the erroneous measurement made by the ToF camera represented by the pixels of the image of the scene can be reduced. In some examples, the compensated image of the scene can be unaffected by light reflection (stray light) from the object.

[0054] In some examples, method 100 further includes determining that an object causes (intense, too much) stray light based on a scene image. As described above, to capture an image of a scene, a ToF camera uses a continuous wave modulation signal to illuminate the scene and generates a measurement signal based on the reflected light from the scene. The image of the scene is based on the correlation according to a correlation function between the continuous wave modulation signal and the measurement signal. Determining that an object causes stray light may include, for example, determining the distance of the object to the ToF camera based on the scene image. Additionally, determining that an object causes stray light may include comparing the correlation value of at least one of the correlations with a threshold. The correlation value represents the intensity of the light received from the object and is related to the depth / distance. Thus, the threshold depends on the determined distance of the object to the ToF camera. If the correlation value is higher than the threshold, it is determined that the object causes (intense, too much) stray light. For example, a conventional four-phase ToF depth image of the scene can be captured, and a highly reflective surface in the scene can be detected by its signal intensity (i.e., the magnitude of the autocorrelation function of the correlation).

[0055] Additionally, method 100 may include adjusting the modulation code such that the code modulation signal correlates like a continuous wave function within the range of the reflective surface. Thus, method 100 may include determining the distance of the object to the ToF camera based on the scene image. Additionally, method 100 may include adjusting the code modulation signal based on the determined distance of the object to the ToF camera.

[0056] Adjusting the code modulation signal may include, for example, selecting one of a plurality of predefined code modulation signals based on the distance of the object to the ToF camera. For example, each of the plurality of predefined code modulation signals may be designed to cover a specific correlation range of the correlation function, i.e., a specific measurement range of the ToF camera. By selecting one of the predefined code modulation signals, the measurement range of the ToF camera can be (substantially) limited to the object, such that the influence of the stray light of the object can be characterized by means of the captured reference image. In an alternative example, a modulation code for the code modulation signal, i.e., the code modulation signal, may be dynamically created (generated) based on the determined distance of the object to the ToF camera. The limited measurement range may be, for example, the foreground, a range independent of the application using the scene image, or the determined distance to the object plus / minus a predefined range (e.g., a few centimeters or tens of centimeters).

[0057] For example, a phase shift modulation code may be used to capture four different reference images. This is exemplarily shown in Figure 7 which is shown exemplarily Figure 7Depicts relevant (associated) correlation functions 710, 720, 730, and 740. The values of the correlation functions 710, 720, 730, and 740 differ only from the constant value of the light reflected from an object within the corresponding distance range (i.e., the first distance range for the first code-modulated signal, the second distance range for the second code-modulated signal, etc.). The phase shift of the modulation code between the four reference images allows mimicking the continuous-wave autocorrelation function within the range of the surface (object) that causes stray light.

[0058] In some examples, a four-phase ToF depth image can be obtained as an image of a scene, and four raw images can be obtained using phase-shift code-modulated signals to compensate for the stray light of the object. In other examples, the coded-modulation measurements can be reduced to two images (e.g., phase offsets of 0° and 90°). The phase values for the 180° and 270° phase offsets can be calculated by subtracting the sum of the phase values from the continuous-wave measurement (for capturing the scene image) from the coded-modulation images with phase offsets of 0° and 90°, because the sum of the phase values from the continuous-wave measurement in a pixel defines the origin. In other words, the image of the scene can be a raw image in a first series of raw images captured by a ToF camera, or derived from the first series of raw images. Additionally, the reference image can be a raw image in a second series of raw images captured by a ToF camera, or derived from the second series of raw images. The first series of raw images includes more raw images compared to the second series of raw images.

[0059] In some examples, the adjustment of the code-modulated signal can also be based on information about the movement of the ToF camera after capturing the image of the scene. This can allow compensating for changes in the position and / or orientation of the ToF camera between capturing the image of the scene and capturing the reference image of the scene. Information about the movement of the ToF camera can, for example, indicate the movement speed of the element that includes / holds the ToF camera, the turning (direction change) of the element that includes / holds the ToF camera, etc. For example, information about the movement of the ToF camera can be provided by a global navigation satellite system (GNSS) such as the Global Positioning System (GPS), Galileo, Beidou, or GLONASS, or an inertial measurement unit (IMU).

[0060] Modifying an image of a scene or an image derived therefrom using a reference image may also include scaling a reference value indicated by a pixel of the reference image by a scaling function to obtain a scaled reference image. Additionally, modifying an image of a scene or an image derived therefrom may include using the scaled reference image to modify the image of the scene or an image derived therefrom to obtain a compensated image. Scaling the reference value indicated by a pixel of the reference image may be used to compensate for differences between continuous wave measurements and coded modulation measurements. For example, different exposure times may be used for continuous wave measurements and coded modulation measurements. Additionally, the power of the light received from the scene is higher for continuous wave measurements compared to coded modulation measurements. By scaling the reference value indicated by a pixel of the reference image, effects such as the above exemplary effects may be compensated for. The scaling function may be, for example, a constant function (i.e., a scaling factor), a variable function depending on one or more parameters, a mapping function, or a look-up table.

[0061] The first result of method 100 is a corrected target object that is not affected by strong reflections of the object. The second result of method 100 is the depth and influence of the strong reflector that is not affected by the background. This may allow for the creation of a confidence image including two depth values (for the object and the surrounding area of the object) and its corresponding signal (correlation) intensity for each pixel. In other words, at least one pixel of the compensated image may indicate the distance to the ToF camera based on the modification of the image of the scene or an image derived therefrom using the reference image, and may additionally indicate the distance to the ToF camera of an object causing stray light.

[0062] In some examples, images of a scene at different illumination frequencies (e.g., 60 MHz and 80 MHz) may be captured and combined to enable long-range ToF measurements. The same code modulation signal may be used to correct the two images of the scene. Thus, method 100 may also include receiving another image of the scene from the ToF camera. The other image of the scene is captured by the ToF camera illuminating the scene using a continuous wave modulation signal that exhibits a different frequency compared to the continuous wave modulation signal used to capture the (initial) image of the scene (e.g., obtaining two images at modulation frequencies of 60 MHz and 80 MHz, respectively). Additionally, method 100 may include using the reference image to modify the other image of the scene or an image derived from the other image of the scene to obtain another compensated image of the scene (e.g., as described above for the image of the scene). A depth image of the scene may be obtained by combining the compensated image of the scene and the other compensated image of the scene.

[0063] Figure 8Further shown in [0] is an example of an apparatus 800 for compensating for stray light caused by an object in a scene according to the proposed concept. The apparatus 800 includes a processing circuit 820. For example, the processing circuit 820 can be a single dedicated processor, a single shared processor, or multiple individual processors (some or all of which can be shared), digital signal processor (DSP) hardware, an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA). The processing circuit 820 can optionally be coupled to, for example, a read only memory (ROM), a random access memory (RAM), and / or a non-volatile memory for storing software. The apparatus 800 can also include other hardware (conventional and / or customized).

[0064] The apparatus 800 includes an input circuit 810 configured to receive input data 801 representing an image of the scene and a reference image of the scene. The processing circuit 820 processes the input data 801 according to the above concept to compensate for stray light caused by an object in the scene. Accordingly, an output circuit 830 of the apparatus outputs output data 802 representing a compensated image of the scene.

[0065] For example, the functionality of the apparatus 800 can be implemented in an application processor coupled to a ToF camera module providing an image or in the ToF camera module itself.

[0066] The examples described herein can be summarized as follows:

[0067] Some examples relate to a method for compensating for stray light caused by an object in a scene sensed by a ToF camera. The method includes receiving an image of the scene from the ToF camera. Additionally, the method includes controlling the ToF camera to illuminate using a code-modulated signal to capture a reference image of the scene such that the measurement range of the ToF camera is limited to a distance range around the object. The method further includes modifying the image of the scene or an image derived therefrom using the reference image to obtain a compensated image of the scene. The method includes outputting the compensated image.

[0068] According to some examples, the image of the scene is one of an original image or an image derived from the original image.

[0069] In some examples, the method further includes determining that the object causes stray light based on the image of the scene.

[0070] According to some examples, to capture an image of a scene, a ToF camera illuminates the scene with a continuous wave modulation signal and generates a measurement signal based on the reflected light from the scene. The image of the scene is based on the correlation according to a correlation function between the continuous wave modulation signal and the measurement signal. Determining that an object causes stray light includes: determining the distance of the object to the ToF camera based on the image of the scene; comparing at least one correlation value in the correlation with a threshold value, where the threshold value depends on the determined distance of the object to the ToF camera; and if the correlation value is higher than the threshold value, determining that the object causes stray light.

[0071] In some examples, the method further includes determining the distance of the object to the ToF camera based on the image of the scene; and adjusting the code modulation signal based on the determined distance of the object to the ToF camera.

[0072] According to some examples, adjusting the code modulation signal includes selecting one of a plurality of predefined code modulation signals based on the distance of the object to the ToF camera.

[0073] In some examples, adjusting the code modulation signal is further based on information about the movement of the ToF camera after capturing the image of the scene.

[0074] According to some examples, using a reference image to modify the image of the scene or an image derived therefrom includes subtracting a reference value indicated by a pixel of the reference image from a value indicated by a pixel of the image of the scene or an image derived therefrom.

[0075] In some examples, the reference value indicates a phase shift between the code modulation signal and a measurement signal generated by the ToF camera based on the reflected light from the scene.

[0076] According to some examples, the reference value indicates a phasor.

[0077] In some examples, using a reference image to modify the image of the scene or an image derived therefrom includes: scaling the reference value indicated by a pixel of the reference image by a scaling function to obtain a scaled reference image; and using the scaled reference image to modify the image of the scene or an image derived therefrom to obtain a compensated image.

[0078] According to some examples, the image of the scene is an original image in a first series of original images captured by the ToF camera, or is derived from the first series of original images. The reference image is an original image in a second series of original images captured by the ToF camera, or is derived from the second series of original images. The first series of original images includes more original images than the second series of original images.

[0079] In some examples, at least one pixel of the compensated image indicates the distance to the ToF camera based on a modification of an image of a scene or an image derived therefrom using a reference image, and additionally indicates the distance of an object to the ToF camera.

[0080] According to some examples, the method further includes: receiving, from the ToF camera, another image of the scene, where the another image of the scene is captured by the ToF camera irradiating the scene with a continuous wave modulation signal that exhibits a different frequency compared to the continuous wave modulation signal used to capture the image of the scene; modifying the another image of the scene or an image derived therefrom using the reference image to obtain another compensated image of the scene; and combining the compensated image of the scene with the another compensated image of the scene to obtain a depth image of the scene.

[0081] Other examples relate to an apparatus for compensating for stray light caused by an object in a scene sensed by a ToF camera. The apparatus includes an input circuit configured to receive an image of the scene from the ToF camera. Additionally, the apparatus includes a processing circuit configured to control the ToF camera to capture a reference image of the scene by irradiating with a code modulation signal such that the measurement range of the ToF camera is limited to the range around the object. The processing circuit is further configured to modify the image of the scene or an image derived from the image of the scene using the reference image to obtain a compensated image of the scene. The apparatus includes an output circuit configured to output the compensated image.

[0082] Other examples relate to another apparatus for compensating for stray light caused by an object in a scene sensed by a ToF camera. The apparatus includes means for receiving an image of the scene from the ToF camera. Additionally, the apparatus includes means for controlling the ToF camera to capture a reference image of the scene by irradiating with a code modulation signal such that the measurement range of the ToF camera is limited to the range around the object. Additionally, the apparatus includes means for modifying the image of the scene or an image derived therefrom using the reference image to obtain a compensated image of the scene. The apparatus includes means for outputting the compensated image.

[0083] Examples relate to a non-transitory machine-readable medium having a program stored thereon with program code that, when executed on a processor or programmable hardware, is for performing the method for compensating for stray light caused by an object in a scene sensed by a ToF camera as described herein.

[0084] Other examples relate to a program having program code that, when executed on a processor or programmable hardware, is for performing the method for compensating for stray light caused by an object in a scene sensed by a ToF camera as described herein.

[0085] Examples according to the proposed concept may allow for surface error correction of continuous wave and coded modulation measurements for ToF cameras.

[0086] The description and drawings merely illustrate the principles of the present disclosure. In addition, all examples recited herein are in principle clearly only for illustrative purposes to assist the reader in understanding the principles of the present disclosure and the concepts contributed by the inventors for further development of the field. All statements herein that refer to the principles, aspects, and examples of the present disclosure, as well as the specific examples thereof, are intended to cover their equivalents.

[0087] Block diagrams, for example, may illustrate high-level circuit diagrams implementing the principles of the present disclosure. Similarly, flowcharts, flow diagrams, state transition diagrams, pseudocode, etc. may represent various processes, operations, or steps, which may, for example, be substantially represented in a computer-readable medium and thus executed by a computer or processor, whether or not such a computer or processor is explicitly shown. The methods disclosed in the description or the book may be implemented by a device having modules for performing each of the corresponding actions of these methods.

[0088] It should be understood that the disclosure of a plurality of actions, processes, operations, steps, or functions in the description or claims may not be construed as in a particular order unless explicitly or implicitly stated otherwise, for example, for technical reasons. Thus, the disclosure of a plurality of actions or functions will not limit them to a particular order unless these actions or functions are not interchangeable for technical reasons. In addition, in some examples, a single action, function, process, operation, or step may respectively include or may be respectively divided into a plurality of sub-actions, sub-functions, sub-processes, sub-operations, or sub-steps. Such sub-actions may be included in the disclosure of the single action and are part of the disclosure of the single action unless explicitly excluded.

[0089] In addition, the following claims are hereby incorporated into the detailed description, where each claim may stand alone as a separate example. Although each claim may stand alone as a separate example, it should be noted that although a dependent claim may refer to a specific combination with one or more other claims in the claims, other examples may also include combinations of the dependent claim with the subject matter of each other dependent or independent claim. Such combinations are explicitly set forth herein unless it is stated that a particular combination is not intended. In addition, it is intended that the features of the claims be included in any other independent claim as well, even if the claim does not directly depend on that independent claim.

Claims

1. A method (100) for compensating for stray light caused by an object in a scene sensed by a time-of-flight camera, the method comprising: receiving (102) an image of the scene from the time-of-flight camera; controlling (104) the time-of-flight camera to irradiate using a code modulation signal to capture a reference image of the scene, such that a measurement range of the time-of-flight camera is limited to a distance range around the object; modifying (106) the image of the scene or an image derived from the image of the scene using the reference image to obtain a compensated image of the scene; and outputting (108) the compensated image.

2. The method according to claim 1, wherein the image of the scene is one of an original image or an image derived from the original image.

3. The method according to claim 1 or 2, further comprising: determining whether the object causes stray light based on the image of the scene.

4. The method according to claim 3, wherein in order to capture the image of the scene, the time-of-flight camera irradiates the scene using a continuous wave modulation signal and generates a measurement signal based on reflected light from the scene, wherein the image of the scene is based on a correlation according to a correlation function between the continuous wave modulation signal and the measurement signal, and wherein determining that the object causes stray light comprising: determining a distance of the object to the time-of-flight camera based on the image of the scene; comparing a correlation value of at least one of the correlations with a threshold, wherein the threshold depends on the determined distance of the object to the time-of-flight camera; and if the correlation value is higher than the threshold, determining that the object causes stray light.

5. The method according to any one of claims 1 to 4, further comprising: determining a distance of the object to the time-of-flight camera based on the image of the scene; and adjusting the code modulation signal based on the determined distance of the object to the time-of-flight camera.

6. The method according to claim 5, wherein adjusting the code modulation signal comprising: selecting one of a plurality of predefined code modulation signals based on the distance of the object to the time-of-flight camera.

7. The method according to claim 5 or 6, wherein adjusting the code modulation signal is further based on information about movement of the time-of-flight camera after capturing the image of the scene.

8. The method according to any one of claims 1 to 7, wherein modifying (106) the image of the scene or an image derived from the image of the scene using the reference image comprising: subtracting a reference value indicated by a pixel of the reference image from a value indicated by a pixel of the image of the scene or an image derived from the image of the scene.

9. The method according to claim 8, wherein the reference value indicates a phase shift between the code modulation signal and a measurement signal generated by the time-of-flight camera based on reflected light from the scene.

10. The method according to claim 8, wherein the reference value indicates a phasor.

11. The method according to any one of claims 1 to 10, wherein the image of the scene or an image derived from the image of the scene is modified (106) using the reference image comprises: scaling a reference value indicated by a pixel of the reference image by a scaling function to obtain a scaled reference image; and using the scaled reference image to modify the image of the scene or an image derived from the image of the scene to obtain the compensated image.

12. The method according to any one of claims 1 to 11, wherein the image of the scene is an original image in a first series of original images captured by the time-of-flight camera, or the image of the scene is derived from the first series of original images, wherein the reference image is an original image in a second series of original images captured by the time-of-flight camera, or the reference image is derived from the second series of original images, and wherein the first series of original images includes more original images than the second series of original images.

13. The method according to any one of claims 1 to 12, wherein at least one pixel of the compensated image indicates a distance to the time-of-flight camera based on the modification of the image of the scene or an image derived from the image of the scene using the reference image, and additionally indicates a distance of the object to the time-of-flight camera.

14. The method according to any one of claims 1 to 13, further comprises: receiving another image of the scene from the time-of-flight camera, wherein the another image of the scene is captured by the time-of-flight camera irradiating the scene with a continuous wave modulation signal that exhibits a different frequency compared to the continuous wave modulation signal used to capture the image of the scene; using the reference image to modify the another image of the scene or an image derived from the another image of the scene to obtain another compensated image of the scene; and combining the compensated image of the scene and the another compensated image of the scene to obtain a depth image of the scene.

15. An apparatus (800) for compensating for stray light caused by an object in a scene sensed by a time-of-flight camera, the apparatus comprises: an input circuit (810) configured to receive an image of the scene from the time-of-flight camera; a processing circuit (820) configured to: control the time-of-flight camera to irradiate and capture a reference image of the scene using a code modulation signal such that a measurement range of the time-of-flight camera is limited to a range around the object; and use the reference image to modify the image of the scene or an image derived from the image of the scene to obtain a compensated image of the scene; and an output circuit (830) configured to output the compensated image.

Images