Method and device for compensating for light reflections from the cover of a time-of-flight camera

Abstract

A method for compensating for light reflection from a cover of a time-of-flight camera in an image of a scene sensed by the time-of-flight camera is provided. The method receives an image of the scene from the time-of-flight camera. In addition, the method includes modifying the image of the scene using a reference image to obtain a compensated image of the scene. Pixels of the reference image indicate reference values related only to the light reflection from the cover of the time-of-flight camera. Additionally, the method includes outputting the compensated image.

Description

Technical Field

[0001] The present disclosure relates to error correction for time-of-flight (ToF) sensing. Specifically, examples relate to methods and devices for compensating for light reflections from a cover of a ToF camera. Background Art

[0002] ToF cameras are typically covered by cover glass to protect the imaging element and the illumination element from the surrounding environment. However, the light emitted by the illumination element is partially reflected by the cover glass. The reflection causes incorrect measurements because the light reflected by the cover glass and the light reflected by the scene sensed by the ToF camera produce an undesired light mixture. Summary of the Invention

[0003] Therefore, there is a need to compensate for light reflections from the cover of a ToF camera.

[0004] This need can be met by the subject matter of the appended claims.

[0005] One example relates to a method for compensating for light reflections from a cover of a ToF camera in an image of a scene sensed by the ToF camera. The method includes receiving an image of the scene from the ToF camera. Additionally, the method includes modifying the image of the scene using a reference image to obtain a compensated image of the scene. The pixels of the reference image indicate reference values related only to the light reflections from the cover of the ToF camera. Additionally, the method includes outputting the compensated image.

[0006] Another example relates to another method for compensating for light reflections from a cover of a ToF camera. The light capture element of the ToF camera is covered by a cover and includes an array of photon mixing devices. Each photon mixing device separates the charge generated by the light arriving at the photon mixing device to provide two charge values for the corresponding photon mixing device. The method includes receiving the charge values of the photon mixing devices. Additionally, the method includes modifying the charge values using reference values related only to the light reflections from the cover of the ToF camera to obtain compensated charge values. The method further includes outputting the compensated charge values. Brief Description of the Drawings

[0007] Some examples of the device and / or method will now be described only by way of example and with reference to the drawings, in which:

[0008] Figure 1 A flowchart of an example of a method for compensating for light reflections from a cover of a ToF camera in an image of a scene sensed by the ToF camera is shown;

[0009] Figure 2 An example of a ToF camera is shown;

[0010] Figure 3Shows an exemplary comparison between two intensity images of a ToF camera;

[0011] Figure 4 Shows an exemplary comparison between two depth images of a ToF camera;

[0012] Figure 5 Shows an example of a measurement setup for determining a reference image;

[0013] Figure 6 Shows a flowchart of an example of another method for compensating for light reflection from a cover of a ToF camera; and

[0014] Figure 7 Shows an example of a device for compensating for light reflection from a cover of a ToF camera. Detailed Description

[0015] Various examples will now be described more fully with reference to the accompanying drawings, which show some examples. In the drawings, for clarity, the thickness of lines, layers, and / or regions may be exaggerated.

[0016] Thus, while further examples are capable of various modifications and alternative forms, some specific examples are shown in the figures and will subsequently be described in detail. However, this detailed description does not limit further examples to the specific forms described. Further examples may cover all modifications, equivalents, and alternatives falling within the scope of the present disclosure. The same or similar numbers represent the same or similar elements throughout the description of the drawings, and these elements may be the same or implemented in a modified form when compared to each other while providing the same or similar functions.

[0017] It should be understood that when an element is referred to as being "connected" or "coupled" to another element, the element may be directly connected or coupled, or via one or more intermediate elements. If two elements A and B are combined using "or", and nothing else is expressly or implicitly stated, this will be understood to disclose all possible combinations, i.e., only A, only B, and A and B. Alternative wordings for the same combination are "at least one of A and B" or "A and / or B". With variations in added details, this also applies to combinations of more than two elements.

[0018] The terms used herein to describe particular examples are not intended to limit further examples. Whenever a singular form such as "a" and "the" is used and there is no explicit or implicit limitation to a single element being mandatory, further examples may also implement the same functionality using multiple elements. Similarly, when a functionality is subsequently described as being implemented using multiple elements, further examples may implement the same functionality using a single element or processing entity. It will be further understood that the terms "comprising" and / or "including", when used, specify the presence of the recited features, integers, steps, operations, processes, acts, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, processes, acts, elements, components, and / or any group thereof.

[0019] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the ordinary meaning ascribed to them in the general field of the examples.

[0020] Figure 1 A flowchart of a method 100 for compensating for light reflection from a cover of a ToF camera in an image of a scene sensed by the ToF camera is shown. Before describing method 100 in detail, for reasons of the teachings related to Figure 2 the following paragraphs will introduce some basic knowledge of ToF depth measurement.

[0021] The ToF camera 200 includes an illumination element 210 for irradiating a scene with modulated light 211 (e.g., infrared light). The illumination element 210 generates the modulated light 211 (e.g., by controlling one or more light-emitting diodes LEDs or one or more laser diodes based on a modulation signal) based on an (electrical) modulated radio frequency signal such as a continuous wave modulation signal. An object 230 in the scene illuminated by the modulated light 211 reflects at least a portion of the modulated light 211 back to a light capture element 220 of the ToF camera 200 (e.g., including optical elements, an image sensor, and driver electronics). In other words, the light capture element 220 receives the reflected light 231 from the object 230. For example, if the ToF camera 200 is used for secure face recognition, the object 230 may be a human face.

[0022] The image sensor of the light capture element 220 is pixelated, and each pixel measures a small portion of the reflected light 231. Accordingly, an (electrical) measurement signal based on the reflected light 231 from the scene is generated. For example, each pixel may include a photon mixing device (PMD) for measuring the reflected light 231.

[0023] According to the distance d between the ToF camera 200 and the object 230 obj , i.e., according to the depth, the reflected light 231 shows a delay with respect to the emission of the modulated light 211. Therefore, the measurement signal experiences a distance-related (depth-related) phase shift with respect to the modulated radio frequency signal.

[0024] According to the (auto) correlation function Correlate the modulation signal and the measurement signal for each pixel to obtain a correlation value L for each pixel. The correlation function Simulate the phase-distance function that describes the distance d measured by each pixel of the ToF camera 200 obj and the relationship between the phase values.

[0025] The correlation function The output is the correlation value L for each pixel. Then, the determined correlation values L are combined into the original image (sometimes also referred to as the "phase image"). That is, the original image includes a plurality of pixels, and each pixel represents the corresponding correlation value L.

[0026] To sample the correlation function Generate a plurality of original images. The phase shift δ between the modulation signal and the measurement signal for correlation varies between the respective original images. In other words, different phase shifts δ are used to correlate the modulation signal and the measurement signal to obtain the corresponding original images.

[0027] Sampling the same object at the same distance and showing the same reflectivity can sample the correlation function For example, phase shifts of 0°, 90°, 180°, and 270° can be used to generate four original images, each of which includes pixels representing the corresponding correlation values L 0° 、L 90° 、L 180° and L 270° .

[0028] For the phase correlation value L 0° , the correlation function is shifted relative to the zero value of the function parameter by the phase shift correlated with the distance between the measurement signal and the modulation signal of the pixel.

[0029] Using the four correlation values L sampling the correlation function 0° 、L 90° 、L 180° and L 270° , the phase shift (phase angle) can be determined as follows

[0030]

[0031] Considering the speed of light c and the modulation frequency f of the emitted light 211 p (i.e., the modulation frequency of the modulation signal), the distance d to the object 230 can be calculated as follows obj , i.e., the depth:

[0032]

[0033] The illumination element 210 and the light trapping element 220 are arranged (accommodated) in a common cavity 250 covered by a cover 240 to protect the illumination element 210 and the light trapping element 220 from the surrounding environment (e.g., dust or moisture). For example, the cover 240 can be made of glass, plastic, or any other suitable material. For example, the cover 240 can be a glass cover of a mobile phone or an automotive ToF system. In some examples, the cover 240 can be an OLED (organic light emitting diode) display. It should be noted that the cover 240 can be any element that can protect the illumination element 210 and the light trapping element 220 from the surrounding environment and is partially transparent to the modulated light 211 emitted by the illumination element 210 and the reflected light 231 to be received by the light trapping element 220.

[0034] As Figure 2 shown, the modulated light 211 emitted by the illumination element 210 is partially reflected by the cover 240. Thus, in addition to the desired reflected light 231 from the object 230, the light trapping element 220 also receives unwanted reflected light 241 from the cover 240. The unwanted reflected light 241 from the cover 240 is mixed with the desired reflected light 231 from the object 230, resulting in an incorrect measurement by the ToF camera. For another scenario sensed by the ToF camera, this is illustrated in Figure 3 and Figure 4 which.

[0035] Figure 3 A comparison between two intensity images captured by the ToF camera 200 is shown. The upper intensity image 310 was captured by the ToF camera 200 without the cover 240. In other words, when the ToF camera 200 captured the image 310, the cover 240 was removed. As a comparison, the lower intensity image 320 was captured by the ToF camera with the cover 240 installed.

[0036] The unwanted reflected light 241 from the cover 240 is mixed with the desired reflected light 231 from the sensed scene. As can be seen from the intensity images 310 and 320, due to the unwanted reflected light 241 from the cover 240, the contrast of the intensity image deteriorates.

[0037] Figure 4 shown in Figure 3Two depth images 410 and 420 of the ToF camera 200 corresponding to the intensity images 310 and 320 shown. The upper depth image 410 is captured by the ToF camera 200 without the cover 240, while the lower depth image 420 is captured by the ToF camera with the cover 240 installed. As can be seen from the depth images 410 and 420, the unwanted reflected light 241 from the cover 240 affects the depth measurement of the ToF camera 200.

[0038] Return to Figure 1 , method 100 may allow compensating for light reflections from the cover of the ToF camera in an image of a scene sensed by the ToF camera. Method 100 includes receiving an image of the scene from the ToF camera (102). The image of the scene can be any type of image that can be provided by the ToF camera, such as an original image, an image derived from the original image, an intensity image, an image derived from the intensity image, a depth image, or an image derived from the depth image. For example, an image derived from the original image can be a phase angle image that includes pixels representing phase shifts (phase angles) derived from one or more correlation values of the corresponding pixels of the illumination element. In other examples, an image derived from one of the images listed above can be an image based on errors of the corresponding image and one or more error corrections for the ToF camera. In some examples, an image derived from one of the images listed above can be an image based on a combination of the corresponding image and another image (e.g., a combination of two depth images captured using modulation signals of different frequencies).

[0039] In addition, method 100 includes modifying the image of the scene using a reference image to obtain a compensated image of the scene (104). The pixels of the reference image indicate (represent) reference values that are only related to the light reflections from the cover of the ToF camera. In other words, the pixels of the reference image indicate (essentially) values that are caused only by the light reflections from the cover of the ToF camera and not by light reflections from any other objects (such as the surroundings of the ToF camera). For example, the reference image can be a reference original image, a reference depth image, or a reference intensity image. For example, the pixels of the reference depth image only indicate depth values that are related to / caused by the light reflections from the cover of the ToF camera. Therefore, the reference image characterizes the influence of the light reflections from the cover on the image captured by the ToF camera. Therefore, the reference image allows compensating for the influence of the light reflections from the cover in the image of the scene.

[0040] When modifying the image of the 100 scene, the value indicated by the pixel of the image of the scene is modified by the reference value indicated by the pixel of the reference image. For example, the image of the scene can be modified pixel by pixel using the reference image. In other words, the value indicated by the pixel of the image of the scene is modified by the reference value indicated by the pixel located at the same pixel position in the reference image. For example, the reference value indicated by the pixel of the reference image can be subtracted from the value indicated by the pixel of the image of the scene. However, the proposed concept is not limited to subtracting the reference value from the value indicated by the pixel of the image of the scene. Generally, the pixels of the image of the scene can be modified / adjusted / changed in any suitable way based on the pixels of the reference image.

[0041] In addition, method 100 includes outputting a compensated image (106) of the scene. Similar to what was described above for the received image of the scene, the compensated image of the scene can be, for example, an original image, an image derived from the original image, an intensity image, an image derived from the intensity image, a depth image, or an image derived from the depth image.

[0042] The compensated image of the scene is corrected for the effect of light reflection from the cover of the ToF camera. Thus, the erroneous measurements made by the ToF camera, represented by the pixels of the image of the scene, can be mitigated at least in the compensated image of the scene. In some examples, the compensated image of the scene can be free from the effect of light reflection from the cover of the ToF camera.

[0043] As described above, in some examples, the reference image can be a reference original image. To sense the scene, the ToF camera can use a continuous wave modulation signal to illuminate the scene and generate a measurement signal based on the reflected light from the scene as described above. Thus, according to the general concept of ToF sensing described above, according to the correlation function The original image of the scene is based on the correlation between the continuous wave modulation signal and the measurement signal. To correctly correct the original image of the scene, the reference original image can be selected from a plurality of reference original images based on the phase shift δ between the continuous wave modulation signal and the measurement signal for the correlation. The plurality of reference original images can correspond to different phase shifts. Thus, for each phase shift δ used, an appropriate (corresponding) reference original image can be selected for correcting the light reflection from the cover of the ToF camera.

[0044] Similarly, the reference original image can be selected from a plurality of reference original images based on the frequency of the continuous wave modulation signal. The plurality of reference original images can correspond to different frequencies of the continuous wave modulation signal. Thus, for each frequency of the continuous wave modulation signal, an (corresponding) reference original image can be selected for correcting the light reflection from the cover of the ToF camera.

[0045] For example, if four original images using four different phase shifts are captured by a ToF camera, four reference original images can be provided to correct the original images. If four additional original images using four different phase shifts are captured for another frequency of the continuous wave modulation signal, four additional reference original images can be provided to correct the four additional original images for another frequency of the continuous wave modulation signal. In other words, for each parameter (phase and frequency), a separate reference image can be provided.

[0046] For example, the reference original images can be based on factory calibration. That is, the reference original images can be generated or provided during the production process of the ToF camera. For example, the reference original images can be captured by the ToF camera in a defined production (calibration) environment that does not contain any reflective objects. For example, the ToF camera can sense a white wall such that only the cover of the ToF camera causes reflection. As described above, the reference original images can be saved and subtracted from the subsequently captured scene images. This method is feasible because the light capture element aggregates all incident light into a relevant value (or phase value).

[0047] In some examples, the reference original images can be generated or (e.g., sporadically) updated during the operation of the ToF camera. For example, coded modulation can be used to fix the measurement range of the ToF camera to the cover of the ToF camera in order to characterize the cover of the ToF camera. For coded modulation, a coded modulation signal is used instead of a continuous wave modulation signal for illumination. In the coded modulation signal, the sequence of pulses is changed. In other words, when the continuous wave modulation signal shows an alternating sequence of high and low pulses of equal length (duration), the coded modulation signal shows pulses of variable length. For example, a Kasami code sequence or an m-sequence can be used for the coded modulation signal.

[0048] The result of the coded modulation signal for illumination is that the correlation function only differs from a constant value with respect to the reflected light from a specific distance range originating from the ToF camera. In other words, only the light reflected from an object within a specific distance range causes the value of the correlation function to differ from the constant value. In terms of a mathematical expression, this can be represented as follows:

[0049]

[0050] where c(d) represents the correlation function, d represents the distance from the object reflecting the light to the ToF camera, a represents a constant, f(d) represents the distance correlation function, d min represents the minimum distance from the object reflecting the light to the ToF camera at which the correlation function is sensitive to the reflected light, and d max represents the maximum distance from the object reflecting the light to the ToF camera at which the correlation function is sensitive to the reflected light.

[0051] In other words, compared with a continuous wave modulation signal, for a code modulation signal, the correlation range of the correlation function is limited. The correlation range of the correlation function that is sensitive to the reflected light from the object sensed by the ToF camera defines the measurement range of the ToF camera. That is, the measurement range of the ToF camera corresponds to the correlation range of the correlation function, and the correlation function outputs a distance-related output value for it.

[0052] Accordingly, in some examples, method 100 may include controlling ToF camera 200 to use a code modulation signal for illumination such that, as Figure 5 shown, the measurement range of ToF camera 200 ends immediately after cover 240. That is, a designed code modulation signal is used to achieve a desired code correlation. Accordingly, the raw image captured by ToF camera 200 based on the coded modulation signal only includes pixels indicating values related only to the light reflection from cover 240 of ToF camera 200. In other words, this raw image only characterizes cover 240 of ToF camera 200. Accordingly, this raw image can be used as a reference raw image. Coding modulation can characterize the cover of ToF camera 200 during operation. In addition, a change in the reflectivity of cover 240 (e.g., caused by fingerprints, dirt, scratches, protective films, fog, rain, etc.) can be detected.

[0053] In other examples, a continuous wave modulation signal can be used instead of the coded modulation signal to characterize the cover of the ToF camera. For example, method 100 may include receiving a depth image from the ToF camera and determining the closest reflecting object drawn in the depth image. In addition, method 100 may include determining the exposure time of the ToF camera such that the measurement range of the ToF camera ends before the closest reflecting object. The power of the reflected light decreases with the distance between the reflecting object and the ToF camera. Accordingly, by adjusting the exposure time of the ToF camera, only the reflected light from the object showing the maximum distance related to the exposure time of the ToF camera will contribute to the measurement signal. The reflected light from objects showing a greater distance will only be significant in the background noise of the light capture element. Accordingly, by appropriately setting the exposure time of the ToF camera, it can be ensured that the reflected light from the closest reflecting object will only contribute to the background noise of the light capture element. Accordingly, method 100 may further include controlling the ToF camera to capture an auxiliary raw image using the determined (set) exposure time. The auxiliary raw image can accordingly be used as a reference image since it only includes pixels indicating values caused only by the cover of the ToF camera.

[0054] In a further example, method 100 may include receiving a depth image from a ToF camera and determining whether any reflective object having a distance greater than the cover of the ToF camera is drawn in the depth image. If no reflective object having a distance greater than the cover is drawn in the depth image, method 100 may include controlling the ToF camera to capture an auxiliary raw image. The auxiliary raw image may accordingly be used as a reference image because it only includes pixels indicating values caused only by the cover of the ToF camera.

[0055] In some examples, the reference image may be a reference depth image as described above. To generate or update the reference depth image, method 100 may, for example, include receiving a depth image from a ToF camera and determining whether any reflective object having a distance greater than the cover is drawn in the depth image. If no reflective object having a distance greater than the cover is drawn in the depth image, the depth image may be used as the reference depth image because it only includes pixels indicating values caused only by the cover of the ToF camera.

[0056] Alternatively, in some examples, the reference image may be a reference intensity image (e.g., a grayscale image). The reference intensity image may be used to correct the intensity image of the ToF and other images of the ToF camera, such as the raw image. For example, if the light capture element of the ToF camera uses PMD to detect reflected light, during normal operation, the gates of the PMD are modulated such that the charges caused by the incident light are separated (e.g., collected in two separate capacitors). In grayscale mode, all charges are added together (e.g., collected in one capacitor). A scaling factor may be used to account for the different operating principles. Thus, modifying the phase image of the 104 scene using the reference intensity image may include scaling the reference values indicated by the pixels of the reference intensity image by the scaling factor to obtain a scaled reference intensity image. Additionally, modifying the phase image of the 104 scene using the reference intensity image may include modifying the image of the scene using the scaled reference intensity image in a similar manner as above to obtain a compensated image.

[0057] In some examples, the exposure times of the scene image and the reference image may be equal. In other words, the exposure times used to capture the scene image and the reference image may be the same.

[0058] Alternatively, the exposure times for capturing the scene image and the reference image can be different. The different exposure times of the scene image and the reference image can be compensated by a scaling factor. For example, if the exposure times of the scene image and the reference image are different, modifying the scene image 104 using the reference image can include scaling the reference values indicated by the pixels of the reference image by the scaling factor to obtain a scaled reference image. The scaling factor is based on the ratio of the exposure times of the scene image and the reference image (e.g., the same as or proportional to it). Additionally, modifying the image of the scene 104 using the reference image can include modifying the image of the scene using the scaled reference image as described above to obtain a compensated image of the scene. Thus, an image of a scene captured by a ToF camera using a first exposure time can be corrected using a reference image captured using a different second exposure time. For example, this can allow for omitting the saving of reference images for each exposure time supported by the ToF camera.

[0059] In the previous description, it was described to correct the light reflection of the cover from the ToF camera at the image level. However, the proposed concept is not limited to error correction at the image level. In some examples, the compensation of the light reflection of the cover from the ToF camera can be performed at the charge level of the light capture element. This will be described in more detail in conjunction with Figure 6 more detail.

[0060] Figure 6 A flowchart of a method 600 for compensating the light reflection of the cover from the ToF camera is shown. As described above (e.g., Figure 2 ), the light capture element of the ToF camera is covered by a cover. In Figure 6 the example, the light capture element includes an array of PMDs such that each PMD measures a small portion of the light arriving at the light capture element (e.g., each pixel of the light capture element can include a PMD). Such PMDs are known, and thus the structure of the PMD will not be described in detail in this disclosure. Each PMD separates the charge generated by the light arriving at the PMD based on the modulation signal for illumination such that (at least) two charge values are provided for the respective PMD (i.e., each PMD of the PMD array).

[0061] Method 600 includes receiving (602) the charge values of the PMDs. For example, the charge values of the PMDs can be represented by analog or digital values provided by the PMDs or connected circuitry (e.g., an analog-to-digital converter, ADC).

[0062] Similar to what was described above for the image level, method 600 also includes modifying (604) the charge value using a reference (charge) value that is only related to the light reflection from the cover of the ToF camera to obtain a compensated charge value. In other words, the reference value is (substantially) only caused by the light reflection from the cover of the ToF camera and not by the light reflection from any other object (such as an object in the surrounding environment of the ToF camera). Therefore, the reference value characterizes the influence of the light reflection from the cover on the charge generated by the PMD. Thus, the reference value allows compensating for the influence of the light reflection from the cover in the charge generated by the PMD.

[0063] For example, the charge value of each PMD can be modified using the reference value. In some examples, a separate reference value can be provided for each PMD. In other examples, separate reference values can be provided for different subsets of the PMD array. In further examples, the same reference value can be provided for all PMDs.

[0064] For example, the reference value can be subtracted from the charge value of the PMD. However, the proposed concept is not limited to subtracting the reference value from the charge value of the PMD. Generally, the charge value of the PMD can be modified / adjusted / changed in any suitable way based on the reference value.

[0065] In addition, method 600 also includes outputting (606) the compensated charge value. For example, the original image can be determined based on the compensated charge value.

[0066] Similar to the image level described above, the compensated charge value is corrected for the influence of the light reflection from the cover of the ToF camera. Thus, the incorrect measurement performed by the ToF camera represented by the charge value of the PMD can be alleviated at least in the compensated charge value. In some examples, the compensated charge value can be free from the influence of the light reflection from the cover of the ToF camera.

[0067] The reference value used to modify the charge value of the PMD can be determined similar to what was described above for the reference image. For example, if no object with a distance greater than the reflective object of the cover is drawn in the depth image of the ToF camera, the ToF camera can be controlled to capture an auxiliary image. The charge values of the PMD obtained when capturing the auxiliary image can be used as the reference value because they are only caused by the cover of the ToF camera. Alternatively, the charge values of the PMD obtained when capturing the depth image can be used as the reference value because they are only caused by the cover of the ToF camera. Similarly, the charge values of the PMD obtained when capturing an image of the coded modulation signal that basically limits the measurement range of the ToF camera to the cover as described above can be used as the reference value.

[0068] Scaling can also be used to compensate for different exposure times of the reference value and the charge value of the PMD. That is, modifying (604) the charge value of the PMD using the reference value can include scaling the reference value by a scaling factor to obtain a scaled reference value. For example, the scaling factor can be based on the ratio of the exposure time used to generate the reference value to the exposure time used to generate the charge value of the PMD (e.g., the same as or proportional to it). Additionally, modifying (604) the charge value of the PMD using the reference value can include modifying the charge value of the PMD using the scaled reference value as described above to obtain a compensated charge value.

[0069] Furthermore, similar to what was described above for the image level, scaling can be used to correct the charge value of the PMD using the reference value obtained when capturing an intensity (grayscale) image.

[0070] In Figure 7 An example of a device 700 for compensating for light reflection from a cover of a ToF camera in accordance with the proposed concept is further shown. The device 700 includes a processing circuit 720. For example, the processing circuit 720 can be a single dedicated processor, a single shared processor, or multiple separate processors (some or all of which may be shared), digital signal processor (DSP) hardware, an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA). The processing circuit 720 can optionally be coupled to, for example, a read-only memory (ROM), a random-access memory (RAM), and / or a non-volatile memory for storing software. The device 700 can also include other conventional and / or custom hardware.

[0071] The device 700 includes an input circuit 710 configured to receive input data 701 representing a scene image from the ToF camera or the charge value of the photon mixing device of the ToF camera. The processing circuit 720 processes the input data 701 in accordance with the concept described above for compensating for light reflection from the cover of the ToF camera. Accordingly, the output circuit 730 of the device outputs output data 702 representing a compensated image of the scene or a compensated charge value.

[0072] For example, the functionality of the device 700 can be implemented in an application processor coupled to a ToF camera module that provides an image or in the ToF camera module itself.

[0073] The examples described herein can be summarized as follows:

[0074] Some examples relate to a method for compensating for light reflection from a cover of a ToF camera in an image of a scene sensed by the ToF camera. The method includes receiving an image of the scene from the ToF camera. Additionally, the method includes modifying the image of the scene using a reference image to obtain a compensated image of the scene. The pixels of the reference image indicate reference values related only to the light reflection from the cover of the ToF camera. Additionally, the method includes outputting the compensated image.

[0075] According to some examples, an illumination element of a ToF camera for illuminating a scene and a light capture element of the ToF camera for receiving reflected light from the scene are arranged in a common cavity covered by a cover.

[0076] In some examples, an image of the scene is one of an original image, an image derived from the original image, an intensity image, an image derived from the intensity image, a depth image, or an image derived from the depth image.

[0077] According to some examples, the reference image is a reference original image.

[0078] In some examples, the ToF camera uses a continuous wave modulation signal to illuminate the scene and generates a measurement signal based on the reflected light from the scene. An image of the scene is an original image according to a correlation function based on the correlation between the continuous wave modulation signal and the measurement signal. The reference original image is selected from a plurality of reference original images based on the phase shift between the continuous wave modulation signal and the measurement signal used for the correlation.

[0079] According to some examples, the ToF camera uses a continuous wave modulation signal to illuminate the scene. The reference original image is selected from a plurality of reference original images based on the frequency of the continuous wave modulation signal.

[0080] In some examples, the reference original image is based on factory calibration.

[0081] According to some examples, the method further includes: controlling the ToF camera to use an encoded modulation signal for illumination such that the measurement range of the ToF camera ends immediately after the cover. The original image captured by the ToF camera based on the encoded modulation signal is used as the reference original image.

[0082] In some examples, the method further includes: receiving a depth image from the ToF camera and determining the closest reflected object drawn in the depth image. Additionally, the method includes: determining an exposure time of the ToF camera such that the measurement range of the ToF camera ends before the closest reflected object. Additionally, the method includes: controlling the ToF camera to use the exposure time to capture an auxiliary original image, where the auxiliary original image is used as the reference image.

[0083] According to some examples, the method further includes: receiving a depth image from the ToF camera and determining whether any reflected object with a distance greater than the cover is drawn in the depth image. If no reflected object with a distance greater than the cover is drawn in the depth image, the method further includes controlling the ToF camera to capture an auxiliary original image, where the auxiliary original image is used as the reference image.

[0084] In some examples, the reference image is a reference depth image.

[0085] According to some examples, the method further includes: receiving a depth image from a ToF camera and determining whether any reflective object with a distance greater than the lid is drawn in the depth image. If no reflective object with a distance greater than the lid is drawn in the depth image, the depth image is used as a reference depth image.

[0086] In some examples, the reference image is a reference intensity image.

[0087] According to some examples, modifying the phase image of a scene using a reference intensity image includes: scaling a reference value indicated by a pixel of the reference intensity image by a scaling factor to obtain a scaled reference intensity image. Additionally, modifying the phase image of a scene using a reference intensity image includes: modifying the image of the scene using the scaled reference intensity image to obtain a compensated image.

[0088] In some examples, the exposure times of the image of the scene and the reference image are equal.

[0089] If the exposure times of the image of the scene and the reference image are different, then according to some examples, modifying the image of the scene using the reference image includes scaling a reference value indicated by a pixel of the reference image by a scaling factor to obtain a scaled reference image. The scaling factor is based on the ratio of the exposure times of the image of the scene and the reference image. Additionally, modifying the image of the scene using the reference image includes modifying the image of the scene using the scaled reference image to obtain a compensated image.

[0090] A further example relates to a device for compensating for light reflection from a lid of a ToF camera in an image of a scene sensed by the ToF camera. The device includes an input circuit configured to receive a phase image of the scene from the ToF camera. Additionally, the device includes a processing circuit configured to modify the image of the scene using a reference image to obtain a compensated image of the scene. The pixels of the reference image indicate reference values related only to the light reflection from the lid of the ToF camera. The device further includes an output circuit configured to output the compensated image.

[0091] An even further example relates to another device for compensating for light reflection from a lid of a ToF camera in an image of a scene sensed by the ToF camera. The device includes means for receiving a phase image of the scene from the ToF camera. Additionally, the device includes means for modifying the image of the scene using a reference image to obtain a compensated image of the scene. The pixels of the reference image indicate reference values related only to the light reflection from the lid of the ToF camera. The device further includes means for outputting the compensated image.

[0092] Other examples relate to further methods for compensating for light reflections from the cover of a ToF camera. The light capture element of the ToF camera is covered by a cover and includes an array of photon mixing devices. Each photon mixing device separates the charge generated by the light arriving at the photon mixing device in order to provide two charge values for the corresponding photon mixing device. The method includes receiving the charge values of the photon mixing devices. Additionally, the method includes modifying the charge values using a reference value that is only related to the light reflection from the cover of the ToF camera to obtain compensated charge values. The method further includes outputting the compensated charge values.

[0093] Further other examples relate to a device for compensating for light reflections from the cover of a ToF camera. The light capture element of the ToF camera is covered by a cover and includes an array of photon mixing devices. Each photon mixing device separates the charge generated by the light arriving at the photon mixing device such that two charge values are provided for the corresponding photon mixing device. The device includes an input circuit configured to receive the charge values of the photon mixing devices. Additionally, the device includes a processing circuit configured to modify the charge values using a reference value that is only related to the light reflection from the cover of the ToF camera to obtain compensated charge values. The device further includes an output circuit configured to output the compensated charge values.

[0094] Further examples relate to a device for compensating for light reflections from the cover of a ToF camera. The light capture element of the ToF camera is covered by a cover and includes an array of photon mixing devices. Each photon mixing device separates the charge generated by the light arriving at the photon mixing device such that two charge values are provided for the corresponding photon mixing device. The device includes means for receiving the charge values of the photon mixing devices. Additionally, the device includes means for modifying the charge values using a reference value that is only related to the light reflection from the cover of the ToF camera to obtain compensated charge values. The device further includes means for outputting the compensated charge values.

[0095] An example relates to a non-transitory machine-readable medium having a program stored thereon, which, when executed on a processor or programmable hardware, has program code for compensating for light reflections from the cover of a ToF camera as described herein.

[0096] Other examples relate to a program which, when executed on a processor or programmable hardware, has program code for performing any of the methods for compensating for light reflections from the cover of a ToF camera as described herein.

[0097] Examples according to the proposed concept may allow for cover glass error correction of a ToF camera (e.g., using continuous wave or coded modulation measurements).

[0098] The description and drawings merely illustrate the principles of the present disclosure. In addition, all examples mentioned herein are specifically for illustrative purposes only to assist the reader in understanding the principles of the present disclosure and the concepts contributed by the inventors to promote the development of the art. All statements in this document that detail the principles, aspects, and examples of the present disclosure and their specific examples are intended to cover their equivalents.

[0099] For example, a block diagram may illustrate a high-level circuit diagram implementing the principles of the present disclosure. Similarly, a flowchart, a state transition diagram, pseudocode, etc. may represent various processes, operations, or steps, which may be substantially represented in a computer-readable medium and executed by a computer or a processor, regardless of whether such a computer or a processor is explicitly shown. The methods disclosed in the description or claims may be implemented by a device having means for performing each corresponding action of these methods.

[0100] It should be understood that, unless otherwise explicitly or implicitly stated (e.g., for technical reasons), the disclosure of multiple actions, processes, operations, steps, or functions in the description or claims should not be construed as being in a particular order. Therefore, the disclosure of multiple actions or functions does not limit them to a particular order, unless for technical reasons, these actions or functions cannot be interchanged. In addition, in some examples, a single action, function, process, operation, or step may include or may be decomposed into multiple sub-actions, sub-functions, sub-processes, sub-operations, or sub-steps. Such sub-actions may be included in the disclosure of such a single action or as part thereof, unless specifically excluded.

[0101] In addition, the following claims are hereby incorporated into the detailed description, where each claim may stand alone as a separate example. Although each claim may stand alone as a separate example, it should be noted that although a dependent claim may refer to a particular combination with one or more other claims in the claims, other examples may also include a combination of the subject matter of such a dependent claim with claims that are dependent on each other or independent. Such combinations are explicitly proposed herein unless it is stated that a particular combination is not intended. In addition, it is also intended to include the features of claims that are dependent on any other independent claim, even if the claim is not directly dependent on that independent claim.

Claims

1. A method (100) for compensating for light reflection in an image of a scene sensed by a time-of-flight camera, the light reflection being from a cover of the time-of-flight camera, the method (100) comprising: Receiving (102) an image of the scene from the time-of-flight camera; Modifying (104) the image of the scene using a reference original image to obtain a compensated image of the scene, wherein pixels of the reference original image indicate reference values related only to the light reflection from the cover of the time-of-flight camera; Outputting (106) the compensated image; and Controlling the time-of-flight camera to illuminate using an encoded modulation signal such that a measurement range of the time-of-flight camera ends immediately after the cover, wherein an original image captured by the time-of-flight camera based on the encoded modulation signal is used as the reference original image.

2. The method according to claim 1, wherein an illumination element of the time-of-flight camera for illuminating the scene and a light capture element of the time-of-flight camera for receiving reflected light from the scene are arranged in a common cavity covered by the cover.

3. The method according to claim 1 or 2, wherein the image of the scene is one of an original image, an image derived from the original image, an intensity image, an image derived from the intensity image, a depth image, or an image derived from the depth image.

4. The method according to claim 1, wherein the time-of-flight camera uses a continuous wave modulation signal to illuminate the scene and generates a measurement signal based on the reflected light from the scene, wherein the image of the scene is an original image based on a correlation between the continuous wave modulation signal and the measurement signal according to a correlation function, and wherein the reference original image is selected from a plurality of reference original images based on a phase shift between the continuous wave modulation signal and the measurement signal used for the correlation.

5. The method according to claim 1 or 4, wherein the time-of-flight camera uses a continuous wave modulation signal to illuminate the scene, and wherein the reference original image is selected from a plurality of reference original images based on the frequency of the continuous wave modulation signal.

6. The method according to claim 1, wherein an exposure time of the image of the scene and the reference original image is equal.

7. The method according to claim 1, wherein if an exposure time of the image of the scene and the reference original image is different, modifying (104) the image of the scene using the reference original image comprises: Scaling the reference values indicated by the pixels of the reference original image by a scaling factor to obtain a scaled reference image, wherein the scaling factor is based on a ratio of the exposure times of the image of the scene and the reference image; And Modifying the image of the scene using the scaled reference image to obtain the compensated image.

8. An apparatus (700) for compensating for light reflection in an image of a scene sensed by a time-of-flight camera, the light reflection being from a cover of the time-of-flight camera, the apparatus (700) comprising: An input circuit (710) configured to receive a phase image of the scene from the time-of-flight camera; A processing circuit (720) configured to modify an image of the scene using a reference original image to obtain a compensated image of the scene, wherein pixels of the reference original image indicate reference values related only to light reflections from the cover of the time-of-flight camera; An output circuit (730) configured to output the compensated image; and wherein the processing circuit (720) is further configured to control the time-of-flight camera to illuminate using an encoded modulation signal such that the measurement range of the time-of-flight camera ends immediately after the cover, and wherein an original image captured by the time-of-flight camera based on the encoded modulation signal is used as the reference original image.

9. An apparatus for compensating for light reflections in an image of a scene sensed by a time-of-flight camera, the light reflections being from a cover of the time-of-flight camera, the apparatus comprising: means for receiving a phase image of the scene from the time-of-flight camera; means for modifying an image of the scene using a reference original image to obtain a compensated image of the scene, wherein pixels of the reference original image indicate reference values related only to light reflections from the cover of the time-of-flight camera; means for outputting the compensated image; and means for controlling the time-of-flight camera to illuminate using an encoded modulation signal such that the measurement range of the time-of-flight camera ends immediately after the cover, and wherein an original image captured by the time-of-flight camera based on the encoded modulation signal is used as the reference original image.

10. A non-transitory machine-readable medium having a program stored thereon, the program having program code for performing the method according to any one of claims 1 to 7 when the program is executed on a processor or programmable hardware.

11. A program having program code for performing the method according to any one of claims 1 to 7 when the program is executed on a processor or programmable hardware.

Images