Time-of-flight camera system and distortion compensation method for time-of-flight camera system

Abstract

A time of flight (TOF) camera device (1) comprises a point projector (10) formed by an array of light emitters (11) and a projection lens (TX), the point projector (10) being configured to project a dot pattern (3) onto a target (2). The TOF camera (1) further comprises an optical sensor (20) formed by an array of sensor pixels (21) and a camera lens (RX), the optical sensor (20) being configured to capture the dot pattern (3) projected onto the target (2). The distortion (d1) of the array of light emitters (11), the transfer function (LTX) of the projection lens (TX) and / or the transfer function (LRX) of the camera lens (RX) are set such that the captured dot pattern (3) is free of optical distortion.

Description

[0001] The present disclosure relates to a time-of-flight (TOF) camera system and to a distortion compensation method for the TOF camera system. Background Art

[0002] A time-of-flight camera is a 3D camera system that is configured to measure the distance to a target, such as an object or a scene, using the time-of-flight TOF method. To this end, the target is illuminated by means of light pulses from a light emitter of the camera, and the sensor element of the camera captures the reflected light from the target. Thus, a processing unit determines for each pixel the time required for the light to reach the target and return again. Therein, the determined time is proportional to the distance to the target. Thus, for each pixel, the camera can provide the distance of the target imaged on the camera. Typically, the illumination is achieved by projecting a dot pattern onto the target, and capturing the reflected dot pattern is achieved by means of an image sensor.

[0003] Conventional TOF cameras include a dot projector for projecting a dot pattern onto a target and an optical sensor for capturing the reflected dot pattern. However, due to the non-ideal lenses used in existing TOF cameras, these systems typically suffer from a large amount of optical distortion in both the dot pattern projected onto the target and the reflected dot pattern received by the optical sensor. Existing methods for overcoming the limitations of distortion include increasing the number of pixels of the optical sensor, and digital pre-distortion. However, these methods have the following disadvantages: excessive energy consumption and high computational effort for analyzing the captured signal.

[0004] Therefore, an object to be achieved is to provide a time of flight camera device that overcomes the limitations of existing solutions. Another object is to provide a distortion compensation method for a TOF camera device.

[0005] These objects are achieved with the subject matter according to the independent claims. Further developments and embodiments are described in the dependent claims. Summary of the invention

[0006] The present disclosure overcomes the above limitations of modern devices by distorting the arrangement of the array of light emitters and / or the transfer function of the optical lenses used in the system to minimize or compensate for optical distortion without any additional components or digital image processing methods. This makes it possible to optimize the number of emitters required according to the number of pixels, thereby reducing costs, increasing optical power throughput, maximizing efficiency, and making the TOF optical system more robust to tolerances.

[0007] In an embodiment, a time-of-flight camera system includes a dot projector formed by an array of light emitters and a projection lens. The dot projector is configured to project a dot pattern onto a target to be determined to a distance thereto. The TOF camera also includes an optical sensor formed by an array of sensor pixels and a camera lens. The optical sensor is configured to capture a dot pattern projected onto the target and reflected from the target back to the TOF camera. Wherein, the distortion of the array of light emitters, the transfer function of the projection lens and / or the transfer function of the camera lens are arranged so that the captured dot pattern has no optical distortion.

[0008] A dot projector can effectively be considered as a structured light source, comprising a device for projecting a grid of dots, i.e. a dot pattern, onto a target, which may be a scene or an object to which the distance and optionally the topographical features are to be determined. The device may comprise an array of light emitters, for example, configured to emit light in the infrared domain so that the illumination using the grid is not dazzling to the human eye. A lens arrangement acting as a projection lens is then configured to direct the light towards the target and create the desired dot pattern on the surface of the target.

[0009] The optical sensor includes an array of sensor pixels, each pixel including a photodiode, for example, configured to generate an electronic light signal based on light captured by the corresponding photodiode during an integration or exposure time. For example, the optical sensor includes an image sensor formed by an array of pixels, such as a grid. As with the dot projector, the optical sensor includes input optics, such as a camera lens, for directing light emitted by the dot projector and reflected from the target onto the sensor pixels. The optical sensor may include other optical elements, such as filters, for rejecting light at undesirable angles of incidence and / or undesirable wavelengths of light.

[0010] The dot projector and the optical sensor may be arranged on a common substrate body such as a chip or wafer substrate. The TOF camera may be a CMOS device that includes an integrated circuit portion for operating the TOF camera. For example, the integrated circuit synchronizes the emission of light from the dot projector and the integration time of the sensor pixel. The integrated circuit may also be configured to determine, for each pixel, the time difference between the emission of light from the dot projector and the generation of a light signal for the corresponding pixel.

[0011] The improved concept optically compensates for any distortion caused by non-ideal optical elements such as projection lenses and camera lenses. For example, the array of light emitters can be distorted so as to reverse the optical distortion introduced by the projection lens and the camera lens. Alternatively, the optical distortion of one of the lenses can be designed via the transfer function of the lens so as to reverse the optical distortion generated by the corresponding other lens. However, alternatively, the array of sensor pixels can be distorted so as to reverse the optical distortion introduced by the projection lens and the camera lens. In addition, a combination of the above features can be used to achieve enhanced suppression of optical distortion, for example by means of distorting the array of light emitters and designing the transfer function of the projection lens, the camera lens, or both lenses. The camera system can be multispectral or monochromatic.

[0012] In an embodiment, each sensor pixel is adapted to detect a point in a pattern of points generated by a light emitter located at a coordinate in the array of light emitters corresponding to the coordinate of the corresponding sensor pixel in the array of sensor pixels. In this way, it can be ensured that the light of each point generated by the point projector via the target reaches the corresponding sensor pixel, wherein the correspondence means that the grid position of the light emitter and the grid position of the corresponding sensor pixel are equal in terms of, for example, row and column numbers. This allows, in an ideal case, to fully control the corresponding positions of the depth points of the 3D system by obtaining exactly the same number of points for each pixel of the sensor. This minimizes the number of emitters required in the emitter array, thereby reducing costs and increasing optical power throughput, which leads to an increased detection range, which is important, for example, in LiDAR systems.

[0013] In an embodiment, the number of light emitters corresponds to the number of sensor pixels. As previously described, ideally, each pixel is configured to detect light from a reflected point in a dot pattern generated by a corresponding one of the light emitters. Conventional methods compensate for distortion by employing an image sensor having a larger number of sensor pixels than the number of light emitters. In this way, all points can be captured, however, a large number of sensor pixels will not detect any point, making the operation of these sensor pixels unnecessary and only resulting in a large amount of energy consumption. If the input and output optics are designed to compensate for optical distortion, as achieved by the improved concept, the optical sensor and the dot projector include an equal number of pixels so that the size and cost of the TOF camera device can be kept to a minimum while maintaining an optimal power consumption budget. For example, the array of light emitters and the array of sensor pixels are equal in the number of rows and columns, with the emitters or sensors arranged at each grid point of the array.

[0014] In an embodiment, the transfer function of the camera lens is set so that the light of each point in the dot pattern is directed to the center of the capture surface of the corresponding sensor pixel. The advantage of this feature is related to tolerance. Tolerance can effectively move the point on the sensor plane. A requirement for reliable TOF measurement is usually to image one point per pixel. Furthermore, in order to be robust to tolerances, the best option is to image each point in the projected dot pattern at the center of each pixel of the optical sensor. However, if a point is imaged by a pixel close to the edge of the photosensitive capture surface, the imaging of that point may fail once tolerance is introduced.

[0015] In an embodiment, the transfer function of the camera lens is arranged such that the distortion of the dot pattern caused by the distortion of the array of light emitters and / or the transfer function of the projection lens is inverted. The camera lens at the input side of the TOF camera can be designed with respect to the transfer function of the camera lens to introduce distortion of the same magnitude but opposite sign to any optical distortion generated by the dot projector (e.g., distortion of the emitter array and optical distortion of the projection lens) into the system compared to an ideal system (i.e., an emitter array without distortion and an ideal projection lens without any optical distortion). Thus, it is ensured that each dot in the projected dot pattern can be captured by a corresponding pixel of the optical sensor for reliable TOF measurements.

[0016] In an embodiment, the distortion of the array of light emitters is arranged so that the dot pattern is free of any distortion. For example, the input optics of the optical sensor do not introduce any distortion. This is achieved by the camera lens behaving similarly to an ideal lens, or by the camera lens being formed by a lens arrangement, wherein, for example, the individual lens elements compensate for each other's distortion or the distortion of other optical elements such as filters. In such a case, a distorted arrangement of light emitters can be used to compensate for the optical distortion introduced by the projection lens. Therefore, the dot pattern projected onto the target is also free of any distortion. For example, the array of light emitters is characterized by barrel distortion, pillow distortion, a combination of barrel distortion and pillow distortion, or a complex type of distortion to compensate for the optical distortion introduced by the projection lens.

[0017] In an embodiment, the transfer function of the projection lens is set so that the dot pattern is free of any distortion. The above-mentioned distortion-free dot pattern on the target can also be achieved or further enhanced by a specially designed transfer function of the projection lens. For example, the transfer function of the projection lens is set to be similar to an ideal lens so that an array of distortion-free light emitters is converted into a projection of a distortion-free dot pattern. Alternatively, the combination of the distortion of the array of light emitters and the lens transfer function customized for this purpose can lead to minimization of any distortion in the projected pattern.

[0018] In an embodiment, the distortion of the array of light emitters is arranged so as to achieve a predetermined distortion of the dot pattern on the target. In contrast to the above-described embodiment in which the dot pattern is generated without any distortion when the input optics of the optical sensor do not introduce any optical distortion, a distorted dot pattern can be generated in which the distortion of the dot pattern matches the optical distortion of the camera lens in magnitude, but has the opposite sign. For example, the camera lens is a non-ideal lens with a transfer function that introduces optical distortion. Therefore, the dot pattern can be generated in such a way that the transfer function of the camera lens accurately inverts the distortion of the dot pattern on the target so that each dot can be captured by a corresponding one of the sensor pixels arranged in, for example, a non-distorted rectangular array. In order to achieve the pre-distortion of the dot pattern, the known transfer function of the projection lens can be used in combination with a specific distortion of the array of light emitters. For example, the array of light emitters is characterized by barrel distortion, pincushion distortion, a combination of barrel distortion and pincushion distortion, or a complex type of distortion, so as to achieve the predetermined distortion of the dot pattern on the target.

[0019] In an embodiment, the transfer function of the projection lens is arranged so as to achieve a predetermined distortion of the dot pattern on the target. The above-mentioned pre-distortion of the dot pattern on the target can also be achieved or further enhanced by a specially designed transfer function of the projection lens. For example, the transfer function of the projection lens is arranged to be similar to a non-ideal lens so that an array of undistorted light emitters is converted into a distorted projection of the dot pattern. Alternatively, the combination of the distortion of the array of light emitters and the lens transfer function customized for this purpose can lead to the achievement of the desired distortion in the enhanced projected pattern.

[0020] In an embodiment, the transfer function of the camera lens is arranged such that a predetermined distortion of the dot pattern on the target is inverted. To ensure that each dot is captured by the correct corresponding pixel in the pixel array, the transfer function of the camera lens introduces an optical distortion of the same magnitude but different sign than the optical distortion of the dot pattern on the target introduced by the transfer function of the non-ideal projection lens and / or the array of distorted light emitters.

[0021] In an embodiment, the light emitter is a coherent light emitter, in particular a vertical cavity surface emitting laser (VCSEL). VCSELs are very energy-efficient light emitters in infrared domains such as NIR or SWIR domains. Therefore, an array of VCSELs can be used to generate a desired dot pattern on the surface of a target. Alternatively, the light emitter can be any other type of coherent light emitter, such as an edge emitter or other type of laser.

[0022] In an embodiment, each of the sensor pixels includes a photodiode, in particular a micro-photodiode. For example, the optical sensor employs an image sensor formed of a silicon-based photodiode having sensitivity in the infrared domain, in particular in the NIR domain. On its photosensitive surface, the photodiode may be coated with a filter layer that primarily or exclusively transmits light at a wavelength corresponding to the light emitted from the light emitter. The micro-photodiode may also support a reduction in the overall size of the image sensor.

[0023] In an embodiment, the projection lens is formed by one of the following: injection molded optics, wafer-level optics, a metalens, a microlens array. In an embodiment, the camera lens is formed by one of the following: injection molded optics, wafer-level optics, a metalens, a microlens array. For example, depending on the specific application and space constraints, both the projection lens and the camera lens can be formed by the above elements. In particular, the metalens can be designed in a straightforward manner to achieve a specific transfer function.

[0024] In an embodiment, the dot projector comprises a single light emitter and further comprises a diffractive optical element DOE configured to generate an array of dots from light received from the single light emitter. The array of light emitters may comprise a single light emitter, the dot pattern being generated from the single light emitter by means of the diffractive optical element. This is particularly relevant for embodiments where low power consumption is absolutely essential, for example for applications in battery-powered devices.

[0025] The above object is further achieved by an electronic device including a TOF camera device according to one of the above embodiments. The electronic device may be a mobile computing device, such as a smart phone, a laptop or a tablet computer, or a wearable device such as a smart watch or smart glasses. In addition, the electronic device may be a sensor unit of a vehicle, in which the TOF camera device is used as a laser radar (LIDAR) system for determining the distance and monitoring the speed of objects and scenes around the vehicle.

[0026] In addition, a distortion compensation method for a time-of-flight camera is provided. The method comprises: providing a dot projector formed by an array of light emitters and a projection lens; and providing an optical sensor formed by an array of sensor pixels and a camera lens. The method further comprises: projecting a dot pattern onto a target by means of the dot projector; and capturing the dot pattern projected onto the target by means of the optical sensor. The distortion of the array of light emitters, the transfer function of the projection lens and / or the transfer function of the camera lens are arranged so that the captured dot pattern has no optical distortion.

[0027] Based on the above-described embodiments of the TOF camera device, other embodiments of the method will become apparent to a skilled person, and vice versa. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The following description of the drawings may further illustrate and explain various aspects of the TOF camera device and the distortion compensation method. Functionally identical or having identical effects of the TOF camera device and parts thereof are represented by the same reference numerals, and identical or substantially identical parts and parts may be described only with reference to the drawings in which they first appear. Their description is not necessarily repeated in consecutive drawings. DETAILED DESCRIPTION

[0029] In the attached picture:

[0030] Figure 1 A schematic diagram showing an exemplary embodiment of a TOF camera device according to an improved concept.

[0031] Figure 2 The working principle of generating and capturing dot patterns is shown.

[0032] Figure 3a An exemplary embodiment of a projection lens employed in a TOF camera device according to an improved concept is shown.

[0033] Figure 3b The distortion of the dot pattern introduced by the projection lens is shown.

[0034] Figure 4a An exemplary embodiment of a camera lens employed in a TOF camera according to an improved concept is shown.

[0035] Figure 4b The distortion of the captured dot pattern introduced by the camera lens is shown.

[0036] 5 to 8 illustrate various embodiments of arrays of light emitters, projection lenses, and camera lenses, and their operating principles; and

[0037] Fig. 9 An exemplary embodiment of an electronic device including a TOF camera device according to an improved concept is shown.

[0038] Figure 1A schematic diagram of an exemplary embodiment of a TOF camera device 1 according to an improved concept is shown. The TOF camera device 1 includes a dot projector 10 and an optical sensor 20 arranged on a common substrate 30, such as a silicon chip substrate. The dot projector 10 includes an array of light emitters 11, which is implemented by means of a grid of vertical cavity surface emitting lasers (VCSELs) arranged on an emitter substrate, for example. The light emitter 11 can alternatively be an edge emitter or any other type of emitter that can be operated to emit coherent light. For example, the light emitted by the light emitter 11 is infrared light, in particular NIR or SWIR light. The dot projector 10 also includes a projection lens TX, which is configured to receive the light emitted by the light emitter 11 and project the light as a dot pattern 3, i.e., as a grid of dots, on the surface of the target 2. The projection lens TX can be a single lens element or an arrangement with multiple lens elements. The projection lens TX can be formed by injection molded optics, wafer-level optics, a super lens, or a microlens array. In the latter case, the number of microlenses may correspond to the number of light emitters 11, so that each of the microlenses is associated with a light emitter. Alternatively, the number of microlenses may be different from the number of light emitters, for example as implemented with a dot projector.

[0039] The projection lens TX may include a coating implementing an optical filter for rejecting light of undesired wavelengths, e.g., wavelengths not emitted by the light emitter 11. The dot projector 10 may also include optical elements, such as optical filters and / or diffractive optical elements DOE such as Bragg gratings. In particular, in embodiments in which the dot projector 10 includes a single light emitter 11, a diffractive optical element DOE may be employed to separate the light from the emitter 11 into a dot pattern.

[0040] The optical sensor 20 comprises an array of sensor pixels 21, which is realized, for example, by means of a grid of photodiodes arranged on a sensor substrate. The sensor pixels 21 may be silicon-based photodiodes, in particular micro-photodiodes, or any other type of light detector operable to detect light at the wavelength of light emitted by the dot projector 10. The optical sensor 20 further comprises a camera lens RX, which is configured to receive light emitted by the light emitter 11 and reflected by the target 2, and is configured to direct the received light to the array of sensor pixels 21. In other words, the camera lens RX is configured to receive the projected dot pattern 3 and provide the dot pattern 3 to the sensor pixels 21. The camera lens RX may be a single lens element or an arrangement of multiple lens elements. The camera lens RX may be formed by injection-molded optics, wafer-level optics, a superlens or a microlens array. In the latter case, the number of microlenses may correspond to the number of sensor pixels 21, so that each of the microlenses is associated with a sensor pixel.

[0041] The camera lens RX may include a coating implementing an optical filter for rejecting light of unwanted wavelengths, e.g. wavelengths not emitted by the light emitter 11. The optical sensor 20 may also include optical elements, such as optical filters, for preventing unwanted light from being detected by the sensor pixels 21.

[0042] The TOF camera device 1 may also include active circuit systems and passive circuit systems for operating the point projector 10 and the optical sensor 20. Specifically, the processing unit may be configured to control the emission of light and control the exposure period of the sensor pixels 21 of the optical sensor 20 to determine the time difference between the emission and reception of the light pulse. The concept of TOF measurement is a well-known concept and is not further described in detail throughout the disclosure. Instead, the following figure shows the relevant aspects of the improved concept in terms of distortion compensation.

[0043] Figure 2 Shows the use of Figure 1 The schematic diagram of FIG. 1 shows an array of light emitters 11, an array of sensor pixels 21, and the working principle of generating and capturing a dot pattern with a projection lens TX and a camera lens RX. The array of light emitters 11 can be a rectangular array as shown, in which case there is no distortion d1 of any arrangement. By defining a coordinate system, the position of each light emitter in the light emitters can be determined by a position vector r V The light emitted by the emitter 11 is transmitted by means of a transfer function L TX The projection lens TX is projected onto the target as a dot pattern 3. Therefore, the position vector r of each point in the dot pattern 3 is W By r W=L TX (r V ). In contrast, for a known dot pattern 3, the array of light emitters 11 is given by Given.

[0044] Similarly, the dot pattern 3 is characterized by the transfer function L RX Therefore, the position vector r of each point in the dot pattern 3 projected onto the corresponding sensor pixel 21 is S By r S =L RX (r W )=L RX (L TX (r V )) is given.

[0045] To illustrate this working principle, the projection lens TX and the camera lens RX are shown as ideal lenses that do not introduce any optical distortion. In other words, the rectangular distortion-free arrangement of the array of light emitters 11 is converted into a distortion-free dot pattern 3 on the target 2, and into a distortion-free dot pattern received by the sensor pixels 21. In other words, each dot in the dot pattern 3 is guided to the center of the photosensitive surface of each sensor pixel 21, as shown. Therein, each dot is generated by a light emitter with specific coordinates, i.e., row and column positions in the array of light emitters 11, and is captured by a corresponding one of the sensor pixels 21 with the same coordinates, i.e., the same column number and row number in the array of sensor pixels 21. However, real lenses do introduce optical distortions, such that a compensation mechanism is required to maintain the full resolution of the TOF measurement.

[0046] Figure 3a An exemplary embodiment of a projection lens TX is shown, in which case the projection lens TX is a continuous arrangement of two lens elements. Also in this case, the projection lens TX can be configured with the aid of a single transfer function L as described above. TX to describe.

[0047] Figure 3b The dot pattern 3 is shown by Figure 3a . If the array of light emitters 11 is a regular grid, i.e. without any distortion d1 as shown in the left panel, the dot pattern 3 on the target 2 will show distortion d2 due to the non-ideal behavior of the projection lens TX. The distorted pattern is shown in the right panel of the figure. In general, the distortion introduced by a real lens is a highly nonlinear effect, Figure 3aThe lens in is shown in the bottom panel. Specifically, for example, a distortion d2 of the dot pattern may cause some dots to be squeezed closer to their neighbors near the center of the array, while dots farther away are pulled apart. This is often referred to as a complex type of distortion. However, other types of distortion, such as barrel or pincushion distortion, are also possible.

[0048] and Figure 3a and Figure 3b similar, Figure 4a and Figure 4b shows the distortion introduced by means of a real non-ideal camera lens RX, as Figure 4a . Given a non-distorted dot pattern as shown in the left panel, the dot pattern 3 on the array of sensor pixels 21 will show a distortion d3 due to the non-ideal behavior of the camera lens RX. The distorted pattern is shown again in the right panel of the figure. For this lens, the distortion d3 of the imaged dot pattern results in the dots being gradually pulled apart with their distance, e.g. what is often referred to as barrel distortion, usually observed for concave spherical lenses. Alternatively, pincushion distortion or other complex types of distortion can be achieved, for example, by adopting a convex spherical lens. Distortions on the input side may result in the fact that not all dots in the dot pattern 3 reach the corresponding sensor pixel in the center of the photosensitive surface, or even not at all.

[0049] FIG5 shows a first exemplary embodiment of a distortion compensation mechanism for a TOF camera 1. In this embodiment, as shown in panel (a), the array of light emitters 11 is characterized by a radially symmetric barrel-type distortion d1. This means that the array can still be defined by rows and columns, however, the lines bulge outwards at the center and are therefore not parallel to each other. The type and degree of distortion d1 is arranged so that the projection lens TX, for example Figure 3a and Figure 3b The optical distortion introduced by the exemplary projection lens TX in is compensated. In other words, the distortion d1 is of the same magnitude as the distortion introduced by the projection lens TX, but of opposite sign. This results in a dot pattern 3 without any distortion as shown in panel (b). In other words, the dot pattern 3 on the target 2 can be described by a rectangular array of dots arranged in rows and columns parallel to each other. Furthermore, the present embodiment assumes an ideal camera lens RX that does not introduce any optical distortion. Therefore, the non-distorted dot pattern 3 is projected onto the array of sensor pixels 21 so that each dot hits the center of the corresponding respective sensor pixel 21, which correspondence is again defined by having the same row and column indexes (or being mirrored by an axis or point). This is shown by the circle representing the dots of the imaged dot pattern being concentric with the square sensor pixels 21.

[0050] Fig. 6 shows a first exemplary embodiment of a distortion compensation mechanism for the TOF camera 1 of Fig. 5, in the case where a real lens is employed as the camera lens RX. In other words, referring to panel (c), the camera lens RX introduces a distortion d3 which manifests itself as a dot pattern not being projected onto the sensor pixel 21 in such a way that each dot hits the center of the corresponding sensor pixel 21. This is an undesirable behavior and serves to illustrate the fact that both the distortion introduced by the projection lens TX and the distortion introduced by the camera lens RX need to be compensated in order to achieve the desired result in a real system.

[0051] FIG7 shows a second exemplary embodiment of a distortion compensation mechanism for a TOF camera 1. Compared to the first embodiment of FIG5 and the first embodiment of FIG6 , in this embodiment the distortion d1 of the array of light emitters 11 is arranged such that not only the optical distortion due to the projection lens TX but also the optical distortion due to the camera lens RX is compensated. This firstly leads to the fact that, compared to FIG5 , as shown in panel (b), the dot pattern 3 is now also subjected to a distortion d2, in this case a pincushion distortion, which is then compensated by the optical distortion of the camera lens RX so that the desired result appears again, namely that each dot of the dot pattern hits a corresponding one of the pixel sensors 21, as shown in panel (c). The type and degree of the distortion d1 of the array of light emitters 11 and the type and degree of the predetermined distortion d2 of the dot pattern depend on the transfer function L of the projection lens TX and the camera lens RX. TX , L RX , and they may have different degrees and / or types for different lenses and lens types, so that the imaged dot pattern on the sensor pixel 21 does not have any distortion d3. The first and second embodiments may be used for direct projection systems, where one emitter 11 produces one dot of the dot pattern 3. It may also be used to minimize distortion in a projection lens TX comprising lens elements and diffractive optical elements, where one emitter produces 1xN dots, where N is the total order of the DOE.

[0052] FIG8 shows a third exemplary embodiment of a distortion compensation mechanism for a TOF camera 1. In contrast to the previously described embodiments, in this embodiment the array of light emitters 11 is free of any distortion d1, so that all light emitters 11 are arranged in a rectangular grid characterized by rows and columns parallel to each other. Therefore, in order to achieve zero distortion or at least minimal distortion d3 of the imaged dot pattern on the sensor pixels 21, the transfer function L of the projection lens TX and the camera lens RX is TX , L RXare designed to compensate each other in terms of the optical distortion introduced. For example, the projection lens TX generates a dot pattern 3 on the target characterized by a pincushion-type distortion d2, as shown in panel (b), which is then restored by the camera lens RX, which inverts the distortion d2. In other words, the projection lens TX and the camera lens RX introduce distortions of equal degree but opposite sign. This type of distortion compensation can be achieved with direct projection and meta-optical lenses, which are prone to high levels of distortion due to their architecture. Specifically, this embodiment enables full correction of the distortion of a distorted dot projector, a projection lens TX including a DOE, or an MLA dot projector, or generally any dot projector with distortion.

[0053] Since it is generally undesirable to distort the array of sensor pixels 21, the three degrees of freedom used for distortion compensation include the distortion d1 of the array of light emitters 11, the distortion d2 of the projection lens TX due to its transfer function L TX The distortion caused by the transfer function L RX Therefore, the above embodiments can also be combined to enhance the distortion correction method by adjusting all, ie the distortion of the array of light emitters 11 and the transfer functions of the two lenses TX, RX.

[0054] Fig. 9 An exemplary embodiment of an electronic device 100, such as a smart phone or a smart watch, including a TOF camera system 101 having a TOF camera 1 according to the improved concept is shown. The camera system 101 includes a TOF camera 1 and a control and processing unit 102 electrically coupled to the TOF camera 1. The control and processing unit 102 is configured to operate the TOF camera 1, i.e., control the emission of the dot projector 10 and the detection of the dot pattern 3 on the target 2 via the optical sensor 20, and is configured to receive a light signal from each of the sensor pixels 21 to determine the distance to the target 2 and / or the terrain features of the target 2 via the time difference between the emission and detection of the light pulse.

[0055] For the purpose of familiarizing the reader with the novel aspects of the concept, the embodiments of the TOF camera device 1 and the distortion compensation method disclosed herein have been discussed. Although preferred embodiments have been shown and described, those skilled in the art may make changes, modifications, equivalents and substitutions to the disclosed concepts without necessarily departing from the scope of the claims.

[0056] It should be understood that the present disclosure is not limited to the disclosed embodiments and the contents particularly shown and described above. On the contrary, the features described in the separate dependent claims or in the specification may be advantageously combined. In addition, the scope of the present disclosure includes those variations and modifications, which will be obvious to those skilled in the art and fall within the scope of the appended claims.

[0057] The term "comprising", whenever used in a claim or in the description, does not exclude the corresponding feature or other elements or steps of a process. Where the terms "a" or "an" are used in conjunction with a feature, they do not exclude a plurality of such features. Furthermore, any reference signs in the claims shall not be construed as limiting the scope.

[0058] This patent application claims the priority of German patent application DE 10 2022 116 581.7, the disclosure content of which is incorporated herein by reference.

[0059] Reference numerals

[0060] 1 TOF camera device

[0061] 2 Objectives

[0062] 3 dot pattern

[0063] 10-point projector

[0064] 11 Optical Transmitter

[0065] 20 Optical Sensor

[0066] 21 sensor pixels

[0067] 30 substrate

[0068] 100 Electronic devices

[0069] 101 TOF camera system

[0070] 102 Control and processing unit

[0071] RX camera lens

[0072] TX Projection Lens

[0073] L TX , L RX Transfer Function

[0074] d1, d2, d3 distortion

[0075] r V , r W , r S Position vector

Claims

1. A time-of-flight (TOF) imaging device (1), comprising: a dot projector (10) formed by an array of light emitters (11) and a projection lens (TX), the dot projector (10) being configured to project a dot pattern (3) onto a target (2); as well as an optical sensor (20) formed by an array of sensor pixels (21) and a camera lens (RX), the optical sensor (20) being configured to capture the dot pattern (3) projected onto the target (2); The distortion (d1) of the array of light emitters (11), the transfer function (L TX ) and / or the transfer function (L RX ) is arranged so that the captured dot pattern (3) has no optical distortion; wherein, The distortion (d1) of the array of light emitters (11) is arranged such that the dot pattern (3) is free of any distortion (d2) or such that a predetermined distortion (d2) of the dot pattern (3) on the target (2) is achieved.

2. The TOF camera device (1) according to claim 1, wherein: Each of the sensor pixels (21) is adapted to detect a point in the dot pattern (3) generated by one of the light emitters (11), the one light emitter being located at a coordinate in the array of the light emitters (11) corresponding to a coordinate of a corresponding one of the sensor pixels (21) in the array of the sensor pixels (21).

3. The TOF camera device (1) according to claim 1 or 2, wherein: The number of light emitters (11) corresponds to the number of sensor pixels (21).

4. The TOF camera device (1) according to one of claims 1 to 3, wherein: The transfer function (L RX ) is arranged so that light from each point in the dot pattern (3) is guided to the center of the capture surface of a corresponding one of the sensor pixels (21).

5. The TOF camera device (1) according to one of claims 1 to 4, wherein: The transfer function (L RX ) is arranged such that the distortion (d1) of the array of light emitters (11) and / or the transfer function (L TX ) the distortion (d2) of the dot pattern (3) caused by is reversed.

6. The TOF camera device (1) according to one of claims 1 to 5, wherein: The transfer function (L TX ) is arranged so that the dot pattern (3) does not have any distortion (d2).

7. The TOF camera device (1) according to one of claims 1 to 5, wherein: The transfer function (L TX ) is arranged so as to achieve a predetermined distortion (d2) of the dot pattern (3) on the target (2).

8. The TOF camera device (1) according to one of claims 1 to 7, wherein: The transfer function (L RX ) is arranged so that a predetermined distortion (d2) of the dot pattern (3) on the target (2) is inverted.

9. The TOF camera device (1) according to one of claims 1 to 8, wherein: The optical transmitter (11) is a coherent optical transmitter, in particular a vertical cavity surface emitting laser (VCSEL).

10. The TOF camera device (1) according to one of claims 1 to 9, wherein: Each of the sensor pixels (21) comprises a photodiode, in particular a micro-photodiode.

11. The TOF camera device (1) according to one of claims 1 to 10, wherein: The projection lens (TX) is formed by one of the following: injection molded optics, wafer level optics, metalens, microlens array.

12. The TOF camera device (1) according to one of claims 1 to 11, wherein: The camera lens (RX) is formed by one of the following: injection molded optics, wafer level optics, superlens, microlens array.

13. The TOF camera device (1) according to one of claims 1 to 12, wherein: The dot projector (10) includes a single light emitter (11) and further includes a diffractive optical element (DOE) configured to generate a dot array based on light received from the single light emitter.

14. An electronic device (100) comprising the TOF camera device (1) according to one of claims 1 to 13.

15. A method for compensating distortion for a time-of-flight camera device (1), the method comprising: Providing a dot projector (10) formed by an array of light emitters (11) and a projection lens (TX); Providing an optical sensor (20) formed by an array of sensor pixels (21) and a camera lens (RX); projecting the dot pattern (3) onto the target (2) by means of the dot projector (10); as well as capturing the dot pattern (3) projected onto the target (2) by means of the optical sensor (20); The distortion (d1) of the array of light emitters (11), the transfer function (L TX ) and / or the transfer function (L RX ) is arranged so that the captured dot pattern (3) is free from optical distortion, wherein, The distortion (d1) of the array of light emitters (11) is arranged such that the dot pattern (3) is free of any distortion (d2) or such that a predetermined distortion (d2) of the dot pattern (3) on the target (2) is achieved.

Images