Upward 3D sensing

Abstract

Embodiments of the present technology provide apparatus and methods for generating a three-dimensional (3D) representation of a scene (also known as 3D sensing) using a time-of-flight imaging system. [Solution] In particular, the present technology provides an apparatus comprising a time-of-flight imaging camera system that emits illumination having a spatially non-uniform intensity across a sensor's field of view and is moved across at least a portion of the sensor's field of view using an actuation mechanism. [Selected Figure] Figure 1

Description

Technical Field

[0001] This application generally relates to an apparatus and method for generating a three-dimensional (3D) representation of a scene (also known as 3D sensing), and more particularly for generating a 3D representation using a time-of-flight imaging system.

Summary of the Invention

[0002] In a first approach of the present technology, an apparatus for generating a three-dimensional representation of a scene is provided. The apparatus includes a multi-pixel sensor and a light source, and a time-of-flight (ToF) imaging camera system arranged to emit illumination having spatially non-uniform intensity across the field of view of the sensor, and an actuating mechanism for enabling the generation of the representation by moving the illumination across at least a portion of the field of view of the sensor. This can be achieved without moving the sensor.

[0003] The non-uniform illumination can be any form of illumination such as a beam of light, a pattern of light, a striped pattern of light, a dot pattern of light, etc. These are merely exemplary types of illumination and are to be understood as non-limiting.

[0004] The apparatus can be (or can be included in) any of, for example, a smartphone, a mobile computing device, a laptop, a tablet computing device, a security system, a game system, an augmented reality system, an augmented reality device, a wearable device, a drone, an aircraft, a spacecraft, a vehicle, a self-driving vehicle, a robotic device, a consumer electronics device, a domestic device, and a home automation device.

[0005] In a second approach of the present technology, a method for generating a three-dimensional representation of a scene is provided, the method comprising emitting illumination having spatially non-uniform intensity across the field of view of a sensor used to receive reflected light, using a time-of-flight (ToF) imaging camera system, and enabling the generation of the representation by moving the illumination across at least a portion of the field of view of the sensor using an actuating mechanism.

[0006] The devices described herein can be used in a number of technologies or purposes (and their associated devices or systems) such as 3D sensing, depth mapping, aerial surveying, terrestrial surveying, surveying in space or from space, hydrographic surveying, underwater surveying, scene detection, collision warning, security, face recognition, augmented reality, advanced driver assistance systems in vehicles, autonomous vehicles, gaming, gesture control / recognition, robotic device control, contactless technology, and home automation. It should be understood that this is a non-exhaustive listing of exemplary technologies that can benefit from the use of this device.

[0007] In a related approach of the present technology, a non-transitory data carrier holding processor control code for implementing any of the methods described herein is provided.

[0008] Preferred features are described in the accompanying dependent claims.

[0009] As will be appreciated, the present technology may be embodied as a system, method, or computer program product. Accordingly, the present technology may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[0010] Furthermore, the present technology can take the form of a computer program product embodied in a computer-readable medium having computer-readable program code embodied therein. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable medium can be, by way of example and not limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of these.

[0011] The computer program code for carrying out operations of the present technology can be written in any combination of one or more programming languages, such as object-oriented programming languages and conventional procedural programming languages. The code components may be embodied as procedures, methods, etc., or may include sub-components that can take the form of instructions or sequences of instructions at any level of abstraction, from direct machine instructions of a native instruction set to higher-level compiled or interpreted language constructs.

[0012] Embodiments of the present technology also provide a non-transitory data carrier that, when executed by a processor, holds code that causes the processor to perform any of the methods described herein.

[0013] The present technology further provides processor control code for implementing the above method, for example, on a general-purpose computer system or on a digital signal processor (DSP). The present technology also provides a carrier that, when executed, holds processor control code for implementing any of the above methods, particularly on a non-transitory data carrier. The code can be provided on a carrier such as a programmed memory such as a disk, a microprocessor, a CD- or DVD-ROM, a non-volatile memory (e.g., flash) or a read-only memory (firmware), or on a data carrier such as an optical or electrical signal carrier. The code (and / or data) for implementing embodiments of the technology described herein can be source, object or executable code in a conventional (interpreted or compiled) programming language such as C, or assembly code, code for configuring or controlling an ASIC (application-specific integrated circuit) or an FPGA (field-programmable gate array), or code for a hardware description language such as Verilog (RTM) or VHDL (very high speed integrated circuit hardware description language). As will be appreciated by those skilled in the art, such code and / or data can be distributed among a plurality of coupled components that communicate with each other. The present technology can comprise a controller including a microprocessor, a working memory and a program memory coupled to one or more of the components of the system.

[0014] All or part of the logical method according to embodiments of the present technology can be appropriately embodied in a logic device comprising logic elements for performing the steps of the above method, and it should also be understood that such logic elements can comprise components such as logic gates in a programmable logic array or an application-specific integrated circuit. Such a logic configuration can be further embodied when activating elements for temporarily or permanently establishing a logic structure in such an array or circuit using a virtual hardware description language that can be stored or transmitted using, for example, a fixed or transmissible carrier medium.

[0015] In one embodiment, the present technology may be implemented in the form of a data carrier having functional data therein, and when the functional data is loaded into a computer system or network and operated thereby, the computer system is enabled to execute all steps of the above method and includes a functional computer data structure.

[0016] With reference to the accompanying drawings, the implementation of the present technology will be described herein only by way of example.

Brief Description of the Drawings

[0017]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

[0018] Generally, embodiments of the present technology provide an apparatus and method (also known as 3D sensing) for generating a three-dimensional (3D) representation of a scene using a time-of-flight imaging system. In particular, the present technology provides an apparatus comprising a time-of-flight imaging camera system that emits illumination having spatially non-uniform intensity across the field of view of a sensor and is moved across at least a portion of the field of view of the sensor using an actuating mechanism.

[0019] Time-of-flight (ToF) camera systems are known for long-distance measurements and are used, for example, to measure distances in building surveys. Time-of-flight camera systems operate by estimating the time it takes for a pulse of light to propagate from a transmitter to a sensor / receiver / detector. The estimated value of the time (in seconds) can be converted to distance (in meters) simply by multiplying that time by half the speed of light (i.e., 1.5×10 8 ms -1 ). The timing of this system preferably needs to be highly accurate and precise with at least nanosecond resolution.

[0020] To avoid disturbing the imaged scene (which may also be imaged with a visible light camera), invisible wavelengths can be used in ToF camera systems. The near-infrared (NIR) band (wavelength 750 nm to 1.4 μm) is usually selected because small (portable) lasers with good resolution capabilities can be utilized without absorption lines.

[0021] There are several different mechanisms for detecting time-of-flight, but the most practical 2D sensors operate on the modulation principle where multiple pulses of light are emitted and the phase shift of the received light is measured. The modulation frequency is typically in the range of 1 to 100 MHz (i.e., 10 ns to 1 μs), which also determines the maximum measurable range (due to the inability to distinguish false signals). Modulation in the range of 1 to 100 MHz corresponds to maximum ranges of approximately 150 m to 1.5 m, respectively.

[0022] It is possible to design a camera with required-level performance under ideal conditions, but the practical signal-to-noise level reduces the available performance, especially from the perspectives of depth range and depth resolution. A typical problem is that other lighting sources, and especially direct sunlight, increase the background illumination, which can crowd the time-of-flight signal and make it difficult (noisy) or impossible (not detectable at all) to detect the time of flight. Usually, due to both power constraints (devices typically operate in an instantaneous power range of 1 to 8 W) and strict limitations on the optical power output from the laser to prevent user hazards, the output power from the lighting source cannot be increased.

[0023] Certain applications require high-precision depth measurement at long distances. For example, artificial augmented reality systems and collision detection systems for vehicle or robot devices may require high-precision depth measurement over a long range, such as a depth resolution of 10 cm at a distance of 10 m from an imaging system.

[0024] Indirect time-of-flight cameras typically attempt to illuminate the field of view and can have a field of view angle of 60×45 degrees. This can be achieved using a VCSEL array (vertical-cavity surface-emitting laser array) as the light source and a diffuser plate to ensure uniform spread of the illumination across the field of view. Considering the electrical and optical power constraints of a standard ToF camera system, this means that the good-quality depth sensing ability is limited to a distance of about 4 meters, so that no useful depth information is returned when the subject is, for example, 6 meters away.

[0025] Therefore, the applicant has recognized the need for an improved mechanism for performing 3D sensing at long distances using ToF.

[0026] A PMD Flexx ToF system with a VCSEL array was tested to identify how the resolution of a ToF-based 3D sensing system can be improved for long distances. The ToF system was set up to image a person who extends their left thumb while standing at least 5 meters away from the system and holds a cube of 10 cm or less in their right hand. The system was set to take 5 frames per second (fps) for all tests. The tests tried to determine whether, at various distances using uniform and non-uniform illumination, (i) the general body shape of the person, (ii) the shape of the left hand and the individual fingers of the left hand, and (iii) the shape of the cube could be clearly identified.

[0027] Figure 6(A) shows an image captured by the ToF imaging system when the ToF system emits uniform illumination and the person being imaged is standing within 5.2 meters from the camera. This shows that the entire scene is sufficiently illuminated by the ToF imaging system. Figure 6(B) shows an image captured by the ToF imaging system when the ToF system emits (spatially) non-uniform illumination. The non-uniform illumination was achieved by removing the diffuser plate from the ToF system. In Figure 6(B), the person being imaged was standing within 6 meters from the camera. This shows that the center of the scene is more sufficiently illuminated than the edges of the scene (i.e., the illumination of the central scene was increased), which in turn shows that the accuracy and / or range of the depth information at the center of the scene is improved.

[0028] Figs. 7(A) and 7(B) respectively show enlarged views of the images shown in Figs. 6(A) and 6(B). (i) Regarding identifying the general body shape of a person, Fig. 7(A) (uniform illumination) shows a rough body shape and poor depth discrimination, while Fig. 7(B) (non-uniform illumination) shows a clearer and sharper body shape and a clear change in depth from the center of the person's torso to the edge of the torso. (ii) Regarding identifying the shape of the left hand and the individual fingers of the left hand, in Fig. 7(A) the shape of the hand is not clear, while in Fig. 7(B) the shape of the hand is clearer and the thumb is somewhat prominent. (iii) Regarding identifying the shape of the cube, in Fig. 7(A) the cube is distorted, while in Fig. 7(B) the square side shape of the cube is more prominent. Therefore, the test shows that when most of the illumination is focused within 25% or less of the field of view, the accuracy of the depth information of a ToF-based 3D sensing system at a distant distance is improved. More generally, the illumination can be focused between 1% or less and 50% or less of the field of view, between 10% or less and 40% or less of the field of view, or between 20% or less and 30% or less of the field of view.

[0029] Therefore, the applicant has found that by removing the diffuser plate of a standard ToF camera system, non-uniform illumination is emitted by the system (i.e., the illuminance is higher in the center than at the edges), and furthermore, with the modified camera system, it is possible to obtain more accurate depth information at a distant distance (e.g., 7 meters or more). Although the power and the total light flux passing through the exit pupil of the camera system are not changed, the peak illuminance in the object field increases. In this sense, a trade-off between the field of view coverage on the one hand and the Z (depth) range and / or accuracy on the other hand is achieved.

[0030] The applicant has discovered that in order to compensate for the loss of XY illuminance in the object field, an actuating mechanism is required to move the emitted light around the imaged scene.

[0031] Accordingly, the present technology provides an apparatus for generating a three-dimensional representation of a scene, the apparatus comprising a multi-pixel sensor having a field of view and a light source, and a time-of-flight (ToF) imaging camera system arranged to emit illumination having spatially non-uniform intensity across the field of view of the sensor, and an actuating mechanism for enabling the generation of the above representation by moving the emitted non-uniform illumination across at least a portion of the field of view of the sensor.

[0032] Referring now to FIG. 1, this shows a schematic diagram of an apparatus 102 or system 100 for generating a three-dimensional (3D) representation of a scene using a time-of-flight (ToF) camera. For example, the apparatus 102 may be or may be included in any one of a smartphone, a mobile computing device, a laptop, a tablet computing device, a security system, a game system, an augmented reality system, an augmented reality device, a wearable device, a drone, an aircraft, a spacecraft, a vehicle, a self-driving vehicle, a robotic device, a consumer electronics device, a domotic device, and a home automation device.

[0033] The apparatus 102 comprises a time-of-flight (ToF) camera 104 configured to emit non-uniform illumination and comprising a light source 106. The ToF camera 104 may comprise a multi-pixel sensor or detector 108 for receiving reflected light from the field of view.

[0034] Non-uniform illumination is any form of illumination and can be provided / emitted by any suitable light source 106. For example, the light source 106 can be an invisible light source or a near-infrared (NIR) light source for the reasons described above. The light source 106 may comprise at least one laser, a laser array (e.g., a VCSEL array), or may comprise at least one light-emitting diode (LED). The non-uniform illumination emitted by the light source 106 (or by the entire device 100) can have any form or shape. For example, the non-uniform illumination can be a light beam having a circular beam shape (e.g., as shown on the left hand side of FIG. 3(A)), or can consist of a parallel stripe pattern of light (e.g., as shown on the left hand side of FIG. 4(B)), or can consist of a uniform or non-uniform pattern of dots or circles of light (e.g., as shown on the left hand side of FIG. 4(A)). These are merely exemplary types of illumination and are to be understood as non-limiting.

[0035] As a general rule, the Applicant has found that, when an increase in the range by a factor of two is required, four times higher illumination intensity is required in the far field (object field) in order to maintain the signal-to-noise ratio.

[0036] The device 102 comprises an actuating mechanism 110 for moving the emitted non-uniform illumination over at least a part of the field of view of the sensor 108. The actuating mechanism 110 can be any suitable actuating mechanism for incorporation into the device 102 and for use in an imaging system. For example, the actuating mechanism 110 can be a shape memory alloy (SMA) actuating system comprising at least one SMA actuator wire. The at least one SMA actuator wire can be coupled to each element of the device 102 that can be operated to move the emitted non-uniform illumination over at least a part of the scene. Additionally or alternatively, the actuating mechanism 110 can comprise a voice coil motor (VCM) or an adaptive beam steering mechanism for steering non-uniform illumination (which can comprise an electrically switchable spatial light modulator). The actuating mechanism 110 can be configured to move the emitted non-uniform illumination by moving any one of the following components of the device 102 or the ToF camera 104: a lens, a prism, a mirror, a dot projector, and a light source 106.

[0037] In an embodiment, the device 102 can comprise at least one movable optical element 114 provided "in front" of the light source 106, i.e., between the light source 106 and the object field / scene. The actuating mechanism 110 can be configured to spin or rotate or move the optical element 114 to move the emitted non-uniform illumination. The optical element 114 can be any one of a lens, a prism, a mirror, and a diffraction grating.

[0038] Figures 5(A) and 5(B) respectively show block diagrams of an apparatus 500 for generating 3D representations in which illumination is directed towards the center of a scene and towards the right side of a scene. The apparatus 500 includes a light source 502 (e.g., a VCSEL array). The light emitted by the light source 502 can pass through one or more optical elements 504 (e.g., lenses, mirrors, diffraction gratings, etc.) before being emitted from the apparatus 500 and projected onto a scene / subject 508. The apparatus 500 can include a receiving lens and filter system 510 and a multi-pixel sensor / detector 512 for sensing reflected light. One or more of the optical elements 504 can be coupled to an actuating mechanism 506. The actuating mechanism 506 is configured to move the optical element 504 to which it is coupled. Figure 5(A) shows the optical element 504 in its central or default position, whereby the emitted non-uniform illumination is projected onto the center of the scene 508 corresponding to the field of view of the sensor 512. Figure 5(B) shows how one of the optical elements 504 can be moved by the actuating mechanism 506 to move the non-uniform illumination to a different area of the scene 508. In the illustration, by moving the optical element 504 to the left in the figure, the non-uniform illumination is projected onto the right side of the scene 508. Thus, the actuating mechanism 506 can be used to steer the illumination towards a particular subject or area in the scene 508 during imaging, thereby irradiating the entire scene 506 with increased intensity so that an improvement and increase in image resolution can be achieved over a larger area.

[0039] Returning to FIG. 1, the actuation mechanism 110 can be used to move / steer the emitted non-uniform illumination in a scanning pattern over at least a portion of the field of view of the sensor 108. For example, FIGS. 3(A) and 3(B) respectively show a 9-point scanning pattern and a 5-point scanning pattern using a circular beam. The scanning pattern can be a raster scanning pattern. The scanning pattern can be meandering. It can be seen from FIGS. 3(A) and 3(B) that increasing the number of points in the scanning pattern results in the field of view being illuminated more uniformly, thereby potentially improving the resolution across the entire field of view. However, as the number of points in the scanning increases, the number of frames to be captured and synthesized to generate the 3D representation increases. With more frames, the high-precision synthesis of the frames becomes even slower and more difficult, and the chance of irresolvable inconsistencies between frames increases. In some cases, the scanning pattern shown in FIG. 3(B) may be suitable when it is acceptable to sacrifice illumination at the corners of the field of view and good resolution near the center of the field of view for improved coverage.

[0040] In FIGS. 3(A) and 3(B), the non-uniform illumination is a substantially circular light beam, which can be the far-field radiation pattern of a light source having no additional optical system whatsoever. A disadvantage of this type of illumination can be that a large steering angle is required to ensure that the illumination is projected over the entire field of view of the sensor 108. For example, for a 60-degree field of view, to cover substantially the entire field of view of the sensor 108 (i.e., the scene for which the 3D representation is to be generated), it is necessary to be steered at approximately 40 degrees along one axis (e.g., the horizontal axis). This can be difficult to achieve by directly moving the light source itself (or some other optical element), as it is difficult to make a reliable electrical connection to something that needs to be moved over a long distance very quickly and very frequently (e.g., in millions of repetition cycles).

[0041] To reduce the amount by which the illumination should move to cover substantially the entire field of view of sensor 108 when a scanning pattern is applied, illumination that is or comprises a pattern of light can be advantageous. Thus, optical elements such as dot projectors or diffraction gratings can be used to fill the subject space field of view, albeit at a low fill factor. This ensures that bright illumination is projected onto the field of view while reducing the movement required to irradiate the entire field of view when the illumination is moved across the field of view in a scanning pattern by adding and subtracting approximately half of the average spacing. FIG. 4(A) shows a six-point scanning pattern using a dot pattern of light, and FIG. 4(B) shows a three-point scanning pattern using a striped light pattern. In FIG. 4, the scanning pattern comprises moving the illumination along two axes, for example, left-right and up-down. Increasing the number of points in the scanning pattern can make the field of view being irradiated more uniform, as described above. In FIG. 4(B), the scanning pattern comprises moving the illumination along one axis, for example, left-right or in one direction (e.g., from left to right). Thus, having striped illumination can be advantageous and the actuation mechanism only needs to move the subject in one direction. Thus, the scanning pattern implemented by the actuation mechanism can comprise moving the emitted non-uniform illumination along one axis across at least a portion of the field of view or along two axes across at least a portion of the field of view.

[0042] With respect to patterned illumination (e.g., the patterns shown in FIGS. 4(A) and 4(B)), the pattern can be regular or irregular. This is in contrast to 3D sensing systems using structured light emitters, where the projection pattern needs to be sufficiently irregular such that the projected dots can be uniquely identified and mapped with respect to their reflections. Furthermore, in contrast to structured light systems, it is not a requirement that the light of the ToF system be accurately focused on the object / subject being imaged.

[0043] Regardless of the type of illumination used, the actuation mechanism may move the emitted non-uniform illumination to independent positions within the field of view, or may continuously move the emitted non-uniform illumination across at least a portion of the field of view. This is because the ToF measurement technique relies only on the illumination intensity with respect to time, and there is no need for the actuation mechanism to stop for sampling the scene.

[0044] Here, referring to FIGS. 8(A)-(E), a predetermined example of the optical field that can be generated by a predetermined modification of the apparatus 102 of FIG. 1 will be described. In each of these modifications, the apparatus 102 includes a vertical-cavity surface-emitting laser (VCSEL) as the light source 106.

[0045] FIG. 8(A) shows an optical field having a single high-intensity region 201. Within the region 201, the irradiance is broadly constant at the peak illumination intensity. This optical field can be realized using only the VCSEL 106 without additional optical components. A simple lens element can be used to control the size of the region 201, and thereby control the intensity of the peak illuminance and the ratio in the field of view 200 of the sensor 108 being illuminated (at a given distance, e.g., 3 to 10 meters or less). To scan the field of view 200, the region 201 has to be steered by a relatively large angle as shown.

[0046] Figure 8(B) shows a pattern corresponding to the VCSEL pattern projection and an optical field having a plurality of high-intensity regions 202. Within each of these regions 202, the irradiance is generally constant and close to the peak illumination intensity. Each region 202 corresponds to light emitted from a single VCSEL cavity. Thus, the design of the VCSEL 106 determines the pattern of the optical field. To generate such a pattern, the apparatus 102 must generally include lens elements focused on the exit surface of the VCSEL 106. These lens elements can be ball lenses or microlens arrays, as will be described below with reference to FIGS. 10 and 11. The pattern extends across the field of view 200 of the sensor 108. As shown, the steering angle required to scan the field of view 200 is significantly reduced compared to FIG. 8(A).

[0047] Figure 8C shows an optical field having a pattern corresponding to the projection of the VCSEL split by a diffractive optical element or a beam splitter. The pattern includes a plurality of high-intensity regions 203, within each of which the irradiance is generally constant and close to the peak illumination intensity. The corresponding regions 203 within each of the plurality of replicas of the VCSEL pattern correspond to light emitted from a single cavity within the VCSEL 106. Thus, the design of the VCSEL 106 determines the pattern within each of these replicas. Optical elements such as holographic diffractive elements are used to split the VCSEL pattern. This can split the VCSEL pattern into an M×N array. In the example shown, M = 2 and N = 2. The pattern extends across the field of view 200 of the sensor 108. As shown, compared to FIG. 8(B) (and the similar VCSEL 106), the number of regions is increased, so the required steering angle is reduced.

[0048] FIG. 8(D) shows the optical field corresponding to a single beam from the VCSEL 106 (see FIG. 8(A)) split into a pattern of multiple beams 204 by a diffractive optical element or a beam splitter. In particular, the optical element splits the input beam into an output beam 204 of an M×N array. In this example, the array is a 2×2 array. Various different types of optical elements can be used. Similar to FIGS. 8(B)-(D), the pattern reduces the steering angle required to scan the field of view 200 of the sensor 108.

[0049] FIG. 8E shows the optical field corresponding to a single beam from the VCSEL 106 (i.e., composed of all of the VCSEL cavities) split into a series of stripes 205 using a suitable diffractive optical element. Such a pattern requires movement in only one direction to fill the field of view 200 of the sensor 108.

[0050] FIGS. 9(A)-(D), 10, and 11 show certain variations of the apparatus of FIG. 1. In each of these variations, the apparatus 102 includes a VCSEL as the light source 106 and a set of one or more optical elements (hereinafter also referred to as an optical stack). The pattern of non-uniform illumination generated by the VCSEL 106 and the optical stack can be steered around the field of view 200 of the sensor 108 by an actuating mechanism 110 corresponding to a small actuator, e.g., an SMA-based actuator. The optical stack can include a lens element for collimation of light, a diffractive optical element for optical field control, and additional lens elements for reducing distortion and improving performance.

[0051] FIG. 9(A) shows an example where the small actuator 110 tilts a submodule 300 composed of the VCSEL 106 and the optical stack 301. The VCSEL 106 and the optical stack 301 have fixed positions and orientations relative to each other. By tilting the submodule 300 in a direction away from the optical axis, the light can be steered. In one example, the submodule 300 is tiltable in both directions away from the optical axis.

[0052] FIG. 9(B) shows an example where a small actuator 110 is used to shift a lens 310 that steers light. The optical stack includes a collimation lens 311 and, in some examples, an optional diffractive element. In the example shown, the collimation lens 311 and the shift lens 310 are separate. However, the collimation lens and the shift lens may be the same lens element, as in the example of FIG. 10 (see below). Steering of the light is effected by translation of the shift lens 310 in a direction perpendicular to the optical axis.

[0053] FIG. 9C shows an example where a mirror system 320 is used to steer light. As shown in FIGS. 9(A) and 9(B), the optical stack may include optional lenses and diffractive elements 321. In this example, a system of two mirrors 320 is used to steer the light. By changing the angle that the mirrors make with the optical axis, the pattern can scan the field of view 200 of the sensor 108. The light can be steered by a single actuated mirror that is rotatable about two longitudinal axes. Alternatively, each of the two mirrors may be rotatable about a single axis, and the axes of the two mirrors are orthogonal. In other examples, the apparatus 102 has a single mirror, and the VCSEL 106 can emit light at an angle of 90 degrees or less with respect to the final approximate direction.

[0054] FIG. 9(D) shows an example where a prism pair 330 is used to steer light. Again, the optical stack may include a diffractive element 331 with an optional collimation lens. The light can be steered by adjusting the relative orientation of the prisms 330 with respect to each other and with respect to the VCSEL 106.

[0055] FIG. 10 shows another example where a ball lens 400 is used to project the pattern of the VCSEL 106 into the far field. Since the ball lens 400 has a short back focal length, it is positioned appropriately close to the surface of the VCSEL 106. The back focal length for a ball lens with a diameter of 0.5 mm to 2 mm is typically 0.3 mm or less.

[0056] The position of the pattern is controllable by translating the ball lens 400 in a direction perpendicular to the direction D in which light is normally emitted. The short back focal length increases the beam steering achieved for a given translation. Thus, a small actuator 106 can be readily used to control the position of the lens 400.

[0057] The ball lens 300 may be constructed from optical glass, glass, plastic, or other optical materials and may be coated with an anti-reflection coating specifically tuned to the wavelength of the VCSEL 106.

[0058] In FIG. 10, additional optical components (not shown) may also be included in the optical stack. For example, a diffractive optical element may be used to create a more detailed pattern, or an additional lens element may be added to reduce pattern distortion in the far field.

[0059] FIG. 11 shows an example having a microlens array 450 disposed near the VCSEL 106. The microlens array 450 is used to generate an illumination pattern in the far field. The microlens array 450 is composed of a plurality of microlenses 450a. There is a microlens 450a on each individual VCSEL cavity 106a. The microlenses 450a are preferably designed to collimate the light from each cavity 106a.

[0060] The position of the pattern in the far field is controllable by translating the microlens array 450 in a direction perpendicular to the direction in which light is normally emitted. Since each microlens 450a can have a very short focal length, a relatively large steering angle can again be achieved with a relatively small displacement.

[0061] Alternatively, the microlens array 450 may have a fixed position with respect to the VCSEL 106, and other optical elements in the device 102 may be translated to steer the pattern of light. The microlens array 450 may be included with additional optical components in the optical stack both when actuated and when stationary. For example, diffractive optical elements may be used to create a more detailed pattern, or additional lens elements may be added to reduce pattern distortion in the far field.

[0062] The microlenses may be manufactured at the wafer level to produce a cost-effective small array.

[0063] A standard sensor 108 may have a field of view of 62 degrees or less by 45 degrees or less. The example shown in FIG. 10 with the ball lens 400 may be capable of achieving steering of 0.025 degrees to 0.07 degrees / μm of shift / stroke. The example shown in FIG. 11 with the microlens array 450 will require significantly lower stroke for the same steering.

[0064] In an embodiment, the illumination pattern may be selected to be non-uniform across the field of view, which may help provide a selective improvement in the range and resolution within a particular field of view. For example, in an embodiment, a first scan of the field of view may be performed to identify one or more target subjects or regions. Thereafter, the illumination may be concentrated on the target subject / region. Returning to FIG. 1, the ToF imaging camera 104 of the device 102 may be configured to perform a first scan of the field of view to identify one or more target subjects / regions within the field of view. Alternatively, a separate camera 112 may be used. For example, a part of the device 102 or a separate optical camera 112 may be configured to perform a first scan of the field of view to identify one or more target subjects / regions within the field of view. However, after the first scan is performed, the actuation mechanism may move the non-uniform illumination emitted mainly across the identified one or more target subjects within the field of view.

[0065] FIG. 2 shows a flowchart of exemplary steps for generating a 3D representation of a scene using the apparatus or system described with reference to FIG. 1. The method begins in step S204 by emitting non-uniform illumination into the field of view of the scene / sensor using the time-of-flight (ToF) imaging camera system of the apparatus (step S204). The method comprises the step of moving the emitted non-uniform illumination across and at least partially over the field of view of the sensor using the actuation mechanism of the apparatus (step S206). The sensor / detector receives the reflected light (step S208), and the time of flight (i.e., the time required from the emission of the light to the reception of the reflected light) is used to identify the depth of the subject within the field of view (step S210). In step S212, the process checks whether all the exposures / frames have been acquired to generate the 3D representation. If not acquired, the process returns to step S206. If acquired, the exposures / frames are synthesized to generate the 3D representation (step S214).

[0066] Optionally, the method may begin by performing an initial scan of the field of view (step S200) and identifying one or more target subjects (or regions) within the field of view (step S202). In this case, the step of moving the non-uniform illumination (step S206) may include the step of moving the emitted non-uniform illumination across at least the one or more identified target subjects within the field of view.

[0067] In an embodiment, the emitted non-uniform illumination can be moved based on both a target area or subject within the field of view and the intensity of the received / detected reflected light or the signal-to-noise ratio. For example, if a very small amount of light is detected by the sensor / detector, the system can determine that the target subject / area is too far away, so the illumination can be moved to a new position. Similarly, when the intensity of the reflected light is very high, sufficient information about the field of view can be collected relatively quickly in such a way that the illumination can be moved to a new position relatively quickly (so as to capture information about other subjects / areas in the field of view), while when the intensity of the reflected light is low, the illumination may need to be held in place longer to enable sufficient information to be collected to generate a reliable 3D representation. Thus, in an embodiment, the actuation mechanism can move the emitted non-uniform illumination in response to the detected intensity of the reflected light and / or the signal-to-noise ratio.

[0068] It should be understood that there can be numerous other variations of the above-described embodiments.

[0069] For example, the optical element can be any one of a lens, a prism, a mirror, and a diffraction grating.

[0070] The actuation mechanism can include a voice coil motor (VCM).

[0071] The actuation mechanism can be configured to move the emitted illumination by moving any one of a lens, a prism, a mirror, a dot projector, and a light source.

[0072] The device may include an optical element disposed between the light source and the scene, and the actuation mechanism can be configured to spin or rotate the optical element.

[0073] References to the field of view of the sensor may also refer to the field of view of the sensor and any associated optical elements. It should be noted that the present invention includes the following aspects. 〔Aspect 1〕 An apparatus for generating a three-dimensional representation of a scene, comprising a multi-pixel sensor and a light source, and a time-of-flight (ToF) imaging camera system arranged to emit illumination having spatially non-uniform intensity across the field of view of the sensor, and an actuating mechanism for enabling the generation of the representation by moving the illumination across at least a part of the field of view of the sensor The apparatus comprising the above. 〔Aspect 2〕 The apparatus according to Aspect 1, wherein the spatially non-uniform intensity corresponds to a set of regions having a substantially constant peak emission intensity and / or the peak emission intensity is at least 50% of the maximum level. 〔Aspect 3〕 The apparatus according to Aspect 2, wherein the set of regions together cover from 1% to 50% of the field of view of the sensor at a given instant, and optionally, the set of regions together cover more than 10% and less than 50%, less than 40%, less than 30% or less than 20% of the field of view of the sensor at a given instant, and optionally, the set of regions together cover more than 20% and less than 50%, less than 40% or less than 30% of the field of view of the sensor at a given instant, and optionally, the set of regions together cover more than 30% and less than 50% or less than 40% of the field of view of the sensor at a given instant, and optionally, the set of regions together cover more than 40% and less than 50% of the field of view of the sensor at a given instant. 〔Aspect 4〕 The apparatus according to Aspect 2 or 3, wherein the actuating mechanism moves the illumination in a scanning pattern across at least a part of the field of view of the sensor. 〔Aspect 5〕 The apparatus according to Aspect 4, wherein the scanning pattern includes moving the illumination along one axis across at least a part of the field of view of the sensor. 〔Aspect 6〕 The apparatus according to Aspect 4, wherein the scanning pattern comprises moving the emitted illumination along two axes across at least a part of the scene. 〔Aspect 7〕 The apparatus according to Aspect 5 or 6, wherein the set of regions is arranged such that the movement during a cycle of the scanning pattern causes the regions to cover more than 75%, more than 90% or substantially the entire field of view of the sensor. 〔Aspect 8〕 The apparatus according to any one of aspects 5 to 7, wherein the set of regions is arranged so as to avoid a region where the movement covers the same portion of the field of view of the sensor two or more times during the cycle of the scanning pattern. 〔Aspect 9〕 The apparatus according to any one of aspects 5 to 8, wherein the movement moves to a specific point in the non-uniform intensity by less than 50%, less than 40%, less than 30%, less than 20%, less than 10%, or less than 5% of the width or height of the field of view of the sensor during the cycle of the scanning pattern. 〔Aspect 10〕 The apparatus according to any one of aspects 5 to 9, wherein the set of regions has periodicity in at least one direction of the field of view of the sensor, and the movement moves to a specific point in the non-uniform intensity by approximately the reciprocal of the periodicity in the at least one direction. 〔Aspect 11〕 The apparatus according to any one of aspects 1 to 10, wherein the emitted illumination is an optical beam having a circular beam shape, consisting of a pattern of parallel fringes of light, or consisting of a pattern of dots or circles of light. 〔Aspect 12〕 The apparatus according to any one of aspects 2 to 11, configured to use information only from pixels of the sensor having a field of view within the set of regions at a given instant when generating the representation. 〔Aspect 13〕 The apparatus according to any one of aspects 2 to 12, wherein the emitted illumination has substantially the same temporal variation across the entire set of regions. 〔Aspect 14〕 The apparatus according to aspect 13, wherein the emitted illumination is varied at a high frequency and moved across the field of view repeatedly at a relatively low frequency. 〔Aspect 15〕 The apparatus according to any one of aspects 1 to 14, wherein the actuating mechanism moves the emitted illumination to an independent position in the scene. 〔Aspect 16〕 The apparatus according to any one of aspects 1 to 15, wherein the actuating mechanism continuously moves the emitted illumination across at least a part of the scene. 〔Aspect 17〕 The apparatus according to any one of aspects 1 to 16, wherein the actuating mechanism includes at least one shape memory alloy (SMA) actuator wire. 〔Aspect 18〕 The apparatus according to any one of aspects 1 to 17, wherein the light source includes a plurality of lasers arranged in an array. 〔Aspect 19〕 The apparatus according to aspect 18, wherein the plurality of lasers corresponds to a vertical cavity surface emitting laser (VCSEL) array. 〔Aspect 20〕 The apparatus according to aspect 18 or 19, comprising a focusing lens that focuses illumination from the plurality of lasers onto a single beam corresponding to a proportion of the field of view of the sensor. 〔Aspect 21〕 The apparatus according to aspect 18 or 19, comprising a plurality of microlenses, each microlens being configured to focus illumination from one of the plurality of lasers onto one of a plurality of beams, each of the plurality of beams corresponding to a proportion of the field of view of the sensor. 〔Aspect 22〕 The apparatus according to aspect 20 or 21, comprising an optical element configured to divide each of the beam or the plurality of beams into a further plurality of beams. 〔Aspect 23〕 The apparatus according to aspect 22, when dependent on aspect 20, wherein the further plurality of beams corresponds to a fan pattern. 〔Aspect 24〕 The apparatus according to any one of aspects 18 to 23, wherein the actuating mechanism is configured to tilt a sub-module comprising the plurality of lasers and one or more further optical elements about at least one axis. 〔Aspect 25〕 The apparatus according to any one of aspects 18 to 23, wherein the actuating mechanism comprises at least one lens movable in one or more orthogonal directions in a plane substantially parallel to the array of lasers in order to move the illumination across at least a portion of the field of view of the sensor. 〔Aspect 26〕 The apparatus according to aspect 25, wherein the at least one lens corresponds to a ball lens. 〔Aspect 27〕 The apparatus according to aspect 25, when dependent on aspect 21, wherein the at least one lens corresponds to the plurality of microlenses. 〔Aspect 28〕 The apparatus according to any one of aspects 25 to 27, wherein the direction of the illumination is changed by an angle greater than 0.025 degrees / μm of movement of the at least one lens. 〔Aspect 29〕 The apparatus according to any one of aspects 18 to 23, wherein the actuating mechanism comprises at least one tilting mirror for steering the emitted illumination. 〔Aspect 30〕 The apparatus according to any one of aspects 18 to 23, wherein the actuating mechanism comprises at least a pair of rotatable prisms for steering the emitted illumination. 〔Aspect 31〕 The apparatus according to any one of aspects 1 to 30, wherein the actuating mechanism comprises an adaptive beam steering mechanism for steering the illumination. 〔Aspect 32〕 The apparatus according to any one of aspects 1 to 31, wherein the ToF imaging camera system or the optical camera is configured to perform an initial scan of the scene to identify one or more target subjects within the scene. [Aspect 33] A method for use in generating a three-dimensional representation of a scene, comprising: emitting, using a time-of-flight (ToF) imaging camera system, illumination having spatially non-uniform intensity across a field of view of a sensor used to receive reflected light; enabling generation of the representation by moving the illumination across at least a portion of the field of view of the sensor using an actuating mechanism; and a method comprising the steps of. [Aspect 34] A non-transitory data carrier holding processor control code for implementing the method according to aspect 33.

Claims

1. An apparatus for use in generating a three - dimensional representation of a scene, comprising a multi - pixel sensor and a light source, and a time - of - flight (ToF) imaging camera system arranged to emit illumination having spatially non - uniform intensity across the field of view of the sensor, and an actuating mechanism for enabling the generation of the representation by moving the illumination across at least a part of the field of view of the sensor, and the actuating mechanism comprising at least one shape memory alloy (SMA) actuator wire, the spatially non - uniform intensity corresponding to one or more regions where the emitted intensity is substantially constant and / or the emitted intensity is at least 50% of a maximum level, the one or more regions being regions of high emitted intensity compared to other regions, the actuating mechanism moving the illumination in a scanning pattern across at least a part of the field of view of the sensor, the apparatus.

2. The apparatus according to claim 1, wherein the one or more regions together cover from 1% to 50% of the field of view of the sensor at a given instant, the one or more regions together cover more than 10% and less than 50%, less than 40%, less than 30% or less than 20% of the field of view of the sensor at a given instant, the one or more regions together cover more than 20% and less than 50%, less than 40% or less than 30% of the field of view of the sensor at a given instant, the one or more regions together cover more than 30% and less than 50% or less than 40% of the field of view of the sensor at a given instant, or the one or more regions together cover more than 40% and less than 50% of the field of view of the sensor at a given instant.

3. The scanning pattern includes moving the illumination along one axis across at least a part of the field of view of the sensor, or the scanning pattern comprises moving the emitted illumination along two axes across at least a part of the scene. The one or more regions are arranged such that during the cycle of the scanning pattern, the movement causes the region to cover more than 75%, more than 90% or substantially the entire field of view of the sensor. The one or more regions are arranged to avoid regions where the movement covers the same part of the field of view of the sensor more than once during the cycle of the scanning pattern. The movement moves less than 50%, less than 40%, less than 30%, less than 20%, less than 10% or less than 5% of the width or height of the field of view of the sensor towards a particular point at the non-uniform intensity during the cycle of the scanning pattern, and / or the one or more regions have periodicity in at least one direction of the field of view of the sensor, and the movement moves towards a particular point at the non-uniform intensity by approximately the reciprocal of the periodicity in the at least one direction. The apparatus according to claim 2. **Claim 4** The emitted illumination is an optical beam having a circular beam shape, consisting of a pattern of parallel fringes of light, or consisting of a pattern of dots or circles of light. The apparatus according to any one of claims 1 to 3. **Claim 5** Configured to use information only from pixels of the sensor having a field of view within the one or more regions at a given instant of generating the representation. The apparatus according to any one of claims 2 to 4. **Claim 6** The emitted illumination has substantially the same time variation across the one or more regions. The emitted illumination is varied at a high frequency and moved across the field of view relatively slowly and repeatedly. The apparatus according to any one of claims 2 to 5. **Claim 7** The actuating mechanism moves the emitted illumination to an independent position in the scene, or the actuating mechanism moves the emitted illumination continuously over at least a part of the scene. The apparatus according to any one of claims 1 to 6. **Claim 8** The light source comprises a plurality of lasers arranged in an array. The apparatus according to any one of claims 1 to 7, wherein the plurality of lasers correspond to a vertical cavity surface emitting laser (VCSEL) array.

9. A focusing lens that focuses illumination from the plurality of lasers onto a single beam corresponding to a proportion of the field of view of the sensor, A plurality of microlenses, each microlens being configured to focus illumination from one of the plurality of lasers onto one of a plurality of beams, each of the plurality of beams corresponding to a proportion of the field of view of the sensor, the plurality of microlenses comprising, The apparatus comprises an optical element configured to divide each of the beam or the plurality of beams into a further plurality of beams, The apparatus according to claim 8, wherein the further plurality of beams correspond to a fan pattern.

10. The apparatus according to claim 8 or 9, wherein the actuating mechanism is configured to tilt a submodule comprising the plurality of lasers and one or more further optical elements about at least one axis.

11. The light source comprises a plurality of lasers arranged in an array, The apparatus according to any one of claims 8 to 10, wherein the actuating mechanism comprises at least one optical element movable in one or more directions in a plane substantially parallel to the array of lasers in order to move the illumination over at least a portion of the field of view of the sensor.

12. The at least one optical element is at least one lens, The apparatus according to claim 11, wherein the at least one lens corresponds to a ball lens.

13. The apparatus according to claim 12, dependent on claim 11, dependent on claim 9, wherein the apparatus comprises a plurality of microlenses and the at least one lens corresponds to the plurality of microlenses.

14. The apparatus according to claim 12 or 13, configured to change the direction of the illumination by an angle of more than 0.025 degrees / μm of the movement of the at least one lens.

15. The apparatus according to claim 8 or 9, wherein the actuating mechanism comprises at least one tilt mirror for steering the emitted illumination, or at least a pair of rotatable prisms for steering the emitted illumination, or an adaptive beam steering mechanism for steering the illumination.

16. The apparatus according to any one of claims 1 to 15, wherein the ToF imaging camera system or the optical camera is configured to perform an initial scan of the scene to identify one or more subject objects in the scene.

17. A method for use in generating a three-dimensional representation of a scene, emitting, using a time-of-flight (ToF) imaging camera system, illumination having a spatially non-uniform intensity across the field of view of a sensor used to receive reflected light; enabling the generation of the representation by moving the illumination across at least a portion of the field of view of the sensor using an actuating mechanism comprising at least one shape memory alloy (SMA) actuator wire; comprising wherein the spatially non-uniform intensity corresponds to one or more regions having a substantially constant emission intensity and / or the emission intensity is at least 50% of a maximum level, the one or more regions are regions of higher emission intensity compared to other regions, the actuating mechanism moving the illumination in a scanning pattern across at least a portion of the field of view of the sensor, method.

18. The time-of-flight imaging camera system comprises a light source including a plurality of lasers arranged in an array, the actuating mechanism comprises at least one optical element, The method according to claim 17, wherein the method moves the at least one optical element in one or more directions in a plane that is at least substantially parallel to the array of lasers using the actuation mechanism, thereby moving the illumination across at least a portion of the field of view of the sensor, thereby enabling the generation of the representation.

19. A non-transitory data carrier holding processor control code configured to cause the apparatus according to claim 1 to perform the method according to claim 17.

Images