Methods and apparatus for use in a time-of-flight imaging system

Abstract

A method for use in a time-of-flight (TOF) imaging system (500), wherein the system emits illumination in a plurality of configurations, each configuration having a different spatially varying intensity over the field of view of an image sensor (512), the method comprising: moving an actuating mechanism (506) to change the illumination from a first configuration (A) to a final configuration (B) via a first sequence of configurations; moving the actuating mechanism (506) to subsequently change the illumination from the final configuration to the first configuration or a second configuration via a second sequence of configurations; and for each configuration (A, B) in the first and second sequences, obtaining a set of data from the image sensor (512), thereby obtaining two sets of data suitable for generating two depth image frames for each configuration, wherein the two sets of data corresponding to the final configuration are obtained consecutively from the first and second sequences. Light emitted by a light source (502) passes through a set of one or more optical elements (504) before being emitted from the TOF system (500). The set of optical elements (504) includes a diffractive optical element to generate a light field and also includes a shifting lens, to which the actuating mechanism (506) is operatively connected. A translational movement of the shifting lens in a direction perpendicular to the optical axis causes steering of the light. The imaging section includes a receiver lens and / or a filter system (510) and an image multi-pixel sensor (512). The TOF system (500) further includes a controller (500).

Description

[0001] Field

[0002] Among other things, the present application relates to a method suitable for use in a time-of-flight imaging system, particularly where the system emits illumination in a plurality of configurations, each configuration having a different spatially-varying intensity over the field of view of an image sensor.

[0003] Background

[0004] Three-dimensional (3D) sensing systems (also known as range imaging systems) typically produce two-dimensional (2D) images (referred to as range images), where each pixel value corresponds to the distance to a point (or region) in a scene. The distance (also referred to as depth) can be measured in various ways, particularly using structured light techniques or time-of-flight (TOF) techniques. TOF calculations are similar to radar, except that light pulses are used instead of RF pulses, producing a range image (or depth image) similar to a radar image. Typically, TOF is scannerless, meaning the entire scene is captured with a single light pulse. Relatively small and relatively high-performance (in terms of range and resolution) 3D sensing systems suitable for use in portable electronic devices such as mobile phones are of interest.

[0005] Overview

[0006] According to a first aspect of the present invention, there is provided a method suitable for use in a time-of-flight (TOF) imaging system, where the system emits illumination in a plurality of configurations, each configuration having a different spatially-varying intensity over the field of view of an image sensor, the method comprising:

[0007] Moving an actuating mechanism to change the illumination from a first configuration to a final configuration via a first sequence of configurations;

[0008] Moving the actuating mechanism to subsequently change the illumination from the final configuration to the first configuration or a second configuration via a second sequence of configurations; and

[0009] For each configuration in the first and second sequences, obtaining a set of data from the image sensor, thereby obtaining two sets of data regarding each configuration suitable for generating two depth (or range) image frames, where the two sets of data corresponding to the final configuration are obtained consecutively from the first and second sequences.

[0010] For example, if the first sequence and the second sequence each include a first configuration and a final configuration and no other configurations in the sequence, two image frames can be obtained in sequence by obtaining data from 1) the first configuration, 2) the second configuration, 3) the second configuration, and 4) the first configuration. In this way, two data sets regarding the final configuration are continuously captured between the first sequence and the second sequence. In addition, in the case where each of the configurations has been accessed at least once, the two sets of data are suitable for generating two depth image frames, and in practice, multiple sets of data can be used in any suitable way for generating multiple depth image frames.

[0011] The second configuration can be a configuration different from the final configuration. That is, the last configuration in the second sequence is not necessarily the final configuration. For example, the first sequence and the second sequence may not include the same set of configurations.

[0012] The phrase "continuously" does not necessarily mean that two data sets of the final configuration are obtained continuously. There may be a time interval between obtaining these two data sets. Either way, after obtaining the data set of the final configuration in the first sequence, the data set of the final configuration in the second sequence can be obtained.

[0013] Preferably, the two sets of data continuously obtained for the final configuration are obtained without moving the actuating mechanism. Even if the obtaining of the two sets of data corresponding to the final configuration is specified as continuous, there may be a time gap between obtaining these two sets of data.

[0014] Therefore, the illumination can be changed between different configurations that can generate greater intensity (thereby improving performance), while the frequency of moving the actuating mechanism can be reduced. Advantageously, since the movement between the first sequence and the second sequence is omitted, this arrangement can reduce the time required to obtain two image frames. For example, if the first sequence and the second sequence each only include the first configuration and the second configuration, by removing the actuator movement between the first sequence and the second sequence (and between the second sequence and the first sequence), the time accumulated for actuator movement in the repeating cycle can be reduced by half. In addition, when using an SMA actuator, the pause of the actuator movement between sequences can advantageously allow the SMA wire to cool more effectively. This is in comparison with existing methods such as simply repeating sequences.

[0015] Optionally, the cumulative intensity of illumination during the first sequence is substantially the same as the cumulative intensity of illumination during the second sequence. Preferably, the configurations in each of the first and second sequences can have substantially the same cumulative intensity. Optionally, the cumulative intensity can vary from one configuration to another within a sequence, for example to accommodate different projection surfaces. Either way, the cumulative intensity of illumination for any given configuration in the first sequence can be the same as the corresponding configuration in the second sequence.

[0016] Optionally, the second sequence corresponds to the first sequence in reverse order. For example, in some embodiments, the first sequence can be in the order of a first configuration, an intermediate configuration, and a final configuration. Thus, the corresponding second sequence can be in the order of the final configuration, the intermediate configuration, and the first configuration.

[0017] Optionally, each of the first and second sequences includes only the first configuration and the final configuration. Thus, in such embodiments, the actuation mechanism can provide only two levels of movement to move the illumination between two positions.

[0018] Alternatively, each of the first and second sequences includes one or more additional configurations between the first and final configurations in the respective sequence. For example, the first sequence can be in the order of a first configuration, an intermediate configuration, and a final configuration, and the second sequence can be in the order of the final configuration, the intermediate configuration, and the first configuration.

[0019] Optionally, the intensity accumulated during each sequence is substantially uniform across substantially the entire field of view of the image sensor. More specifically, the illumination can be a pattern where the actuation mechanism is movable through the field of view and movable or extendable beyond the boundaries of the field of view within the field of view.

[0020] Optionally, for each sequence, the illumination in each configuration of the sequence substantially does not overlap with the illumination in any other configuration of the sequence. For each sequence, a portion of the illumination in each configuration of the sequence can overlap with the illumination in one or more other configurations. For example, in any given sequence, the illumination within two consecutive configurations can overlap in the illumination area of the configuration by up to 50%, less than 20%, less than 10%.

[0021] Optionally, each sequence includes at least one configuration in which the illumination is substantially non-uniform across at least a portion of the field of view of the camera. For example, the illumination can be focused or partially focused to form a projection pattern or a spot projection commonly referred to as spot illumination.

[0022] Optionally, each sequence includes at least one configuration in which the illumination is substantially uniform over substantially the entire field of view of the camera. Typically, such a configuration can be achieved by defocusing the illumination on the projection surface. Such illumination can be referred to as flood illumination.

[0023] Optionally, the method includes moving an actuator mechanism to focus and defocus the illumination, thereby switching between uniform (flood) illumination and non-uniform (spot) illumination. Generally, this can be achieved by moving a lens or a light emitter along its optical axis.

[0024] Optionally, the field of view of the image sensor corresponds to the field of view of the image sensor together with one or more optical elements associated with the image sensor. However, the field of view of the emitter (or the projected area of the emitter) is not necessarily equal to the field of view of the image sensor. For example, the illumination from the emitter can be projected outside or inside the boundaries of the field of view of the image sensor. The optical elements can include one or more of the following: lens elements (such as microlens arrays), lenses, prisms, mirrors, or diffraction gratings.

[0025] Optionally, the method is performed in a series of sub-frames, in which a set of data is obtained and then the actuator mechanism is moved during each sub-frame associated with each sequence except the final sub-frame. More specifically, each sub-frame corresponds to a configuration of the sequence.

[0026] Optionally, the actuator mechanism is moved within the first part of each sub-frame in which it is moved, and the data is obtained within the second part of each sub-frame.

[0027] Optionally, in each final sub-frame, data is obtained substantially throughout the entire sub-frame. That is, the final sub-frame can be shorter than the other sub-frames in the sequence or the same as the other sub-frames in the sequence.

[0028] Optionally, the sub-frame has a frequency between 10 and 50 Hz (e.g., about 30 Hz) or between 40 and 80 Hz (e.g., about 60 Hz) or between 100 and 140 Hz (e.g., about 120 Hz). Optionally, the actuation mechanism moves at a frequency between 5 and 25 Hz (e.g., about 15 Hz) or between 20 and 40 Hz (e.g., about 30 Hz) or between 50 and 70 Hz (e.g., about 60 Hz). This may be an exception for the final sub-frame. Such a low frequency range can be applied to systems in which the actuator mechanism moves the illumination only between a limited number of configurations (e.g., 2 configurations in a sequence, such as A - B... B - A). In such a system, the depth image frame can be obtained at 30 Hz, and the actuation mechanism frequency is 60 Hz, where the first sequence and the final sequence each have two configurations (A - B and B - A). Generally, the ratio of the actuation mechanism frequency to the sub-frame frequency can be (N - 1) / N, where N is the number of configurations.

[0029] In systems characterized by a higher number of configurations in a sequence, e.g., 4 configurations (A - B - C - D or D - C - B - A) in a sequence as characterized in a typical light spot scanning system, the corresponding sub-frame can accommodate a higher frequency of 120 Hz in order to maintain a frame rate of 30 fps when obtaining the depth image. Conversely, the sub-frame can operate at a frequency of 60 Hz, which results in a reduced depth image frame rate of 15 fps.

[0030] Depending on the limitations of the actuator mechanism, the sub-frame frequency can be further increased to accommodate more configurations in a sequence. For example, for a light spot scanning system characterized by 16 configurations in a sequence, the sub-frame frequency can be further increased to 480 Hz, which can produce a depth image at a frame rate of 30 fps.

[0031] Optionally, each first part has a first duration, and each second part has a second duration that is longer than the first duration. Alternatively, each first part has a first duration, and each second part has a second duration that is shorter than the first duration. Optionally, the first duration is less than 10 milliseconds.

[0032] Optionally, the actuation mechanism includes one or more shape memory alloy actuators that are resistively heated to provide movement.

[0033] Optionally, one or more shape memory alloy actuators are resistively heated within each first part and cooled within each second part such that the average temperature of the wires does not increase substantially between consecutive sub-frames. Optionally, the temperature of the wires decreases during the final sub-frame.

[0034] Optionally, the set of data obtained for each configuration includes the set of data for the first configuration obtained without moving the actuating mechanism previously. For example, once the second sequence is completed and the data set for the first configuration has been obtained, a new first sequence can be started without moving the actuating mechanism. For example, after obtaining the data set for the first configuration in the previous second sequence, another data set for the first configuration in the new first sequence can be continuously obtained.

[0035] Optionally, the changes in illumination each include moving the illumination through the field of view in a scanning pattern. In other words, the first sequence and the second sequence can include moving the focused illumination in multiple discrete movements, or one or more continuous movements, or a combination of both. Subsequent cycles of the scanning pattern can then use the same cycle as previously used, or a different cycle. For example, a different cycle can allow scanning at different points (e.g., different regions of interest) in each cycle, or scanning the same points in a different order (e.g., different scan paths). Non-uniform illumination can move through at least a portion of the field of view at discrete positions, or can move continuously through at least a portion of the field of view. This is because in some embodiments, the time-of-flight measurement technique only depends on the illumination intensity over a period of time and does not require the movement of the illumination to be aborted in order to obtain the data set.

[0036] Optionally, the scanning pattern includes moving the illumination through the field of view in at least two non-parallel directions in each of the first sequence and the second sequence. For example, the scanning pattern can include moving the illumination through at least a portion of the field of view along a first direction in one configuration. The scanning pattern can also include moving the illumination through at least a portion of the field of view along a second direction in another configuration. The first direction can be perpendicular to the second direction, or at an angle to the second direction in a plane. That is, the first direction can be at a non-zero angle to the second direction. The scanning pattern can be a raster scanning pattern. The scanning pattern can be boustrophedonic. Increasing the number of (scanning) points in the scanning pattern can produce a more uniformly illuminated field of view, which can achieve an improvement in the resolution of the entire field of view. However, the more points in the scanning pattern, the more data sets need to be obtained and combined to generate the output image frame. Therefore, a scanning pattern suitable for the application can be selected.

[0037] Optionally, the illumination includes a beam having a beam projection configured as a checkerboard pattern, a beam having a circular or polygonal beam projection, a pattern of parallel light bars, or a pattern of points or circles of light. It should be understood that these are merely example types of illumination and are non-limiting. Regarding the checkerboard pattern, this means that the beam shape is configured to substantially cover the field of view when the focused illumination is moved, and the beam shapes substantially do not overlap. This may be that there are no gaps between the projections, or there may be gaps between the projections.

[0038] Optionally, the method includes a plurality of scan cycles, each of which has substantially the same first configuration and final configuration whether the configured first sequence and second sequence are the same or different. More specifically, the plurality of scan cycles may have substantially the same first configuration and final configuration in the case of different scan patterns.

[0039] Optionally, the second sequence is from the final configuration to the second configuration, and the method further includes: moving the actuating mechanism to subsequently change the illumination from the second configuration to the final configuration or the third configuration via a configured third sequence; and obtaining a set of data from the image sensor for each configuration in the third sequence, wherein two sets of data corresponding to the second configuration are obtained without moving the actuating mechanism.

[0040] Optionally, the third sequence is from the second configuration to the third configuration, and the method further includes: moving the actuating mechanism to subsequently change the illumination from the third configuration to the final configuration via a configured fourth sequence; and obtaining a set of data from the image sensor for each configuration in the fourth sequence, wherein two sets of data corresponding to the third configuration are obtained without moving the actuating mechanism.

[0041] A non-transitory data carrier can be provided that carries code for causing a time-of-flight imaging system to perform the method.

[0042] Apparatus for use in a time-of-flight imaging system can be provided, the apparatus being configured to perform the method.

[0043] Optionally, the apparatus includes:

[0044] an illumination section for emitting illumination;

[0045] an actuating mechanism, wherein the actuating mechanism is included in the illumination section or operatively connected to the illumination section; and

[0046] an imaging section including an image sensor;

[0047] at least one controller operatively connected to at least the actuating mechanism and the imaging section and configured to perform the method.

[0048] Optionally, the actuating mechanism includes one or more shape memory alloy (SMA) components. These can be SMA wires. Brief Description of the Drawings

[0050] Certain embodiments of the present invention will now be described by way of example only with reference to the drawings, in which:

[0051] Figure 1APerspective view of an SMA actuator according to a first embodiment of the present invention.

[0052] Figure 1B Schematic diagram of a TOF system that emits illumination in multiple configurations, where each configuration has a different spatially varying intensity over the field of view of the image sensor. The system is shown generating a first configuration (A) and a second configuration (B) of illumination.

[0053] Figure 2 Shows a first configuration (A) and a second configuration (B) of illumination generated by the system of FIG. 1, and the sum C = (A + B) of the illumination of the first and second configurations.

[0054] Figure 3 Shows what can be performed by Figure 1B system.

[0055] Figure 4A Shows an illumination pattern according to a second embodiment of the present invention.

[0056] Figure 4B Shows a first configuration (A) and a final configuration (D) and intermediate configurations (B, C) of illumination generated by the system of FIG. 1.

[0057] Figure 5A Perspective view of an SMA actuator according to a second embodiment of the present invention.

[0058] Figure 5B and Figure 5C Show the movement of the lenses of a prior art TOF system and a TOF system of the second embodiment, respectively.

[0059] Figure 5D and Figure 5E Show illumination patterns for non-uniform (condensing) illumination and uniform (flood) illumination, respectively.

[0060] Detailed description

[0061] Referring to FIGS. 1 to Figure 3 , an example of a TOF system 500 will now be described.

[0062] Figure 1A Shows an SMA actuator device 506 implemented in a time-of-flight sensor system. The actuator device 506 includes a total of four SMA actuator wires 11, 12, 13, 14 connected between a support block 16 and a movable element 15, the support block 16 forming part of a support structure and being mounted to a base.

[0063] Each of the SMA actuator wires 11 to 14 is kept taut so as to apply a force between the movable element 15 and the support block 16 in a direction perpendicular to the envisioned main axis (here called the optical axis). In operation, the SMA actuator wires 11 to 14 move the movable element 15 relative to the support block 16 in two orthogonal directions perpendicular to the optical axis.

[0064] The SMA actuator wires 11 to 14 are connected at one end to the movable element 15 by respective crimping members 17 and at the other end to the support block 16 by crimping members 18. The crimping members 17, 18 crimp the wires to mechanically hold the wire, optionally strengthening the hold by using an adhesive. The crimping members 17, 18 also provide an electrical connection to the SMA actuator wires 11 to 14. However, it will be understood that any suitable means for connecting the SMA actuator wires 11 to 14 may be used optionally.

[0065] The four SMA wires 11 to 14 are arranged in a loop around the optical axis. The four SMA wires consist of a first pair of SMA wires 11, 13 arranged on opposite sides of the optical axis and a second pair of SMA wires 12, 14 arranged on opposite sides of the optical axis. The first pair of SMA wires 11, 13 can be selectively driven to move the movable element 15 relative to the support structure in a first direction, and the second pair of SMA wires 12, 14 can be selectively driven to move the movable element 15 relative to the support structure in a second direction transverse to the first direction. In addition to the movement in the direction parallel to the SMA wires 11 to 14 being driven by the combination of the actuation of these two pairs of SMA wires to provide a linear combination of the movement of the movable element in the transverse direction. Another way of observing this movement is that the simultaneous contraction of any pair of SMA wires adjacent to each other in the loop will drive the movement of the movable element in the direction bisecting these two SMA wires (i.e., generating a diagonal movement).

[0066] Thus, the SMA wires 11 to 14 can be selectively driven to move the movable element 15 relative to the support structure to any position within the movement range in two orthogonal directions perpendicular to the optical axis. The size of the movement range depends on the geometry and the contraction range of the SMA wires within their normal operating parameters.

[0067] With particular reference Figure 1B , the TOF system 500 includes an illumination section that includes a light source 502 and a set of optical elements 504.

[0068] The light source 502 can be a vertical cavity surface emitting laser (VCSEL) array or another type of laser light source or another type of light source (such as an LED light source).

[0069] The light emitted by the light source 502 passes through a set of one or more optical elements 504 before being emitted from the TOF system 500.

[0070] The set of optical elements 504 may include one or more lens elements for light collimation, one or more diffractive optical elements for light field control, one or more lenses for guiding light, and / or one or more other types of optical elements, such as prisms, mirrors, etc.

[0071] The SMA actuator mechanism 506 is operably connected to one or more of the set of optical elements 504 and is configured to move one or more of the set of optical elements 504.

[0072] Thus, the TOF system 500 is capable of emitting illumination in a variety of configurations, each of which has a different spatially varying intensity over the field of view of the image sensor.

[0073] In the example shown, the set of optical elements 504 includes diffractive optical elements to generate a light field, which will be described in more detail below. The set of optical elements 504 also includes a lens (hereinafter referred to as the shift lens), to which the actuator mechanism 506 is operably connected. Translational movement of the shift lens 310 in a direction perpendicular to the optical axis causes steering of the light. In Figure 1A , the light is steered to the left, and in Figure 1B , the light is steered to the right. The actuator mechanism 506 described in WO 2013 / 175197 A1 is incorporated herein by reference.

[0074] The TOF system 500 also includes an imaging section that includes a receiver lens and / or filter system 510 and an image sensor (also referred to as a multi-pixel sensor) 512 for sensing light reflected from a scene.

[0075] Specific reference is made to Figure 2 , the TOF system 500 is configured to emit illumination in two different configurations (hereinafter sometimes referred to as Configuration A and Configuration B). In each configuration, the illumination is largely concentrated in a series of stripes that together fill approximately 50% of the field of view of the image sensor 512, where the illumination in Configuration A substantially does not overlap with the illumination in Configuration B. Thus, as shown in the lower panel of the accompanying drawings, the sum of the illumination in Configuration A and Configuration B is substantially uniform over the field of view of the image sensor 512. In some other embodiments, there may be overlap between the illumination in Configuration A and Configuration B. For example, in each configuration, the illumination may be largely concentrated in a series of stripes that together fill more than 50% of the field of view of the image sensor, where the illumination in Configuration A overlaps with the illumination in Configuration B.

[0076] The fringe light field is generated by the above-described diffractive optical element. In Configuration A, the fringes are located on the left side of the field of view, while in Configuration B, the fringes are located on the right side of the field of view. As described above, the movement between the two configurations is generated by translating the shift lens 310 by the actuation mechanism 506. The shift lens 310 can be moved a distance of approximately 100 μm.

[0077] With particular reference to FIG. 1, the TOF system 500 further includes a controller 550, which is operably connected to the actuation mechanism 506 and the image sensor 512. The controller 550 is configured to operate the other parts of the TOF system 500 as described herein.

[0078] In particular, referring to Figure 3 , a set of operations performed by the TOF system 500 will now be described.

[0079] At a first step S1, the TOF system 500 is arranged to emit illumination in Configuration A (see Figure 1A and Figure 2 A).

[0080] At a second step S2, the TOF system 500 obtains a set of data (hereinafter referred to as "Data A1") from the image sensor 512. For the TOF system 500, this step itself can be performed in a conventional manner. For the illumination area (fringes) of Configuration A, relatively high-resolution and / or distance-depth data can be obtained.

[0081] At a third step S3, the actuation mechanism 506 moves the shift lens such that the TOF system 500 emits illumination in Configuration B (see Figure 1B and Figure 2 B).

[0082] At a fourth step S4, the TOF system 500 obtains a set of data (hereinafter referred to as "Data B1") from the image sensor 512 and subsequently processes the set of data for frame m at step S4B. Again, for the TOF system 500, this step itself can be performed in a conventional manner, and for the illumination area (fringes) of Configuration B, relatively high-resolution and / or distance-depth data can be obtained.

[0083] At a fifth step S5, the TOF system 500 obtains a set of data (hereinafter referred to as "Data B2") from the image sensor 512. This step is similar to the previous step S4.

[0084] At a sixth step S6, the actuation mechanism 506 moves the shift lens such that the TOF system 500 emits illumination in Configuration A.

[0085] At the seventh step S7, the TOF system 500 obtains a set of data (hereinafter referred to as "data A2") from the image sensor 512, and then processes the set of data for frame n at step S7B. This step is similar to the second step S2.

[0086] The eighth step S8 illustrates whether the above steps S2 - S7 can be repeated any number of times.

[0087] The eighth step S9 illustrates that the TOF system 500 processes the data obtained in the above steps. This can be done after the above steps as shown, or during these steps. The data obtained (i.e., data A1, data A2, data B1, data B2) can be used to generate two depth (or distance) image frames (frames m and n). In particular, data A1 and data B1 can be combined ("fused") to generate a first depth image that substantially covers the entire field of view of the image sensor 512. In addition, data A2 and data B2 can be combined to generate a second depth image that substantially covers the entire field of view of the image sensor 512 (see Figure 2 C).

[0088] The actuation mechanism 506 is not used before obtaining data A1 (and the same applies when repeating steps S2 - S7), and the actuation mechanism 506 is not used before obtaining data B2. In other words, compared to a sequence such as A - B - A - B, etc., by adopting a sequence with a configuration such as A - B - B - A, etc., the frequency of movement of the actuation mechanism 506 is halved. In other words, the requirement for actuator bandwidth has been halved while ensuring that there are always adjacent A - B pairs to perform depth fusion, thus maintaining 30fps of depth map information.

[0089] As described above, moving between the two configurations requires a movement of approximately 100μm, and the typical cycle rate (60Hz) of the TOF imaging system 500 is much higher than the tropical width of the wire (7Hz).

[0090] Changing the sampling order to the above order allows the transmission of composite frames at 30 frames per second [since adjacent 33ms blocks contain A and B frames]. However, for a specific SMA actuator wire, this halves the number of required conversions and more than doubles the amount of cooling time available between conversions. This reduces the risk to the SMA actuator due to thermal soak (i.e., heat accumulated by repeated actuation when there is not enough time in the cycle to dissipate heat).

[0091] The above techniques are particularly applicable to SMA actuator technology because they have asymmetry in heating and cooling response, which means that it is desirable to have a relatively fast movement (about 5ms) every, for example, 60ms rather than every, for example, 30ms.

[0092] It should be understood that the above embodiments can have many other variations.

[0093] For example, the illumination pattern can be different (e.g., checkerboard), can have different fill factors, and / or more than two different configurations (e.g., A, B, C), in which case the sequence can correspond to A - B - C - C - B - A, etc.

[0094] Figure 4A An illumination pattern according to a second embodiment of the present invention is shown. Figure 4B The first configuration (A) and the final configuration (D) of the illumination generated by the system of FIG. 1, as well as the intermediate configurations (B, C), are shown.

[0095] In this embodiment, the illumination takes the form of a pattern of light spots evenly distributed across the field of view. Thus, the illumination can be referred to as focused illumination. However, it should be understood that any beam shape can be used as described above. The spatially non - uniform intensity of the focused illumination corresponds to a set of regions 118 in which the peak emission intensity is substantially constant. In this example, at any given time, the set of regions 118 together cover 40% to 50% of the sensor surface. The ratio of the illumination intensity at the light spots to the illumination intensity between the light spots is greater than 30, and this ratio may depend on the ambient / background noise.

[0096] The TOF system 500 is configured to move the focused illumination across at least a portion of the field of view to generate an output frame. The movement of the focused illumination is caused by the SMA actuator 506. The SMA actuator moves the focused illumination across the field of view in a scanning mode. In this example, the scanning mode 120 includes a first sequence that includes: obtaining a data set for configuration A, moving the focused illumination in a first direction to switch the illumination from the first configuration A to the first intermediate configuration B, moving the focused illumination in a second direction to switch the illumination from the first intermediate configuration B to the second intermediate configuration C, and moving the focused illumination in a third direction to switch the illumination from the second intermediate configuration C to the final configuration D. The first and third directions are parallel and perpendicular to the second direction. After each movement, a data set is obtained at each configuration.

[0097] The scanning mode 120 also includes a second sequence, in which a data set for the final configuration D of the illumination is obtained before moving the focused illumination in the first direction to switch the illumination from the final configuration D to the second intermediate configuration C. The second sequence also includes: moving the focused illumination in a fourth direction to switch the illumination from the second intermediate configuration C to the first intermediate configuration B, and moving the focused illumination in the third direction to switch the illumination from the first intermediate configuration B back to the first configuration A. The first and third directions are perpendicular to the fourth direction. After each movement, data sets are obtained at each configuration C, B, A.

[0098] Thus, in a repeating cycle, two data sets are continuously obtained at each of the first configuration A and the final configuration D, without moving the illumination between sequences. Thus, the sequence corresponds to A - B - C - D - D - C - B - A... A - B - C - D - D - C - B - A.

[0099] As shown, the order of configurations A - D in the second sequence of the scan pattern 120 is opposite to that of the first sequence. However, in other embodiments, such an opposite order may not be necessary. This can be better illustrated by the example scan pattern 122 as Figure 4A shown.

[0100] In the scan pattern 122, an optional first sequence includes moving the condenser illumination diagonally to directly switch the illumination from the first configuration A to the second intermediate configuration C, moving the condenser illumination in a fourth direction to switch the illumination from the second intermediate configuration C to the first intermediate configuration C, and moving the condenser illumination diagonally to directly switch the illumination from the first intermediate configuration B to the final configuration D. Thus, the resulting sequence corresponds to A - C - B - D - D - C - B - A... A - C - B - D - D - C - B - A.

[0101] The set of regions 118 is arranged such that the movement of the condenser illumination causes the set of regions 118 to cover more than 90% of the field of view during a cycle of the scan pattern. The set of regions 118 is also arranged such that the movement of the condenser illumination substantially avoids the regions in the set of regions 118 covering the same part of the sensor surface more than once during a cycle of the scan pattern.

[0102] As the size of the scan region 118 increases, the variation of the scan pattern may increase correspondingly. Figure 4C An example scan pattern 124 for a scan region 118 with 4×4 scan light points is shown. In sequence 1, the condenser illumination scans the scan region 118 progressively row by row. Once the data set for the last configuration in sequence 1 (the lower left light point as Figure 4C shown) is obtained, the condenser illumination advances to sequence 2, where the data set for its first configuration (the lower left light point as Figure 4C shown) is obtained again before moving the condenser illumination to scan the remaining scan region 118 column by column. Similarly, two data sets for the lower right light point (as Figure 4C shown) are continuously obtained from the last configuration of sequence 2 and the first configuration of sequence 3, without moving the condenser illumination therebetween. Likewise, two data sets for the upper right light point (as Figure 4C shown) are continuously obtained from the last configuration of sequence 3 and the first configuration of sequence 4, without moving the condenser illumination therebetween. Finally, the condenser illumination scans the scan region 118 column by column in sequence 4 and returns to the upper left light point in the final configuration (as Figure 4Cas shown). Thereafter, the scan cycle can be repeated by the start sequence 1, where a data set for the first configuration (such as the upper left light spot shown Figure 4C is obtained without moving the condenser illumination.

[0103] Thus, the example scan pattern shown Figure 4C includes four sequences, each having a different scan pattern on the scan region 118, and two data sets are obtained for each light spot.

[0104] The TOF system 500 can be adapted to different applications. For example, in some other embodiments, the illumination can be substantially uniform across substantially the entire field of view of the camera (i.e., corresponding to flood illumination), while in at least one other configuration, the illumination can be non-uniform, e.g., as described above or patterned in some other way. This can be achieved by different movements of one or more of the set of optical elements 504 relative to the above-described optical elements.

[0105] Figure 5A An SMA actuator 606 is shown for cooperative or separate use with the SMA actuator 506 in the TOF system 500 of FIG. 1. The SMA actuator 606 is configured to drive the lens 504 to move along its optical axis to achieve focusing / defocusing of the illumination. The actuator mechanism 606 described in GB 2569668 B is incorporated herein by reference.

[0106] As Figure 5A shown, the SMA actuator 606 includes a suspension system 30 that supports the lens 504 on a support structure 2. The suspension system 30 is configured to guide the movement of the lens 504 relative to the support structure 2 along the optical axis O, which is the direction of movement in this example, while restricting the movement of the lens 504 relative to the support structure 2 in other degrees of freedom.

[0107] The SMA actuator 606 also includes two segments of SMA actuator wire 40 that are arranged as follows to drive the lens 504 to move along the optical axis O. The two segments of SMA actuator wire 40 are part of a single SMA actuator wire 41 that is connected to the support structure 2 at each end by a crimp portion 42 that is fixed at opposite corners on top of the support structure. The single SMA actuator wire 41 is also connected to the lens element 20 by hooking around a protrusion 22. Thus, one end of each of the two segments of SMA actuator wire 40 is connected to the support structure 2 and the other end is connected to the lens element 20.

[0108] When viewed along the optical axis O, there is an angle of 90 degrees between the two sections of SMA actuator wire 40. In this example, the optical axis O is the direction of movement. More generally, the orientation of the two sections of SMA actuator wire 40 can be changed such that when viewed along the optical axis O, the angle therebetween has any dimension less than 180 degrees, preferably in the range of 70 degrees to 110 degrees.

[0109] The two sections of SMA actuator wire 40 drive the movement of the lens 504 along the optical axis O when a drive signal is applied, and this drive signal causes heating and cooling of the two sections of SMA actuator wire 40. The two sections of SMA actuator wire 40 are resistively heated by the drive signal and cooled due to heat conduction to the surroundings when the power of the drive signal decreases. The two sections of SMA actuator wire 40 contract when heated to drive the lens 504 to move upward along the optical axis O (in Figure 5A ). Another set of SMA actuator wires 40 can be provided in an inverted configuration to drive the movement of the lens 504 along the optical axis I in the downward direction. Alternatively, this downward movement can be provided by a biasing element such as a flexure.

[0110] Figure 5B and Figure 5C respectively show the movement of the lenses of a known TOF system and the TOF system of the second embodiment. Figure 5D and Figure 5E respectively show the illumination patterns for non-uniform (condensing) illumination and uniform (flood) illumination.

[0111] As Figure 5D shown, by controlling the actuator to move the lens 504 to focus the illumination source onto the projection surface in the field of view, condensing (focused or spatially non-uniform) illumination is achieved. In condensing illumination, one or more discrete illumination light spots can be observed.

[0112] Conversely, as Figure 5E shown, by controlling the actuator to move the lens 504 to defocus the illumination source on the projection surface through the controller, flood (defocused or spatially uniform) illumination is achieved. In flood illumination, the illumination of the entire field of view is uniform. The uniform illumination can be within the boundaries of the field of view, or it can extend beyond the boundaries of the field of view. For example, the illumination can be projected onto a surface not captured by the camera.

[0113] In a known TOF system as Figure 5B shown, the graph 710 shows the movement of the actuator between the flood position and the condensing (spot) position. That is, the sequence starts with obtaining a dataset of illumination in the first configuration A and then moves the illumination to the final configuration B to obtain another dataset. Thereafter, the illumination is moved back to the first configuration A to start another sequence. Thus, the sequence of this known method corresponds to A - B - A - B.

[0114] In a focusing SMA actuator such as actuator 606, it takes a finite time to move lens 504 between the flood position and the spot position. Additionally, around the target position, there are damped oscillations in the actuator position over time. Notably, the amplitude and decay time of the damped oscillations are greater near the flood position than at the spot position. Either way, the switching between flood illumination and spot illumination should be minimized.

[0115] Figure 5C The improved illumination movement provided by TOF system 500 is shown in graph 720, where the first sequence and the second sequence are represented by solid and dashed lines respectively. Similar to Figure 5B a known TOF system, the sequence starts with a dataset for illumination in a first configuration A and then moves the illumination to a final configuration B to obtain another dataset. However, in this embodiment, the illumination does not immediately move back to the first configuration A to start another first sequence. Instead, in the second sequence, the illustration stays at the final configuration B to obtain yet another dataset before returning to the first configuration A. Thus, two datasets are continuously obtained at each first configuration and the final configuration before moving the illumination. Therefore, the first sequence and the second sequence of this method correspond to A - B - B - A... A - B - B - A.

[0116] For different levels of focus, there can be additional configurations C, D between the first configuration A and the final configuration B, whereby the configurations in the second sequence may or may not be in the reserve order of the first sequence. Thus, such a sequence can correspond to A - C - D - B - B - D - C - A... A - C - D - B - B - D - C - A, etc.

[0117] For more and more complex illumination configurations, the optimal order corresponds to the solution of a "traveling salesman" type problem (TSP), without requiring the first and final destinations (configurations) to be the same. As long as all the points are visited, then the outbound tour (sequence) can be reversed to generate two sequences that include all the required data while giving the minimum requirements for movement. In the case of using an SMA actuator, the most suitable might be the "bottleneck traveling salesman problem", i.e., finding a route where all edges are less than or equal to x (where x is typically the distance or time of the actuator mechanism transition).

Claims

1. A method applicable to a time-of-flight (TOF) imaging system, wherein, The system emits illumination in a plurality of configurations, each configuration having a different spatially varying intensity over the field of view of an image sensor, and the method includes: Moving an actuation mechanism to change the illumination from a first configuration to a final configuration via a first sequence of configurations; Moving the actuation mechanism to subsequently change the illumination from the final configuration to the first configuration or a second configuration via a second sequence of configurations; and For each configuration in the first sequence and the second sequence, obtaining a set of data from the image sensor, thereby obtaining two sets of data suitable for generating two depth image frames for each configuration, wherein the two sets of data corresponding to the final configuration are obtained consecutively from the first sequence and the second sequence.

2. The method according to claim 1, wherein, Obtaining the two sets of data obtained consecutively for the final configuration without moving the actuation mechanism.

3. The method according to claim 1 or 2, wherein, The cumulative intensity of the illumination during the first sequence is substantially the same as the cumulative intensity of the illumination during the second sequence.

4. The method according to claim 1 or 2, wherein, The second sequence corresponds to the first sequence in reverse order.

5. The method according to claim 1 or 2, wherein, The first sequence and the second sequence include an equal number of configurations.

6. The method according to claim 1 or 2, wherein, Each of the first sequence and the second sequence includes only the first configuration and the final configuration.

7. The method according to claim 1 or 2, wherein, Each of the first sequence and the second sequence includes one or more additional configurations between the first configuration and the final configuration in the respective sequence.

8. The method according to claim 1 or 2, wherein, The intensity accumulated during each sequence is substantially uniform over substantially the entire field of view of the image sensor.

9. The method according to claim 1 or 2, wherein, For each sequence, the illumination in each configuration of the sequence substantially does not overlap with the illumination in any other configuration in the sequence.

10. The method according to claim 1 or 2, wherein, Each sequence includes at least one configuration in which the illumination is substantially non-uniform over at least a portion of the field of view of the camera.

11. The method according to claim 1 or 2, wherein, Each sequence includes at least one configuration in which the illumination is substantially uniform over substantially the entire field of view of the camera.

12. The method according to claim 11, comprising moving the actuating mechanism to focus and defocus the illumination so as to switch between uniform illumination and non-uniform illumination.

13. The method according to claim 1 or 2, wherein, The field of view of the image sensor corresponds to the field of view of the image sensor together with one or more optical elements associated with the image sensor.

14. The method according to claim 1 or 2, the method being performed in a series of sub-frames, wherein a set of data is obtained and then the actuating mechanism is moved during each sub-frame associated with each sequence except the final sub-frame.

15. The method according to claim 1, wherein, The actuation mechanism is moved within a first portion of each sub-frame in which the actuation mechanism is moved, and the data is obtained within a second portion of each sub-frame.

16. The method according to claim 15, wherein, In each final sub-frame, the data is obtained substantially throughout the sub-frame.

17. The method according to claim 15, wherein, The sub-frames have a frequency between 40 and 80 Hertz.

18. The method according to claim 17, wherein, The actuation mechanism moves at a frequency between 20 and 40 Hertz.

19. The method according to any one of claims 15 to 18, wherein, Each first portion has a first duration, and each second portion has a second duration that is longer than the first duration.

20. The method according to claim 19, wherein, The first duration is less than 10 milliseconds.

21. The method according to claim 19, wherein, The actuation mechanism includes one or more shape memory alloy components that are resistively heated to provide the movement.

22. The method according to claim 21, wherein, The one or more shape memory alloy components are resistively heated within each first portion and cooled within each second portion such that the average temperature of the wire does not substantially increase between consecutive sub-frames.

23. The method according to claim 1 or 2, wherein, The set of data obtained for each configuration includes the set of data for the first configuration obtained without previously moving the actuating mechanism.

24. The method according to claim 1 or 2, wherein, Each change in illumination includes moving the illumination through the field of view in a scanning pattern.

25. The method according to claim 24, wherein, The scanning pattern includes moving the illumination in at least two non-parallel directions across the field of view in each of the first and second sequences.

26. The method according to claim 1 or 2, wherein, The illumination includes a beam having a beam projection configured as a grid moiré, a beam having a circular or polygonal beam projection, a parallel light bar pattern, or a light pattern of dots or circles.

27. The method according to claim 1 or 2, comprising a plurality of scan cycles, wherein, Each of the plurality of scan cycles has a substantially same first configuration and a final configuration whether the first and second sequences of configurations are the same or different.

28. The method according to claim 1 or 2, wherein, The second sequence is from the final configuration to the second configuration, and the method further includes: Moving the actuating mechanism to subsequently change the illumination from the second configuration to the final configuration or a third configuration via a third sequence of configurations; and For each configuration in the third sequence of configurations, obtaining a set of data from the image sensor, wherein two sets of data corresponding to the second configuration are obtained without moving the actuating mechanism.

29. The method according to claim 28, wherein, The third sequence is from the second configuration to the third configuration, and the method further includes: Moving the actuating mechanism to subsequently change the illumination from the third configuration to the final configuration via a fourth sequence of configurations; and For each configuration in the fourth sequence of configurations, obtaining a set of data from the image sensor, wherein two sets of data corresponding to the third configuration are obtained without moving the actuating mechanism.

30. A non - transitory data carrier carrying code for causing a time - of - flight imaging system to perform the method according to any one of claims 1 to 29.

31. An apparatus for use in a time - of - flight imaging system, the apparatus being configured to perform the method according to any one of claims 1 to 29.

32. The apparatus according to claim 31, comprising: An illumination section for emitting illumination; The actuating mechanism, wherein the actuating mechanism is included in the illumination section or operatively connected to the illumination section; and An imaging section including the image sensor; At least one controller operatively connected to at least the actuating mechanism and the imaging section and configured to perform the method.

33. The apparatus according to claim 32, wherein,The actuating mechanism includes one or more shape memory alloy components.

Images