Time-of-flight imaging circuit, time-of-flight imaging system, time-of-flight imaging method

Abstract

The present disclosure generally relates to a time-of-flight imaging circuit configured to: obtain first image data from an image sensor, the first image data indicating a scene illuminated with dot light; determine a first image feature in the first image data; obtain second image data from the image sensor, the second image data indicating the scene; determine a second image feature in the second image data; estimate a motion of the second image feature relative to the first image feature; and merge the first image data and the second image data based on the estimated motion.

Description

Technical Field

[0001] The present disclosure generally relates to a time-of-flight imaging circuit, a time-of-flight imaging system, and a time-of-flight imaging method. Background Art

[0002] Generally, time-of-flight imaging systems are known. Such systems typically measure the round-trip delay of emitted light, the phase shift of the light (which can indicate the round-trip delay), the distortion of the emitted light, etc., to determine a depth map or a three-dimensional model of an object.

[0003] Generally, in order to measure or image a three-dimensional object, it is desirable to have a relatively accurate measurement output. However, in known systems, so-called multipath artifacts, aliasing effects, etc. may deteriorate such measurements.

[0004] Although there are techniques for processing time-of-flight image data, it is generally desirable to provide a time-of-flight imaging circuit, a time-of-flight imaging system, and a time-of-flight imaging method. Summary of the Invention

[0005] According to a first aspect, the present disclosure provides a time-of-flight imaging circuit configured to: obtain first image data from an image sensor, the first image data indicating a scene illuminated with a dot light; determine a first image feature in the first image data; obtain second image data from the image sensor, the second image data indicating the scene; determine a second image feature in the second image data; estimate the motion of the second image feature relative to the first image feature; and merge the first image data and the second image data based on the estimated motion.

[0006] According to a second aspect, the present disclosure provides a time-of-flight imaging system including: a dot light source configured to illuminate a scene with a dot light; an image sensor; and a time-of-flight imaging circuit configured to: obtain first image data from the image sensor, the first image data indicating the scene illuminated with the dot light; determine a first image feature in the first image data; obtain second image data from the image sensor, the second image data indicating the scene; determine a second image feature in the second image data; estimate the motion of the second image feature relative to the first image feature; and merge the first image data and the second image data based on the estimated motion.

[0007] According to a third aspect, the present disclosure provides a time-of-flight imaging method, including: obtaining first image data from an image sensor, the first image data indicating a scene illuminated with dot light; determining a first image feature in the first image data; obtaining second image data from the image sensor, the second image data indicating the scene; determining a second image feature in the second image data; estimating a motion of the second image feature relative to the first image feature; and merging the first image data and the second image data based on the estimated motion.

[0008] Other aspects are set forth in the dependent claims, the following description, and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Embodiments are explained by way of example with reference to the accompanying drawings, in which:

[0010] Figure 1 A block diagram of a time-of-flight imaging system according to the present disclosure is depicted;

[0011] Figure 2 A block diagram of a time-of-flight imaging method according to the present disclosure is depicted;

[0012] Figure 3 A block diagram of another embodiment of a time-of-flight imaging method according to the present disclosure is depicted;

[0013] Figure 4 Another embodiment of a time-of-flight imaging method according to the present disclosure is depicted;

[0014] Figure 5 A block diagram of a mobile phone according to the present disclosure is depicted;

[0015] Figure 6 A block diagram of another embodiment of a time-of-flight imaging method according to the present disclosure is depicted; and

[0016] Figure 7 A method using a reference frame is depicted. DETAILED DESCRIPTION

[0017] Before giving a detailed description of the embodiments with reference Figure 1 to, a general explanation is made.

[0018] As mentioned at the beginning, measurements with known time-of-flight systems may result in multipath artifacts, which may deteriorate the measurements, and it is generally desirable to reduce imaging artifacts.

[0019] Furthermore, in some cases, it is generally desirable to improve the resolution of the time-of-flight measurements, depth maps, and / or three-dimensional models of objects.

[0020] It has been recognized that, in the case of time-of-flight at a point, a scene can be illuminated with a finite number of light points. Accordingly, the resolution of the acquired time-of-flight image can be limited to that finite number of light points.

[0021] It has also been recognized that the resolution can be increased by increasing the number of light points, where it is desirable to maintain the size of the light source rather than increasing the size of the light source in order to increase the number of light points.

[0022] Accordingly, it has been recognized that this can be achieved by movement or motion of the light source between consecutive frames of time-of-flight image acquisition such that a wider area of the scene (and / or object) can be illuminated.

[0023] In addition, it has been recognized that the image quality can be improved by movement or motion of the image sensor. In a single time-of-flight measurement, in some cases it may not be possible to distinguish the light signal that should be measured from a reflected light signal that may have traveled a longer distance. Generally, this effect is referred to as the multipath effect.

[0024] Accordingly, it has been recognized that, as described above, the influence of the multipath effect can be reduced by considering two (or more) (consecutive) measurements from different positions, which can be based on the movement of the image sensor.

[0025] For example, by using a set of light points having a predetermined number, multipath artifacts can be identified or estimated, such that the artifacts can be removed from the obtained depth and / or from the obtained confidence value.

[0026] Multipath artifacts can be identified by considering at least one adjacent light point with respect to a light point (of a set of light points). This is possible because in some embodiments a non-continuous distribution of light (points) can be used.

[0027] In other embodiments where a continuous distribution of (modulated) light can be used (e.g., indirect time-of-flight), the movement of the imaging element (e.g., pixel) with respect to a fixed position of the object or the object can be used to identify multipath artifacts.

[0028] A continuous distribution of light can be achieved by increasing the point density of the dot-like light source such that such light points (partially) overlap. However, the present disclosure is not limited to generating a continuous distribution of light with a dot-like light source, and thus any light source can be utilized.

[0029] In some embodiments, the identification of multipath artifacts results in a reduction in resolution (of a single frame). However, the reduced resolution is compensated for by acquiring multiple frames and combining them (as discussed herein).

[0030] Accordingly, some embodiments relate to a time-of-flight imaging circuit configured to: obtain first image data from an image sensor, the first image data indicative of a scene illuminated with dot light; determine a first image feature in the first image data; obtain second image data from the image sensor, the second image data indicative of the scene; determine a second image feature in the second image data; estimate a motion of the second image feature relative to the first image feature; and merge the first image data and the second image data based on the estimated motion.

[0031] Generally, a time-of-flight imaging circuit can include any circuit configured to perform, process, evaluate, perform time-of-flight measurements, etc., such as a processor, e.g., a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), etc., where multiple such components can also be contemplated, and combinations of these components can also be contemplated. Additionally, a time-of-flight imaging circuit can be included in or can be included in a computer, a server, a camera, etc. and / or combinations thereof.

[0032] A time-of-flight imaging circuit can be configured to obtain first image data from an image sensor.

[0033] An image sensor can generally be any known image sensor, which can include one or more imaging elements (e.g., pixels) and is based on known semiconductor or diode technologies, such as complementary metal-oxide semiconductor (CMOS), charge-coupled device (CCD), current-assisted photon demodulator (CAPD), single-photon avalanche diode (SPAD), etc.

[0034] An image sensor can be configured to generate an electrical signal in response to light incident on an image plane (e.g., on one or more pixels), which is well-known, where the electrical signal can be processed such that it can indicate, generate first image data, etc.

[0035] Furthermore, the image plane can have a larger area than the image sensor. For example, the image plane can be established by moving the total area of the image sensor such that the image plane can cover at least that area (or even more).

[0036] Obtaining the first image data and / or the second image data may include sending a request to an image sensor (or any circuit coupled to the image sensor, such as a memory, etc.) to (actively) acquire the first image data, while in some embodiments, the first (and / or second) image data is (passively) received by the time-of-flight imaging circuit at a predetermined time point. This means that obtaining may further include receiving the first image data and / or the second image data in response to a request sent to the image sensor received via a bus from a data memory, etc. Generally, it should be noted that the active acquisition of the first image data may also include passive reception, i.e., the (active) request may establish the passive reception of the first image data and / or the second image data at a predetermined time point, etc.

[0037] The first (and / or second) image data may indicate a scene. The scene may include an object for which depth measurement should be (or is being) performed. Additionally, the scene may include the surroundings of the object (e.g., the background), the projected area of which on the image plane may be larger than that of the object, such that the scene can still be captured even if the image sensor does not (fully) capture the object after movement of the image sensor, etc.

[0038] Generally, the scene may be illuminated with dot light, which may come from a dot light source, such as a diode laser (or multiple diode lasers), a vertical-cavity surface-emitting laser (VCSEL), etc., which may be configured to illuminate the scene with a plurality of light dots generated by the dot light source. The dot light may be based on a predetermined pattern, where the shape and / or arrangement of the plurality of light dots may be predetermined, such that distortion, smearing, deformation, etc. of at least one of the plurality of light dots and / or the pattern (e.g., change in the distance / arrangement of different light dots) may indicate the distance or depth between the image sensor and / or the light source and the scene.

[0039] In some embodiments, the distance between (at least) two light dots may indicate the object (and / or scene) and / or the corresponding (relative or absolute) depth of the two light dots, or may indicate an image feature.

[0040] Therefore, the time-of-flight measurement according to the present disclosure may be performed under different illumination scenarios (e.g., under dark illumination conditions (e.g., with little background light), under bright illumination conditions (e.g., in sunlight), indoors, in daylight, etc.).

[0041] Additionally, different wavelength bands (or channels) may be used in the dot light source.

[0042] For example, the light source can emit different light colors (light having different wavelength ranges, e.g., infrared (wavelength range) and green (wavelength range)) in order to perform more precise feature reconstruction of features that are more sensitive to the corresponding colors (wavelength ranges). For example, human skin can have a known reflectance of infrared light (e.g., 80%), while flowers or plants can have a known reflectance of green light (e.g., 90%), such that in such an embodiment, the light source can be configured to emit infrared light and green light, without limiting the present disclosure in this regard as light of any wavelength range can be emitted. Additionally, the present disclosure is not limited to the case of two different colors as three, four, five, or more colors can also be emitted.

[0043] In the first image data, a first image feature can be determined.

[0044] As is well known, the first image data can indicate confidence and / or depth.

[0045] For example, at least one light point of the scene can be analyzed in the (first and / or second) image data for image characteristics.

[0046] Image characteristics can include the shape (of at least a part of the object and / or scene), the pattern (of at least a part of the object and / or scene), etc.

[0047] In addition, at least one point (another point or the same point) can be analyzed for image conditions.

[0048] Image conditions can include the light intensity, reflectance, scattering characteristics, etc. of the object / and / or scene.

[0049] In some embodiments, only image conditions (or more than one image condition) can be analyzed, while in some embodiments, only image characteristics (or more than one image characteristic) can be analyzed. However, in some embodiments, at least one image condition and at least one image characteristic can be analyzed.

[0050] At least one of such image conditions and / or image characteristics can correspond to the first image feature, or can be represented or included by the first image feature.

[0051] Therefore, the first image feature can include (or be based on) at least one image condition and / or at least one image characteristic.

[0052] In addition, the first image feature can be identified based on at least one image condition and / or at least one image characteristic, e.g., through artificial intelligence, an artificial neural network, which can utilize one or more machine learning algorithms to determine the first image feature, etc.

[0053] In addition, a time-of-flight imaging circuit can obtain second image data, which can be generated in a manner similar to the first image data without limiting the present disclosure in this regard.

[0054] In the second image data, second image features can be determined. The determination of the second image features can be performed in a manner similar to the determination of the first image features without limiting the present disclosure in this regard. The second image features can correspond to the first image features such that the second image features can include the same image features, where, in some embodiments, the second image features can include the first image features imaged or measured from different angles, e.g., after movement of the image sensor.

[0055] However, in some embodiments, the second image features can be different from the first image features. For example, in response to (intentional (e.g., controlled) or unintentional (e.g., hand shake)) movement of the image sensor or during (intentional (e.g., controlled) or unintentional (e.g., hand shake)) movement of the image sensor, it can be determined that the second image features were projected onto the same (set of) pixels as the first image features before the movement or were identified in the same (set of) pixels as the first image features.

[0056] However, in some embodiments, the second image features can be different from the first image features because they can be identified based on a different set of image conditions or image characteristics, or because a particular value or magnitude of a set of image conditions can be different from the value or magnitude of the first image features.

[0057] For example, if the second image features are an object pattern different from the first image features, it can indicate that different parts of the object are being imaged. This part may not have been imaged at all in the first image data or may have been imaged with a different pixel composition.

[0058] In the first case (not imaged), the resolution of the resulting image can be improved. In the second case (different pixel groups), multipath effects can be filtered.

[0059] However, in some embodiments, multipath effects can be ignored because, for example, the multipath effects are small or may have been filtered (as described above).

[0060] In addition, a correspondence between the first image features and the second image features (or a set of points that can be indicated by the first image features and / or the second image features) can be identified. For example, the first image features can indicate a set of reference points, and the second image features can indicate a second set of points compared to the set of reference points.

[0061] In addition, the identified correspondence can be used to merge at least two frames with each other based on the identified correspondence.

[0062] Thus, the image quality (e.g., resolution) can be improved.

[0063] Such a correspondence and / or the merged recognition can be iteratively repeated in some embodiments to maximize the image quality.

[0064] Based on the second image feature, the motion of the second image feature relative to the first feature can be estimated.

[0065] For example, the positions of the first image feature and the second image feature can be determined. The corresponding positions can include positions on the image plane, positions on the image sensor, positions within the scene, etc. Then the different positions can be compared such that the motion of the second image feature relative to the first image feature can be estimated or determined.

[0066] Based on the estimated motion, the first image data and the second image data can be merged. Thus, merged image data can be generated that can include the first image data and the second image feature and / or the corresponding positions of the first image data and the second image feature.

[0067] The merging can include the combination of the first image data and the second image data taking into account the estimated motion and the first image data and / or the second image feature. Thus, compared with the first image data and the second image data, the merged image data (or an image based on the merged image data) can have a higher resolution and can have fewer imaging artifacts (e.g., multipath artifacts).

[0068] In some embodiments, the motion is based on the vibration of at least one of an image sensor and a light source that generates dot light.

[0069] The vibration can be a motion caused by a vibration device such as an eccentric motor, a linear resonant actuator, etc., as is commonly known, such that it causes the motion, displacement, etc. of the image sensor and / or the light source, as discussed herein, for determining the first image feature and the second image feature.

[0070] It should be noted that the motion or movement is not limited to being caused by or based on vibration, because the time-of-flight imaging circuit can process the first image data and the second image data based on any type of motion (or even no motion).

[0071] For example, the movement can be caused by the (unintentional) shaking of a hand, the motion of a vehicle that can cause (random) movement, the vibration caused by the motor of a vehicle, etc. (in embodiments where a time-of-flight imaging circuit is provided in the vehicle).

[0072] Furthermore, the motion or movement can include a controlled (slow) motion at a predetermined position, where the amplitude of such a controlled movement is generally greater than the amplitude of the vibration.

[0073] In addition, the (point - like) light source may be adapted to illuminate an object (or scene) in such a way that different parts of the object (or scene) may be illuminated in each illumination cycle. Thus, movement can be simulated.

[0074] In some embodiments, the time - of - flight imaging circuit is further configured to perform triangulation including first image features and second image features for estimating motion and / or depth.

[0075] The triangulation may include known distances, such as reference points for further specifying the position of the second image feature, etc.

[0076] In addition, considering the respective positions of the first image feature and the second image feature, triangulation can be used to determine another image feature or the position of another image feature.

[0077] In some embodiments, as described above, a set of reference points and a second set of points can be obtained, and correspondences can be identified. Based on the identified correspondences, an essential matrix or a fundamental matrix can be determined, which can indicate the rotation and translation between the set of reference points of the second set of points, thus indicating the rotation and translation between the first image feature and the second image feature.

[0078] Based on the rotation and translation, triangulation can be performed, and the depth of the second image feature can be determined, which can be compared with the depth of the first image. In the case where the comparison of the corresponding depths is below a predetermined threshold, it can be assumed that the determined rotation and translation have a predetermined accuracy.

[0079] If the predetermined accuracy is achieved, the translation and rotation (e.g., for correcting motion distortion) can be considered to merge (e.g., blend) the first frame and the second frame (including the first image feature and the second image feature), thereby improving the resolution.

[0080] In some embodiments, as described above, the triangulation can be based on the disparity of the determined depth of the second image feature compared to the first image feature. The disparity can be defined as the disparity in depth. Thus, the difference (disparity) in the positions of the second image feature and the first image feature can be expressed as: x 2 -x 1 =(l / depth), x 1 including the position of the first image feature, x 2 including the position of the second image feature. Assuming that the first image feature and the second image feature correspond (correspond to each other), and assuming a constant depth, the position of the first image feature can be determined, for example.

[0081] In some embodiments, the time - of - flight imaging circuit is further configured to match the first image feature and the second image feature.

[0082] If the first image feature and the second image feature are the same but are shifted due to displacement caused by motion, the matching as described above can be performed.

[0083] The matching can be performed based on a second image feature having the same (or similar, e.g., below a predetermined threshold) image condition or image characteristic as the first image feature.

[0084] For example, when the light intensity of the second image feature is within a predetermined threshold of the light intensity of the first image feature, the second image feature can be matched with the first image feature.

[0085] In some embodiments, the time-of-flight imaging circuit is further configured to determine a first depth based on the matching.

[0086] Since the first image feature and the second image feature can correspond to each other, but their positions can be based on displacement as described herein, the corresponding features can be symbolically represented in different (local) coordinate systems.

[0087] Based on such a coordinate system, the depth or distance can be determined more precisely (more precisely than known in the art using only one time-of-flight measurement), such that based on the matching, the distance can be determined, for example, by taking the (weighted) average of the distance of the first image feature and the distance of the second image feature.

[0088] In addition, two (or at least two) local coordinate systems and the corresponding positions of the first image feature and the second image feature can be considered to determine the first distance in a global coordinate system or an image sensor coordinate system.

[0089] In some embodiments, the time-of-flight imaging circuit is further configured to determine at least one third image feature based on third image data indicating a second depth.

[0090] The third image feature can be another image feature different from the first image feature and / or the second image feature, which can be, for example, a different feature of an object, a pattern, etc. (and thus can be indicated by different imaging conditions and / or imaging characteristics). The third image feature can have a second depth.

[0091] It should be noted that the second depth can generally be (roughly) the same as the first depth, but as indicated, it may be found at different positions of the object and / or scene.

[0092] In some embodiments, the time-of-flight circuit is further configured to determine a third depth based on the first depth and the second depth.

[0093] The third depth can be determined based on the processing of the first depth and the second depth, and thus, further time-of-flight measurements may not have to be performed, such that the third depth can indicate a further or fourth (virtual) image feature. In some embodiments, the third depth can be determined by interpolation including the first depth and the second depth.

[0094] In some embodiments, the time-of-flight imaging circuit is further configured to temporally align the first image data and the second image data based on the estimated motion.

[0095] Since the estimated motion can be represented in terms of velocity, acceleration, rate, etc., but can be (symbolically) described or interpreted as a vector between a first image feature and a second image feature, where the time between the determination (or acquisition) of the first image feature and the second image feature can be known, the first image feature and the second image data can be temporally aligned, thereby simplifying the resulting image (or depth measurement) and improving the accuracy of the measurement.

[0096] As described herein, some embodiments relate to a time-of-flight imaging system, including: a point light source configured to illuminate a scene with the point light source; an image sensor; and a time-of-flight imaging circuit configured to: obtain first image data from the image sensor, the first image data indicating the scene illuminated with the point light; determine a first image feature in the first image data; obtain second image data from the image sensor, the second image data indicating the scene; determine a second image feature in the second image data; estimate the motion of the second image feature relative to the first image feature; and merge the first image data and the second image data based on the estimated motion.

[0097] Generally, a time-of-flight imaging system may include other elements, such as a lens (stack), etc., as they are generally known, and thus descriptions of such known components are omitted.

[0098] The elements of the time-of-flight imaging system (point light source, image sensor, time-of-flight imaging circuit, etc.) can be distributed in several subsystems, or can be provided in an integrated system, such as a time-of-flight camera, a mobile phone, an automobile, etc.

[0099] For example, if the time-of-flight imaging system is a mobile phone, such that the application of the time-of-flight imaging system can be regarded as three-dimensional scanning, registration, and / or identification of an object, time-of-flight acquisition can be performed within a predetermined distance between the mobile phone and the object.

[0100] A mobile phone can be set with a trigger (e.g., a virtual or physical button) for initiating 3D acquisition. In response to the trigger, vibration of the mobile phone can be initiated. The vibration can continue until a frame for acquiring (or extracting) time-of-flight measurements (e.g., for 10 seconds) is obtained. During such an acquisition cycle, a time-of-flight imaging method (described below) can be performed.

[0101] In addition, in some embodiments, further acquisitions can be initiated from different angles (or viewpoints) relative to the object.

[0102] Thus, a 3D model of the object including a mesh, shading, and / or texture, etc. can be generated.

[0103] However, the present disclosure is not limited to multiple acquisitions, such that the time-of-flight imaging method (discussed below) can also be based on a single acquisition (a single shot), and for example, for identification, face authentication, etc.

[0104] In some embodiments, as described herein, the time-of-flight imaging system further includes: a vibration device configured to generate vibration of the time-of-flight imaging system, wherein the vibration indicates the movement of a second image feature relative to a first image feature.

[0105] As described above, the vibration device can include an eccentric motor, a linear resonant actuator, etc.

[0106] As discussed herein, some embodiments relate to a time-of-flight imaging method, including: obtaining first image data from an image sensor, the first image data indicating a scene illuminated with dot light; determining a first image feature in the first image data; obtaining second image data from the image sensor, the second image data indicating the scene; determining a second image feature in the second image data; estimating the movement of the second image feature relative to the first image feature; and merging the first image data and the second image data based on the estimated movement.

[0107] The time-of-flight imaging method according to the present disclosure can be performed by a time-of-flight imaging circuit according to the present disclosure, a time-of-flight system according to the present disclosure, etc.

[0108] In some embodiments, as discussed herein, the motion is based on the vibration of at least one of an image sensor and a light source that generates dot light. In some embodiments, as discussed herein, the time-of-flight method further includes performing triangulation including first image features and second image features for estimating motion. In some embodiments, as discussed herein, the time-of-flight method further includes matching the first image features and the second image features. In some embodiments, as discussed herein, the time-of-flight method further includes determining a first depth based on the matching. In some embodiments, as discussed herein, the time-of-flight method further includes determining at least one third image feature based on third image data indicative of a second depth. In some embodiments, as discussed herein, the time-of-flight method further includes determining a third depth based on the first depth and the second depth. In some embodiments, as discussed herein, the third depth is based on an interpolation including the first depth and the second depth. In some embodiments, as discussed herein, the time-of-flight method further includes temporally aligning the first image data and the second image data based on the estimated motion.

[0109] The methods described herein are also implemented as computer programs in some embodiments, which, when executed on a computer and / or a processor, cause the computer and / or the processor to execute the methods.

[0110] In some embodiments, a non-transitory computer-readable recording medium storing a computer program product is also provided, which, when executed by a processor (such as the above-mentioned processor), causes the execution of the methods described herein.

[0111] Return Figure 1 , depicts a block diagram of a time-of-flight imaging system 1 according to the present disclosure.

[0112] The time-of-flight imaging system 1 has a lens stack 2 configured to focus light onto an image sensor 3, as discussed herein.

[0113] In addition, as discussed herein, the time-of-flight imaging circuit 4 can obtain (first and second) image data from the image sensor 3, determine first image features and second image features, estimate the motion of the second image features relative to the first image features, and merge the first image features and the second image data based on the estimated motion, as discussed herein.

[0114] As discussed herein, the time-of-flight imaging system 1 further includes a dot light source 5 and a vibration device 6.

[0115] Figure 2 Depicts a block diagram of a time-of-flight imaging method 10 according to the present disclosure.

[0116] In 11, as discussed herein, first image data is obtained from an image sensor, where the first image data indicates a scene illuminated with dot light. In the present embodiment, a time-of-flight imaging circuit configured to perform the time-of-flight imaging method 10 is connected to the image sensor via a bus, such that the image sensor transmits the first image data to the time-of-flight imaging circuit.

[0117] In 12, a first image feature is determined in the first image data by a pattern recognition algorithm implemented in the time-of-flight imaging circuit.

[0118] In 13, second image data is obtained from the image sensor via the bus.

[0119] In 14, as discussed herein, a second image feature is determined in the second image data by the pattern recognition algorithm.

[0120] In 15, the motion of the second image feature relative to the first image feature is estimated by comparing the position of the second image feature relative to the first image feature.

[0121] In 16, as discussed herein, the first image data and the second image data are merged based on the estimated motion.

[0122] Figure 3 Another embodiment of the time-of-flight imaging method 20 according to the present disclosure is depicted in the block diagram.

[0123] The time-of-flight imaging method 20 is different from the time-of-flight imaging method 10 described with respect to Figure 2 in that motion is detected based on triangulation, the first image data and the second image data are temporally aligned, and a third image feature is determined based on a first depth and a second depth.

[0124] In the present embodiment, motion is determined based on confidence data, which is generally known in the time-of-flight field. Based on the confidence data, triangulation is performed to determine depth.

[0125] In 21, as discussed herein, first image data is obtained from an image sensor, where the first image data indicates a scene illuminated with dot light. In the present embodiment, the image sensor and the time-of-flight imaging circuit performing the time-of-flight imaging method 20 are connected via a bus, and the image sensor transmits the first image data through the bus.

[0126] In 22, a first image feature is determined in the first image data by a pattern recognition algorithm implemented in the time-of-flight imaging circuit.

[0127] In 23, second image data is obtained from the image sensor via the bus.

[0128] In 24, second image features are determined in the second image data by a pattern recognition algorithm.

[0129] In 25, triangulation including the first image features and the second image features is performed. That is, based on the positions of the reference points and the first image features, the positions of the second image features are determined.

[0130] In 26, the motion of the second image features relative to the first image features is estimated based on the triangulation.

[0131] In 27, the first image features and the second image features are matched. That is, based on the motion, the corresponding positions of the first image features and the second image features are transformed into a global coordinate system.

[0132] In 28, the first depth is determined based on the matching because the multipath effect is eliminated in 27 by transforming the first image features and the second image features into a global coordinate system.

[0133] In 29, as discussed herein, the first image data and the second image data are aligned in time based on the motion.

[0134] In 30, the first image data and the second image data are merged based on the estimated motion such that the resulting merged image data has the determined depth based on a point in the global coordinate system.

[0135] In 31, at least one third image feature is determined based on third image data indicating a second depth. The third image feature is determined as the first image feature and / or the second image feature. However, the third image feature is at a different position of the object and thus is different from the first image feature and the second image feature.

[0136] In 32, a third depth is determined based on the first depth and the second depth and the interpolation between the first depth and the second depth.

[0137] Figure 4 A time-of-flight imaging method 40 according to the present disclosure is depicted. The time-of-flight imaging method 40 is different from the previous embodiment of the time-of-flight imaging method 20 in that it is performed by a mobile phone including a vibration device.

[0138] The mobile phone 41 includes a time-of-flight imaging system 42. It should be noted that in this embodiment, the vibration device is not included in the time-of-flight imaging system 42 but is included in the mobile phone 41 such that the time-of-flight imaging system 42 experiences motion when the mobile phone 41 vibrates.

[0139] As is generally known and described herein, from the mobile phone 41 (and the time-of-flight imaging system 42), initial motion and position information, confidence data, and depth data are determined in 43.

[0140] Based on the initial motion and position information (as described above) used as a reference point in the global coordinate system, the further position of the time-of-flight image sensor of the time-of-flight imaging system 42 is determined in 44.

[0141] In addition, for a plurality of confidence sub-frames 45 continuously acquired at time points t, t + 1, and t + 2, a confidence value of the time-of-flight measurement is determined. For a plurality of depth sub-frames 46 continuously acquired approximately at time points t, t + 1, and t + 2, a depth value is determined.

[0142] Based on the sensor position and movement and the previously measured confidence value and reference frame 47 at time point t - 1, motion estimation is performed in 48.

[0143] In addition, in 49, the confidence values determined based on the confidence sub-frames 45 at time points t, t + 1, and t + 2 and the confidence values of the reference frame are matched and triangulated.

[0144] In 50, the matched confidence values are compared with the confidence values of the reference frame. In addition, the depth values of the depth sub-frames 46 are compared with the depth values from the reference frame. Based on these comparisons, further refinement of the measurement is performed.

[0145] If the comparisons in 50 cause the confidence values of the confidence sub-frames 45 and the depth values of the depth sub-frames 46 to converge to the confidence values and depth values of the reference frame, then in 51, each confidence value and depth value of the sub-frames 45 and 46 are associated with the estimated motion, and based on this association, temporal alignment and spatio-temporal (after temporal alignment and after determining the second depth, as described above) interpolation between the determined confidence values and depth values are performed, as discussed herein.

[0146] The sub-frames 45 and 46 are processed as confidence frames and depth frames, which then (together) serve as a reference frame T for subsequent measurements, as shown in 52.

[0147] Figure 5 A block diagram of a mobile phone 60 (as a time-of-flight imaging system) is depicted. The mobile phone 60 includes a vibration device 61, an inertial measurement unit (IMU) 62, a dot light source 63, a time-of-flight image sensor 64, and a control circuit 65 for controlling vibration, for controlling the timing of the dot light source 63, and / or for controlling the time-of-flight image sensor 64. In addition, the control circuit 65 is adapted to be a time-of-flight imaging circuit according to the present disclosure.

[0148] The dot light source 63 is a dot projector configured to project a grid of small infrared dots (or light dots) onto an object (or scene, as described above), and in this embodiment includes a plurality of vertical cavity surface emitting lasers (VCSELs) for projecting the grid.

[0149] With such a configuration, three-dimensional scanning and registration of an object can be achieved, where multipath effects are minimized and where geometric photometric (brightness) resolution is achieved.

[0150] Accordingly, object recognition (e.g., face recognition) can be effectively performed.

[0151] Figure 6 A block diagram depicting another embodiment of a time-of-flight imaging method 70 is shown.

[0152] In 71, an image plane (e.g., an image sensor) including a plurality of first image features 72 based on first image data and a plurality of second image features 73 based on second image data is shown. The first image features 72 and the second image features 73 correspond to light spots projected from a light source onto an object, where the light spots are captured by the image sensor and analyzed by a time-of-flight imaging circuit to identify the first image features 72 and the second image features 73.

[0153] Accordingly, as discussed herein, the first image features 72 and the second image features 73 of the light spots are attributed to the motion of the vibration-based light source.

[0154] In 74, based on the first image features 72 and the second image features 73, a global motion estimation of the second image features 73 relative to the first image features 72 is performed, and the second image features are aligned on the image plane 71 based on the estimated motion.

[0155] In 75, inpainting, interpolation, and filtering are performed to improve resolution and filter artifacts.

[0156] In 76, an image frame is output.

[0157] The output image frame is then used as a reference image frame for continuous measurements, as described above.

[0158] Figure 7 A method 80 for using a reference frame is shown.

[0159] A plurality of sub-frames 81 acquired at time points t to t + 5 are shown.

[0160] For the first image frame, the first three sub-frames (t, t + 1, and t + 2) are considered such that the first image frame (frame 1) is output at a first output time T. For the second image frame, the third to sixth sub-frames (t + 2, t + 3, t + 4, t + 5) are considered such that the second image frame (frame 2) is output at a second output time T + 1 using frame 1 as a reference frame.

[0161] It should be recognized that the embodiments describe methods having an exemplary order of method steps. However, the specific order of the method steps is given for illustrative purposes only and should not be construed as being binding. For example, the order of 11 and 13 in the embodiment of Figure 2 can be swapped. In addition, the order of 12 and 14 in the embodiment of Figure 2 can be swapped. In addition, the order of 29 and 31 in the embodiment of Figure 3 can also be swapped. Other variations in the order of the method steps may be obvious to those skilled in the art.

[0162] Note that the division of the time-of-flight imaging system 60 into units 62 and 65 is for illustrative purposes only, and the present disclosure is not limited to any specific functional division in a particular unit. For example, the control circuit 65 and the IMU 62 can be implemented by corresponding programmed processors, field-programmable gate arrays (FPGAs), etc.

[0163] The method can also be implemented as a computer program that, when executed on a computer and / or processor, causes the computer and / or processor (such as the time-of-flight imaging circuit 4 discussed above) to execute the method. In some embodiments, a non-transitory computer-readable recording medium storing a computer program product is also provided, which, when executed by a processor (such as the above-mentioned processor), causes the described method to be executed.

[0164] Unless otherwise specified, all units and entities described in this specification and claimed in the appended claims can be implemented as integrated circuit logic, for example on a chip, and unless otherwise specified, the functions provided by such units and entities can be implemented by software.

[0165] Insofar as the above-described embodiments are implemented at least in part using a software-controlled data processing device, it should be understood that providing such a software-controlled computer program and the transmission, storage, or other medium providing such a computer program are contemplated aspects of the present disclosure.

[0166] Note that the present technology can also be configured as described below.

[0167] (1) A time-of-flight imaging circuit configured to:

[0168] Obtain first image data from an image sensor, the first image data indicating a scene illuminated with a dot light;

[0169] Determine a first image feature in the first image data;

[0170] Obtain second image data from the image sensor, the second image data indicating the scene;

[0171] Determine second image features in the second image data;

[0172] Estimate the motion of the second image features relative to the first image features; and

[0173] Merge the first image data and the second image data based on the estimated motion.

[0174] (2) The time-of-flight imaging circuit according to (1), wherein the motion is based on the vibration of at least one of an image sensor and a light source that generates dot light.

[0175] (3) The time-of-flight imaging circuit according to any one of (1) and (2), further configured to perform triangulation including the first image features and the second image features for estimating the motion.

[0176] (4) The time-of-flight imaging circuit according to any one of (1) to (3), further configured to match the first image features and the second image features.

[0177] (5) The time-of-flight imaging circuit according to any one of (1) to (4), further configured to determine a first depth based on the matching.

[0178] (6) The time-of-flight imaging circuit according to (5), further configured to determine at least one third image feature based on third image data indicating a second depth.

[0179] (7) The time-of-flight imaging circuit according to (6), further configured to determine a third depth based on the first depth and the second depth.

[0180] (8) The time-of-flight imaging circuit according to (7), wherein the determination of the third depth is based on interpolation including the first depth and the second depth.

[0181] (9) The time-of-flight imaging circuit according to any one of (1) to (8), further configured to temporally align the first image data and the second image data based on the estimated motion.

[0182] (10) A time-of-flight imaging system, comprising:

[0183] A dot light source configured to illuminate a scene with dot light;

[0184] An image sensor; and

[0185] A time-of-flight imaging circuit configured to:

[0186] Obtain first image data from the image sensor, the first image data indicating a scene illuminated with dot light;

[0187] Determine first image features in the first image data;

[0188] Obtain second image data from an image sensor, the second image data indicating a scene;

[0189] Determine a second image feature in the second image data;

[0190] Estimate the motion of the second image feature relative to the first image feature; and

[0191] Merge the first image data and the second image data based on the estimated motion.

[0192] (11) The time-of-flight imaging system according to (10) further includes: a vibration device configured to generate vibrations of the time-of-flight imaging system, wherein the vibrations indicate the motion of the second image feature relative to the first image feature.

[0193] (12) A time-of-flight imaging method includes:

[0194] Obtain first image data from an image sensor, the first image data indicating a scene illuminated with dot light;

[0195] Determine a first image feature in the first image data;

[0196] Obtain second image data from an image sensor, the second image data indicating a scene;

[0197] Determine a second image feature in the second image data;

[0198] Estimate the motion of the second image feature relative to the first image feature; and

[0199] Merge the first image data and the second image data based on the estimated motion.

[0200] (13) The time-of-flight imaging method according to (12), wherein the motion is based on vibrations of at least one of the image sensor and a light source that generates dot light.

[0201] (14) The time-of-flight imaging method according to any one of (12) and (13) further includes:

[0202] Perform triangulation including the first image feature and the second image feature for estimating the motion.

[0203] (15) The time-of-flight imaging method according to any one of (12) to (14) further includes:

[0204] Match the first image feature and the second image feature.

[0205] (16) The time-of-flight imaging method according to (15) further includes:

[0206] Determine a first depth based on a match.

[0207] (17) The time-of-flight imaging method according to (16) further includes:

[0208] Determine at least one third image feature based on third image data indicating a second depth.

[0209] (18) The time-of-flight imaging method according to (17) further includes:

[0210] Determine a third depth based on the first depth and the second depth.

[0211] (19) The time-of-flight imaging method according to (18), wherein the third depth is based on an interpolation including the first depth and the second depth.

[0212] (20) The time-of-flight imaging method according to any one of (12) to (19) further includes:

[0213] Temporally align the first image data and the second image data based on the estimated motion.

[0214] (21) A computer program including program code which, when executed on a computer, causes the computer to execute the method according to any one of (11) to (20).

[0215] (22) A non-transitory computer-readable recording medium storing a computer program product which, when executed by a processor, causes the execution of the method according to any one of (11) to (20).

Claims

1. A time-of-flight imaging circuit, configured to: Obtain first image data from an image sensor, the first image data indicating a scene illuminated with dot light; Determine a first image feature in the first image data; Obtain second image data from the image sensor, the second image data indicating the scene; Determine a second image feature in the second image data; Estimate the motion of the second image feature relative to the first image feature; And Merge the first image data and the second image data based on the estimated motion; Wherein the motion is based on the vibration of at least one of the image sensor and a light source that generates the dot light, Wherein the motion includes rotation and translation between the first image feature and the second image feature, And wherein merging the first image data and the second image data based on the estimated motion includes: when the rotation and the translation reach a predetermined accuracy, being able to consider the rotation and the translation to merge the first image feature and the second image feature.

2. The time-of-flight imaging circuit according to claim 1, further configured to perform triangulation of the first image feature and the second image feature including for estimating the motion.

3. The time-of-flight imaging circuit according to claim 1, further configured to match the first image feature and the second image feature.

4. The time-of-flight imaging circuit according to claim 3, further configured to determine a first depth based on the matching.

5. The time-of-flight imaging circuit according to claim 4, further configured to determine at least one third image feature based on third image data indicating a second depth.

6. The time-of-flight imaging circuit according to claim 5, further configured to determine a third depth based on the first depth and the second depth.

7. The time-of-flight imaging circuit according to claim 6, Wherein, The determination of the third depth is based on interpolation including the first depth and the second depth.

8. The time-of-flight imaging circuit according to claim 1, further configured to temporally align the first image data and the second image data based on the estimated motion.

9. A time-of-flight imaging system, Comprising: A dot light source, configured to illuminate a scene with dot light; An image sensor; And A time-of-flight imaging circuit, configured to: Obtain first image data from an image sensor, the first image data indicating the scene illuminated with the dot light; Determine a first image feature in the first image data; Obtain second image data from the image sensor, the second image data indicating the scene; Determine a second image feature in the second image data; Estimate the motion of the second image feature relative to the first image feature; And Merge the first image data and the second image data based on the estimated motion; Wherein the motion is based on the vibration of at least one of the image sensor and a light source that generates the dot light, Wherein, the motion includes rotation and translation between the first image feature and the second image feature, and wherein, merging the first image data and the second image data based on the estimated motion includes: when the rotation and the translation reach a predetermined accuracy, being able to consider the rotation and the translation to merge the first image feature and the second image feature.

10. The time-of-flight imaging system according to claim 9, further comprising: a vibration device configured to generate vibration of the time-of-flight imaging system, wherein the vibration indicates the motion of the second image feature relative to the first image feature.

11. A time-of-flight imaging method, comprising: obtaining first image data from an image sensor, the first image data indicating a scene illuminated with dot light; determining a first image feature in the first image data; obtaining second image data from the image sensor, the second image data indicating the scene; determining a second image feature in the second image data; estimating the motion of the second image feature relative to the first image feature; and merging the first image data and the second image data based on the estimated motion; wherein the motion is based on vibration of at least one of the image sensor and a light source generating the dot light, wherein the motion includes rotation and translation between the first image feature and the second image feature, and wherein, merging the first image data and the second image data based on the estimated motion includes: when the rotation and the translation reach a predetermined accuracy, being able to consider the rotation and the translation to merge the first image feature and the second image feature.

12. The time-of-flight imaging method according to claim 11, further comprising: performing triangulation of the first image feature and the second image feature including for estimating the motion.

13. The time-of-flight imaging method according to claim 11, further comprising: matching the first image feature and the second image feature.

14. The time-of-flight imaging method according to claim 13, further comprising: determining a first depth based on the matching.

15. The time-of-flight imaging method according to claim 14, further comprising: determining at least one third image feature based on third image data indicating a second depth.

16. The time-of-flight imaging method according to claim 15, further comprising: determining a third depth based on the first depth and the second depth.

17. The time-of-flight imaging method according to claim 16, wherein, the third depth is based on interpolation including the first depth and the second depth.

18. The time-of-flight imaging method according to claim 11, further comprising: temporally aligning the first image data and the second image data based on the estimated motion.

Images