Analysis unit, time-of-flight imaging device and method

Abstract

The present disclosure relates to an analysis unit for a time-of-flight imaging unit, wherein the time-of-flight imaging unit includes at least one first type of imaging element and at least one second type of imaging element, wherein the at least one first type of imaging element and the at least one second type of imaging element are arranged in a predetermined pattern, and the analysis unit is configured to: construct first imaging data of at least one first type of imaging element based on second imaging element data of at least one second type of imaging element, wherein the first imaging data is constructed based on a machine learning algorithm.

Description

Technical Field

[0001] The present disclosure generally relates to an analysis section for a time-of-flight imaging section, a time-of-flight imaging apparatus, and a method for controlling a time-of-flight imaging section. Background Art

[0002] Typically, time-of-flight (ToF) devices are known for use, for example, in imaging or creating depth maps of scenes such as objects, people, etc. Direct ToF (dToF) and indirect ToF (iToF) can be distinguished to measure distance by measuring the time of flight of emitted and reflected light (dToF) or by measuring one or more phase shifts of emitted and reflected light (iToF).

[0003] To measure distance, known time-of-flight devices need to go through thousands or millions of measurement cycles, which can result in a time-consuming process. Furthermore, in order to reduce the number of measurement cycles while maintaining a sophisticated imaging chip capable of acquiring information other than depth / distance (such as color), complex algorithms must be found to demosaic the raw imaging data.

[0004] Therefore, it is generally desirable to provide an analysis section for a time-of-flight imaging section, a time-of-flight imaging apparatus, and a method for controlling an analysis section for a time-of-flight imaging section. Summary of the Invention

[0005] According to a first aspect, the present disclosure provides an analysis unit for a time-of-flight imaging unit, wherein the time-of-flight imaging unit includes at least one first type of imaging element and at least one second type of imaging element, wherein the at least one first type of imaging element and the at least one second type of imaging element are arranged in a predetermined pattern, wherein the analysis unit is configured to: construct first imaging data of at least one first type of imaging element based on second imaging element data of at least one second type of imaging element, wherein the first imaging data is constructed based on a machine learning algorithm.

[0006] According to a second aspect, the present disclosure provides a time-of-flight imaging device, comprising: a time-of-flight imaging unit, comprising at least one first-type imaging element and at least one second-type imaging element, wherein the at least one first-type imaging element and the at least one second-type imaging element are arranged in a predetermined pattern; and an analysis unit for the time-of-flight imaging unit, configured to: construct first imaging data of at least one first-type imaging element based on second imaging element data of at least one second-type imaging element, wherein the first imaging data is constructed based on a machine learning algorithm.

[0007] According to a third aspect, the present disclosure provides a method for controlling an analysis unit for a time-of-flight imaging unit, wherein the time-of-flight imaging unit includes at least one first type of imaging element and at least one second type of imaging element, wherein the at least one first type of imaging element and the at least one second type of imaging element are arranged in a predetermined pattern, and the method includes: constructing first imaging data of at least one first type of imaging element based on second imaging element data of at least one second type of imaging element, wherein the first imaging data is constructed based on a machine learning algorithm.

[0008] Further aspects are set out in the dependent claims, the following description and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The embodiments are explained by way of example with reference to the accompanying drawings, in which:

[0010] Figure 1 Six embodiments of the ToF imaging section are shown;

[0011] Figure 2 An embodiment of a ToF imaging device is shown;

[0012] Figure 3 A method for constructing imaging data is shown;

[0013] Figure 4 A representation of the mosaicked raw data is shown;

[0014] Figure 5 A first example of first and second imaging data and output data is shown;

[0015] Figure 6 A second example of first and second imaging data and output data is shown;

[0016] Figure 7 is a perspective view depicting a first example of an external configuration of a stacked image sensor;

[0017] Figure 8 is a perspective view depicting a second example of the external configuration of the stacked image sensor;

[0018] Figure 9 is a block diagram depicting a configuration example of a peripheral circuit;

[0019] Figure 10 is an abstract diagram of an embodiment of a time-of-flight device; and

[0020] Figure 11 A flow chart illustrating an embodiment of a method according to the present disclosure is shown. DETAILED DESCRIPTION

[0021] In giving reference Figure 1 Before describing the embodiments in detail, a general explanation is given.

[0022] As explained at the outset, it is often desirable to have a small number of imaging cycles (eg one). Therefore, it has been recognized that complex algorithms must be found to be able to demosaic raw imaging data acquired with a small number of imaging cycles.

[0023] Therefore, some embodiments relate to an analysis unit for a time-of-flight imaging unit, wherein the time-of-flight imaging unit includes at least one first type of imaging element and at least one second type of imaging element, wherein the at least one first type of imaging element and the at least one second type of imaging element are arranged in a predetermined pattern, wherein the analysis unit is configured to: construct first imaging data of at least one first type of imaging element based on second imaging element data of at least one second type of imaging element, wherein the first imaging data is constructed based on a machine learning algorithm.

[0024] Typically, the analysis unit can be provided by (any) circuit system configured to perform the method as described herein (such as any device, chip, etc. that can be configured to analyze (imaging) data, etc.), and the portion may include one or more processors, circuits, circuit systems, etc., which may also be distributed in the time-of-flight device or the imaging unit. For example, the analysis unit (circuit system) can be a processor, such as a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), etc., or several units of a CPU, GPU, FPGA (also in combination). In other embodiments, the analysis unit (circuit system) is a (personal) computer, a server, an AI accelerator, etc.

[0025] The time-of-flight imaging portion can be implemented as a camera, for example, as a standalone device, or it can be combined with other camera technologies to create a depth map. The time-of-flight device can also be included in or integrated into another device, such as a smartphone, tablet, handheld computer, camera system, etc.

[0026] Generally, the present technology may also be implemented in any field of technology using time-of-flight technology, such as automotive technology, traffic systems, etc.

[0027] Embodiments of the time-of-flight imaging portion may be based on different time-of-flight (ToF) technologies. Generally, ToF devices can be divided into two main technologies, namely indirect ToF (iToF) and direct ToF (dToF), as described above.

[0028] The time-of-flight imaging unit based on iToF technology can indirectly obtain depth measurement by recovering the phase of the correlation wave, which represents the phase shift between the modulated emission light and the light received from the scene reflection. The analysis unit (for example, a unit configured in the iToF pixel sensor or configured to read a signal from the iToF pixel sensor) demodulates the illumination modulation period reflected from the scene to sample the correlation wave (between the emitted modulated light signal and the received demodulated light signal or a signal indicating the emitted modulated light signal and the received demodulated light signal), which is based on the correlation obtained by correlating the emission light and the detection light.

[0029] In some embodiments, time-of-flight imaging based on dToF technology directly obtains depth measurements by measuring the time of flight of photons emitted by a light source and reflected from a scene (eg, based on hundreds of short illumination pulses emitted).

[0030] Typically, the imaging element can be based on any type of known sensing technology for time-of-flight systems and can be based on, for example, complementary metal oxide semiconductor (CMOS), charge coupled device (CCD), single photon avalanche diode (SPAD), current assisted photodiode (CAPD) technology, etc., where SPAD can be used for dToF based technology and CAPD can be used for iToF based technology.

[0031] In addition, the time-of-flight imaging portion may include an imaging element (e.g., a single pixel) or multiple imaging elements (e.g., multiple pixels), which, as is generally known, may be arranged in an array, pattern, etc. Specifically, the ToF imaging portion may have a small number of imaging elements (pixels) (e.g., 64×64 pixels), but in other embodiments, the number of pixels may be smaller (e.g., 32×32, 16×16, 16×32, etc.) or larger (e.g., 128×128 pixels, 128×256, 256×256, etc.).

[0032] The imaging elements may also be grouped into, for example, predetermined imaging element groups (eg, four, eight, etc.), which are specifically arranged, for example, in rows, columns, squares, rectangles, etc.

[0033] In some embodiments, a predetermined group of imaging elements may share circuitry, for example, circuitry configured to read out information generated by the imaging element (such as an analysis unit). In addition, in some embodiments, an imaging element includes two or more pixels, and these pixels may share circuitry for reading out pixel information.

[0034] The first type of imaging element and the second type of imaging element may generally be imaging elements used for the same purpose. For example, as described above, both imaging elements may be used to measure the phase of the relevant wave. In these embodiments, the first type of imaging element and the second type of imaging element may each be an iToF pixel sensor indicating phase information, wherein the signal caused by the first type of imaging element (i.e., first imaging element data) may indicate the first phase information and the signal caused by the second type of imaging element (i.e., second imaging element data) may indicate the second phase information. In other embodiments, the first type of imaging element and the second type of imaging element are used for different purposes. For example, the first type of imaging element may provide information indicating the phase (first imaging element data), while the second type of imaging element may provide information indicating the color or any other signal from the spectrum (such as infrared, ultraviolet, etc.) (second imaging element data).

[0035] As described above, in some embodiments, at least one imaging element of the first type and at least one imaging element of the second type are arranged in a predetermined pattern.

[0036] This arrangement may be based on the manufacturing process, such as the production of a chip for a time-of-flight imaging unit. In other embodiments, the arrangement may be specified after the manufacturing process. For example, in an embodiment including imaging elements that indicate phases, which imaging element is assigned to which phase may be determined after manufacturing, for example, through wiring or programming of the imaging elements.

[0037] In this context, the pattern of arrangement can be predetermined. The pattern can be a row arrangement of two or more phases, colors, etc., a checkerboard arrangement of two phases, colors, etc., a grid arrangement of two or more phases, colors, etc. (such as a quincunx grid), a random pattern of at least two phases, colors, etc. (for example, generated by a random generator, etc.), or any other regular or irregular pattern.

[0038] The pattern may be selected (dynamically) depending on the imaging situation (eg, dark, bright, etc.) or depending on the scene (eg, lots of motion).

[0039] The construction of first imaging data can refer to an algorithm, program, or the like that processes imaging data to generate new imaging data. In this context, in some embodiments, at least one imaging element of the first type and at least one imaging element of the second type are alternately driven, i.e., while at least one imaging element of the second type is turned on or modulated and acquiring second imaging element data, at least one imaging element of the first type is turned off. This results in missing first imaging data, which in turn results in, for example, an image with missing pixels, i.e., missing information from pixels of other types. Thus, the first imaging data is constructed based on the second imaging element data.

[0040] Thereby, missing imaging data can be acquired.

[0041] This construction is based on a machine learning algorithm. In some embodiments, an algorithm derived from a machine learning process is applied to the second imaging element data to construct the first imaging data. Various machine learning algorithms (such as supervised learning, semi-supervised learning, unsupervised learning, reinforcement learning, feature learning, sparse dictionary learning, anomaly detection learning, decision tree learning, association rule learning, etc.) can be applied to construct the first imaging data.

[0042] The machine learning algorithm may further be based on at least one of the following: feature extraction techniques, classifier techniques, and deep learning techniques. Feature extraction may be based on at least one of the following: size-invariant feature transform (SIFT), gray-level co-occurrence matrix (GLCM), Gaboo features, Tubeness, etc. The classifier may be based on at least one of the following: random forest, support vector machine, neural network, Bayesian network, etc. Deep learning may be based on at least one of the following: autoencoder, generative adversarial network, weakly supervised learning, bootstrap method, etc.

[0043] Thus, a method for constructing the first / second imaging data of the “missing pixels of other types” can be found.

[0044] In some embodiments, the algorithm can be hard-coded on the analysis part, that is, the machine learning algorithm can provide the image processing algorithm, function, etc., and then provide the image processing algorithm, function, etc. at the chip (such as GPU, FPGA, CPU, etc.), which can save processing power instead of storing the artificial intelligence on the time-of-flight imaging part.

[0045] However, in other embodiments, a machine learning algorithm may be developed and / or used by a (strong or weak) artificial intelligence (such as a neural network, support vector machine, Bayesian network, genetic algorithm, etc.) that constructs the first imaging data, which in some embodiments enables the algorithm to be applicable to situations, scenarios, etc.

[0046] In some embodiments, the analysis unit is further configured to construct second imaging data of at least one second type of imaging element based on first imaging element data of at least one first type of imaging element, wherein the first imaging element data corresponds to imaging data of a first modulation phase and the second imaging element data corresponds to imaging data of a second modulation phase, as described above. The modulation phase may refer to indirect ToF as discussed above and as generally known to those skilled in the art.

[0047] The second imaging data can be constructed similarly to the first imaging data, wherein at least one imaging element of the first type is driven while at least one imaging element of the second type is turned off, as described herein (referring to the case where the first and second imaging elements have different modulation phases). However, in some embodiments, the elements are driven simultaneously, and the phase shift (i.e., different modulation phases) can be caused by other measurements (e.g., correspondingly controlling the light sources and shutters of the different imaging elements).

[0048] However, in addition to the algorithm or parameters used to construct the first imaging data, other algorithms or other parameters may be used to construct the second imaging data.

[0049] In some embodiments, the time-of-flight imaging unit includes at least one third type of imaging element, which is included in a predetermined pattern, and the analysis unit is further configured to: construct third imaging data of at least one third type of imaging element based on any one of the first imaging element data and the second imaging element data, wherein the third imaging data indicates color information.

[0050] As described above, at least one third-type imaging element may include color information. Thus, the third type may specifically refer to a type corresponding to a specific color (e.g., red or multiple (at least two) colors) (note that color herein may refer to any wavelength of the electromagnetic spectrum, regardless of its visibility to the human eye, such as infrared light, ultraviolet light, or any other type of electromagnetic radiation). For example, there may be third-type imaging elements, such as red, blue, and green, that acquire imaging data for corresponding colors, as well as fourth-type imaging elements and fifth-type imaging elements. Furthermore, there may be multispectral image sensors, etc.

[0051] In other embodiments, there may be at least one imaging element of a third type and at least one imaging element of a fourth type, both acquiring additional phase information.

[0052] Providing at least one third type of imaging element can improve image quality and / or produce more complex images. For example, providing at least one third type of imaging element and at least one fourth type of imaging element can increase the signal-to-noise ratio. Alternatively, image complexity can be increased by including additional color information.

[0053] In other embodiments, multiple imaging elements that acquire phase information are combined with multiple imaging elements that acquire color information (eg, two phase and three colors, four phase and three colors, etc.).

[0054] In this context, the third imaging data may be a collection of different imaging data, such as third and fourth phase information and first to third color information.

[0055] The construction of the third imaging data can be similar to the construction of the first imaging data and the second imaging data, and thus the third imaging data can be constructed from the first imaging element data and / or the second imaging element data. In addition, in some embodiments, the first imaging data can be constructed based on the second imaging element data and / or the third imaging element data, and the second imaging data can be constructed based on the first imaging element data and / or the third imaging element data. Specifically, color information can be constructed based on phase information, or phase information can be constructed based on color information.

[0056] In some embodiments, a machine learning algorithm is applied to a neural network, as already described.

[0057] In some embodiments, the neural network is trained based on a predetermined pattern. The predetermined pattern can be hard-coded in the machine learning algorithm. In addition, ground truth data (i.e., reference data for the machine learning algorithm) can be provided to the machine learning algorithm, such as a ToF image with desired properties (e.g., low noise, high resolution, etc.), to learn to construct the first imaging data / second imaging data.

[0058] In some embodiments, a function or algorithm obtained by a machine learning algorithm for constructing the first (or second or third) imaging data is provided at the analysis section, as has been described above.

[0059] In some embodiments, the first imaging data and the second imaging data are constructed in response to a single exposure of the time-of-flight imaging unit. In this context, the time-of-flight imaging unit may be modulated for only one imaging cycle and, for example, acquire only the first imaging data, while constructing the second imaging data and / or the third imaging data based on the first imaging data.

[0060] Additionally, in some embodiments, the first and second (and third) imaging data may be acquired within one exposure by having two (three) modulation cycles within one exposure, thereby achieving a minimum number of exposures (eg, one).

[0061] Some embodiments relate to a time-of-flight imaging device, comprising: a time-of-flight imaging unit, comprising at least one first type of imaging element and at least one second type of imaging element, wherein the at least one first type of imaging element and the at least one second type of imaging element are arranged in a predetermined pattern, as described herein; and an analysis unit for the time-of-flight imaging unit, configured to: construct first imaging data of at least one first type of imaging element based on second imaging element data of at least one second type of imaging element, wherein the first imaging data is constructed based on a machine learning algorithm, as described herein.

[0062] In some embodiments, the time-of-flight imaging portion and the analysis portion are stacked on each other. In addition, in some embodiments, if not included in the time-of-flight imaging portion or the analysis portion, the memory can be additionally stacked on the analysis portion, on the time-of-flight imaging portion, or between the analysis portion and the time-of-flight imaging portion.

[0063] By providing a stacked configuration, the size of the produced chip can be reduced compared to a side-by-side configuration, signal paths can be shortened, etc.

[0064] In some embodiments, the predetermined pattern corresponds to an alternating arrangement of at least one first type imaging element and at least one second type imaging element. As described above, the alternating arrangement may be a row, checkerboard, grid, or the like, depending on the situation or scenario.

[0065] In some embodiments, the predetermined pattern is a random pattern, as described herein.

[0066] In some embodiments, the time-of-flight imaging device further includes at least one third-type imaging portion included in a predetermined pattern indicating color information, as described herein.

[0067] Some embodiments relate to a method for controlling an analysis unit for a time-of-flight imaging unit (or a circuit system, device, etc. as described herein), wherein the time-of-flight imaging unit includes at least one first type of imaging element and at least one second type of imaging element, wherein the at least one first type of imaging element and the at least one second type of imaging element are arranged in a predetermined pattern, and the method includes: constructing first imaging data of at least one first type of imaging element based on second imaging element data of at least one second type of imaging element, wherein the first imaging data is constructed based on a machine learning algorithm as described herein.

[0068] In some embodiments, the method further includes: constructing second imaging data of at least one second type of imaging element based on first imaging element data of at least one first type of imaging element, wherein the first imaging element data corresponds to imaging data of a first modulation phase, and wherein the second imaging element data corresponds to imaging data of a second modulation phase, as described herein.

[0069] In some embodiments, the time-of-flight imaging portion includes at least one third type of imaging element, which is included in a predetermined pattern, and the method further includes: constructing third imaging data of at least one third type of imaging element based on any one of the first imaging element data and the second imaging element data, wherein at least the third imaging data indicates color information, or wherein any one of the first imaging data and the second imaging data is further constructed based on the third imaging element data, as described herein.

[0070] In some embodiments, a machine learning algorithm is applied to a neural network, as described herein.

[0071] In some embodiments, a neural network is trained based on predetermined patterns, as described herein.

[0072] In some embodiments, a function obtained by a machine learning algorithm for constructing the first imaging data and the second imaging data is provided at the analysis portion, as described herein.

[0073] In some embodiments, the first imaging data and the second imaging data are constructed in response to a single exposure of the time-of-flight imaging portion, as described herein.

[0074] The methods described herein are also implemented in some embodiments as a computer program that, when executed on a computer and / or processor, causes the computer and / or processor to perform the methods. In some embodiments, a non-transitory computer-readable recording medium having a computer program product stored thereon is also provided that, when executed by a processor (such as the aforementioned processor), causes the methods described herein to be performed.

[0075] return Figure 1 , six embodiments of a ToF imaging portion are shown. In each of these embodiments, and also in other embodiments, the pattern is predetermined, ie, hard-coded into the training algorithm of the artificial intelligence.

[0076] In the embodiment indicated by reference numeral 1, a method for obtaining two phases is shown. and The phase information of the first type of imaging elements and the second type of imaging elements are arranged alternately in rows.

[0077] In the embodiment indicated by reference numeral 2, a method for obtaining two phases is shown. and The phase information of the first type of imaging elements and the second type of imaging elements are arranged in a quincunx grid.

[0078] In other embodiments, alternating arrangement also refers to column arrangement. In addition, alternating arrangement can refer to other types of regular arrangements of imaging elements, such as two first type imaging elements and one second type imaging element (or other combinations), row, column, diagonal, etc.

[0079] In the embodiment indicated by reference numeral 3, the phase information is obtained. and A grid-like arrangement of four types of imaging elements is repeated.

[0080] In the embodiment indicated by reference numeral 4, two types of and An irregular (random) pattern of imaging elements.

[0081] In the embodiment with reference numeral 5 , an irregular (random) pattern of two types of phase information acquisition imaging elements and three color acquisition imaging elements red (R), green (G) and blue (B) is shown.

[0082] In the embodiment indicated by reference numeral 6, two phase acquisition pixels are shown. and and three colors to obtain a regular pattern of pixels R, G, and B.

[0083] Figure 2 FIG. 1 shows an embodiment of a time-of-flight imaging device 10 according to the present disclosure. The time-of-flight imaging device 10 includes a pixel array 11 corresponding to pixels such as Figure 1 The time-of-flight imaging device 10 further includes a parameter memory 12 and a demosaicing pipeline 13 .

[0084] In other embodiments, the pixel array 11 may further include auxiliary imaging elements (pixels), such as infrared pixels for acquiring image information of the infrared spectrum, other color filters, and the like.

[0085] Parameter memory 12 stores or calculates calibration data and learned hyperparameters. The calibration data includes characteristics of pixel array 11, i.e., the distribution and type of pixels, offset data, gain data, and auxiliary information (e.g., confidence levels). The memory also stores parameters for the operation of demosaicing pipeline 13, such as parameters for signal processing to obtain a depth map. The parameters included in parameter memory 12 include machine learning parameters learned from a given data set (such as calibration data).

[0086] The machine learning parameters are obtained by training. Therefore, in a calibration mode that is different from the operating mode described herein and corresponds to a known operating mode, ground truth data is captured using the time-of-flight imaging device 10 under predetermined conditions (such as conditions that minimize noise, a specific temperature, etc.). The ground truth data corresponds to a full-resolution raw ToF image. The time-of-flight imaging device 10 is then operated as explained herein to acquire first and / or second and / or third imaging data (based on a predetermined pattern of pixel arrays that is hard-coded into the training), and a machine learning algorithm is applied to map the first and / or second and / or third imaging data to the ground truth data in order to find machine learning parameters to construct a final image from the first, second and / or third imaging data, as explained herein.

[0087] The found machine learning parameters are then written to the parameter memory 12 and recalled when applying the machine learning algorithm to correct the parameters (depending on the situation).

[0088] In this embodiment, the demosaicing pipeline 13 corresponds to the analysis unit, as described herein. However, in other embodiments, the analysis unit includes the parameter memory 12 and the demosaicing pipeline 13. The demosaicing pipeline 13 is based on a machine learning engine capable of performing signal processing tasks. The demosaicing pipeline receives raw ToF data from the pixel array 11 and auxiliary data from the parameter memory 12.

[0089] The demosaicing pipeline constructs imaging data from the pixel array based on the found algorithm, as explained herein.

[0090] In reference Figure 3 The demosaicing pipeline is further explained in the context of

[0091] Figure 3 A method 20 for constructing imaging data is shown.

[0092] In response to the acquisition of the pixel array 11, the mosaicked raw data (including the first / second / third imaging data) is sent from the pixel array 11 to the demosaicing pipeline 13 in 21. In addition, calibration parameters are sent from the parameter memory 12 to the demosaicing pipeline in 22. The calibration parameters include, for example, pixel value offsets and gains.

[0093] In 23 , mosaic layout information (including the predetermined pattern and the type of imaging elements) is sent from the pixel array 11 to the demosaicing pipeline 13 .

[0094] At 24 , a calibration function comprising calibration parameters and mosaic layout information is applied to the mosaicked raw data, thereby generating mosaicked calibrated data which is output at 25 .

[0095] The calibration functions further include functions to remove noise, correct for non-idealities in the phase response, such as gain removal.

[0096] At 26 , a pre-processing function including the mosaic layout information is applied to the mosaicked calibration data, thereby generating pre-processed data which is output at 27 .

[0097] The preprocessing functions further include normalization functions, upscaling functions, and calculation functions for determining intermediate data, which are useful for learning-based functions such as nearest neighbor interpolation, stacking, and normalization of inputs.

[0098] The learning-based function is applied to the preprocessed data at 28. The learning-based function includes mosaic layout information and trained model parameters, which are sent from the parameter memory 12 to the demosaicing pipeline 13 at 29. By applying the learning-based function and a forward-pass algorithm to the preprocessed data, demosaiced data output at 30 is generated.

[0099] Figure 4 Shown in Figure 3 21 is a representation of the mosaicked original data 40 sent in the embodiment of the present invention, wherein the mosaicked original data 40 is Figure 1 The time-of-flight imaging portion of the embodiment of reference numeral 3 is acquired.

[0100] Therefore, the mosaicked raw data correspond to imaging signals because they are acquired with the corresponding imaging element of Example 3. Different vignettes of the imaging element represent different depth information.

[0101] Figure 5 A first example of first imaging data 51 acquired by the time-of-flight imaging unit of Example 2 is shown, wherein only the imaging element Acquire the signal, and construct the second imaging data based on the first imaging data, as shown in 52, which corresponds to Figure 3 De-mosaiced data 30.

[0102] also, Figure 5 A representation of second imaging data 53 acquired using the time-of-flight imaging portion of Example 2 is shown, wherein only the imaging element Acquire the signal, and construct the first imaging data based on the second imaging data, as shown in 54, which corresponds to Figure 3 De-mosaiced data 30.

[0103] exist Figure 5 In, and Figure 4 Similarly, different haloes correspond to different depth information. It should be noted that, despite displaying the same scene, the haloes for images 51 and 52 differ from those for images 53 and 54. This is because the corresponding depth information is relative to a predetermined reference value, which is different in the two cases. However, combining the phase information of the two images 52 and 54 allows normalization to the reference value and produces a uniform output.

[0104] Figure 6 A second example of the first imaging data 61 acquired by the time-of-flight imaging unit of Example 1 is shown, wherein only the imaging element A signal is acquired, and second imaging data is constructed based on the first imaging data, as shown at 62 .

[0105] also, Figure 6 A representation of second imaging data 63 acquired using the time-of-flight imaging portion of Example 1 is shown, wherein only the imaging element A signal is acquired, and first imaging data is constructed based on the second imaging data, as shown at 64 .

[0106] therefore, Figure 6 Mainly corresponds to Figure 5 , but with another time-of-flight imaging section.

[0107] Figure 7 is a perspective view depicting a first example of the external configuration of a stacked image sensor 70 to which the present technology is applied.

[0108] For example, the image sensor may be a complementary metal oxide semiconductor (CMOS) image sensor. This is a three-layer structured image sensor, consisting of (semiconductor) substrates 71, 72, and 73 stacked in order from top to bottom.

[0109] A pixel array section 74 is formed on the substrate 71. The pixel array section 74 is configured to perform photoelectric conversion and has a plurality of pixels (not shown) arranged in a matrix pattern to respectively output pixel signals, as described herein.

[0110] A peripheral circuit 75 is formed on the substrate 72. The peripheral circuit 75 performs various signal processing such as AD conversion of the pixel signal output from the pixel array section 74. In addition, the peripheral circuit 75 includes a demosaic pipeline, as described herein.

[0111] A memory 76 is formed on the substrate 73. The memory 76 functions as a storage section that temporarily stores pixel data resulting from AD conversion of pixel signals output from the pixel array section 74. Furthermore, the memory 76 includes a parameter memory, as described herein.

[0112] Figure 8 A second configuration example of a stacked image sensor 80 is depicted.

[0113] exist Figure 8 In the components in Figure 7 Components that find their corresponding counterparts in FIG. 1 are denoted by the same reference numerals, and their explanations are appropriately omitted hereinafter.

[0114] and Figure 7 Like its counterpart 70 , the image sensor 80 has a substrate 71 . However, it should be noted that the image sensor 80 differs from the image sensor 70 in that a substrate 81 is provided in place of the substrates 72 and 73 .

[0115] The image sensor 80 has a two-layer structure, that is, the image sensor has substrates 71 and 81 stacked in sequence from top to bottom.

[0116] The peripheral circuit 75 and the memory 76 are formed on the substrate 81 .

[0117] Figure 9 It is a depiction Figure 7 and Figure 8 75 is a block diagram of a configuration example of the peripheral circuit 75.

[0118] The peripheral circuit 75 includes a plurality of analog-to-digital (AD) converters (ADC) 91 , an input / output data control section 92 , a data path 93 , a signal processing section 94 , and an output interface (I / F) 95 .

[0119] There are the same number of ADCs 91 as there are columns of pixels constituting the pixel array section 74. Pixel signals output from the pixels arranged in each row (line) are subjected to parallel column AD conversion, which involves parallel AD conversion of the pixel signals. The input / output data control section 92 is supplied with pixel data of digital signals obtained for each row by the ADCs 91 performing parallel column AD conversion on the pixel signals, which are analog signals.

[0120] The input / output data control section 92 controls writing of pixel data from the ADC 91 to the memory 76 and reading of pixel data from the memory 76. The input / output data control section 92 also controls output of pixel data to the data path 93.

[0121] The input / output data control section 92 includes a register 96 , a data processing section 97 , and a memory I / F 98 .

[0122] Under instructions from an external device, information used by the input / output data control section 92 to control its processing is set (recorded) to the register 96. Based on the information set in the register 96, the input / output data control section 92 performs various processing.

[0123] The data processing unit 97 directly outputs the pixel data from the ADC 91 to the data path 93 .

[0124] Alternatively, the data processing section 97 may perform necessary processing on the pixel data supplied from the ADC 91 before writing the processed pixel data into the memory 76 via the memory I / F 98 .

[0125] Furthermore, the data processing section 97 reads the pixel data written in the memory 76 via the memory I / F 98 , processes the pixel data retrieved from the memory 76 as necessary, and outputs the processed pixel data to the data path 93 .

[0126] Whether the data processing section 97 outputs the pixel data from the ADC 91 directly to the data path 93 or writes the pixel data to the memory 76 can be selected by setting appropriate information to the register 96 .

[0127] Likewise, whether or not the data processing section 97 processes the pixel data fed from the ADC 91 can be determined by setting appropriate information to the register 96 .

[0128] The memory I / F 98 functions as an I / F that controls writing and reading of pixel data to and from the memory 76 .

[0129] The data path 93 is composed of signal lines serving as a path for feeding pixel data output from the input / output data control section 92 to the signal processing section 94 .

[0130] The signal processing section 94 performs signal processing such as black level adjustment, demosaicing, white balance adjustment, noise reduction, development, or other signal processing on the pixel data fed from the data path 93 as necessary before outputting the processed pixel data to the output I / F 95 , as described herein.

[0131] The output I / F 95 functions as an I / F that outputs the pixel data fed from the signal processing section 94 to the outside of the image sensor.

[0132] refer to Figure 10, shows an embodiment of a time-of-flight (ToF) device 100 that can be used for depth sensing or providing distance measurements, particularly for the techniques discussed herein, wherein the ToF device 100 is configured as an iToF camera. The ToF device 100 has a circuit system 107 that is configured to perform the methods discussed herein and form the control of the ToF device 100 (and it includes corresponding processors, memory and storage devices generally known to those skilled in the art, not shown, and analysis units as discussed herein).

[0133] The ToF device 100 has a continuous light source 101 , and the light source includes a light-emitting element (based on a laser diode), wherein in this embodiment, the light-emitting element is a narrow-band laser element.

[0134] A light source 101 emits light, ie, modulated light, toward a scene 102 (an area of interest or target), which reflects light. The reflected light is focused by an optical stack 103 onto a light detector 104.

[0135] As discussed herein, the light detector 104 has a time-of-flight imaging portion implemented based on a plurality of CAPDs formed in a pixel array and a microlens array 106 that focuses light reflected from the scene 102 onto the time-of-flight imaging portion 105 (onto each pixel of the time-of-flight imaging portion 105).

[0136] When light reflected from scene 102 is detected, the light emission time and modulation information is fed to circuitry 107, which includes a time-of-flight measurement unit 108, which also receives corresponding information from time-of-flight imaging section 105. Based on the modulated light received from light source 101 and the demodulation performed (and demosaicing discussed herein), time-of-flight measurement unit 108 calculates the phase shift of the received modulated light that has been emitted from light source 101 and reflected by scene 102, and based on this calculates the distance d (depth information) between time-of-flight imaging section 105 and scene 102, as also described above.

[0137] The depth information is fed from the time-of-flight measurement unit 108 to a 3D image reconstruction unit 109 of the circuit system 107 , which reconstructs (generates) a 3D image of the scene 102 based on the depth information received from the time-of-flight measurement unit 108 .

[0138] Figure 11 A method according to the present disclosure is shown, for example, for controlling a time-of-flight device or imaging portion as discussed herein (e.g., Figure 10 ToF device 100, Figure 2 Flowchart of an embodiment of a method 120 of a ToF device 10, etc.

[0139] In 121 , second imaging element data is acquired within one exposure of the time-of-flight imaging section.

[0140] At 122, first imaging data is constructed based on the second imaging element data using a machine learning algorithm. To construct the first imaging data, the machine learning algorithm is applied to a neural network trained based on a predetermined pattern of the time-of-flight imaging unit. The training results are provided to an analysis unit of the time-of-flight imaging device to construct the first imaging data.

[0141] In 123 , first imaging element data is acquired within the same exposure of the time-of-flight imaging portion but in another modulation phase than the second imaging element data.

[0142] At 124 , second imaging data is constructed based on the first imaging element data using an adapted machine learning algorithm similar to the machine learning algorithm of 122 .

[0143] In 125, third imaging element data is acquired within the same exposure of the time-of-flight imaging portion but in another modulation phase than the first imaging element data and the second imaging element data. The third imaging element data corresponds to color information.

[0144] It should be appreciated that the embodiments describe the method in an exemplary order of method steps. However, the specific order of method steps is given for illustrative purposes only and should not be interpreted as binding. For example, Figure 3 The order of 24 and 26 in the embodiment can be exchanged. In addition, Figure 3 In the embodiment of 26, 28 and 21, the order can be exchanged. In addition, Figure 9 The order of 93 and 94 in the embodiment of the present invention can also be exchanged. Other changes in the order of the method steps will be obvious to the skilled person.

[0145] Please note that the division of the apparatus 10 into units 12 to 13 is for illustrative purposes only, and the present disclosure is not limited to any particular functional division in a particular unit. For example, the apparatus 10 may be implemented by a correspondingly programmed processor, a field programmable gate array (FPGA), etc.

[0146] If not stated otherwise, all units and entities described in this specification and claimed in the appended claims may be implemented as, for example, integrated circuit logic on a chip, and if not stated otherwise, the functions provided by such units and entities may be implemented by software.

[0147] To the extent that the embodiments of the present disclosure described above are at least partially implemented using software-controlled data processing apparatus, it should be understood that providing such software-controlled computer programs and transmission, storage or other media providing such computer programs are contemplated as aspects of the present disclosure.

[0148] Note that the present technology can also be configured as described below.

[0149] (1) An analysis section for a time-of-flight imaging section, wherein the time-of-flight imaging section includes at least one imaging element of a first type and at least one imaging element of a second type, wherein the at least one imaging element of the first type and the at least one imaging element of the second type are arranged in a predetermined pattern, wherein the analysis section is configured to:

[0150] First imaging data of at least one imaging element of the first type is constructed based on second imaging element data of at least one imaging element of the second type, wherein the first imaging data is constructed based on a machine learning algorithm.

[0151] (2) The analysis unit according to (1), further configured to:

[0152] Second imaging data of at least one second type of imaging element is constructed based on first imaging element data of at least one first type of imaging element, wherein the first imaging element data is based on a first modulation phase and the second imaging element data is based on a second modulation phase.

[0153] (3) The analyzing section according to any one of (1) or (2), wherein the time-of-flight imaging section includes at least one imaging element of the third type, the at least one imaging element of the third type being included in a predetermined pattern, and wherein the analyzing section is further configured to:

[0154] Third imaging data of at least one third type of imaging element is constructed based on any one of the first imaging element data and the second imaging element data, wherein the third imaging data indicates color information.

[0155] (4) The analyzing section according to any one of (1) to (3), wherein any one of the first imaging data and the second imaging data is further constructed based on third imaging element data.

[0156] (5) The analysis unit according to any one of (1) to (4), wherein the machine learning algorithm is applied to a neural network.

[0157] (6) The analyzing section according to any one of (1) to (5), wherein the neural network is trained based on a predetermined pattern.

[0158] (7) The analysis section according to any one of (1) to (6), wherein a function obtained by a machine learning algorithm is provided at the analysis section.

[0159] (8) The analyzing section according to any one of (1) to (7), wherein the first imaging data is constructed in response to one exposure of the time-of-flight imaging section.

[0160] (9) A time-of-flight imaging device comprising:

[0161] a time-of-flight imaging section including at least one first-type imaging element and at least one second-type imaging element, wherein the at least one first-type imaging element and the at least one second-type imaging element are arranged in a predetermined pattern; and

[0162] The analysis section for the time-of-flight imaging section is configured to:

[0163] First imaging data of at least one imaging element of the first type is constructed based on second imaging element data of at least one imaging element of the second type, wherein the first imaging data is constructed based on a machine learning algorithm.

[0164] (10) The time-of-flight imaging device according to (9), wherein the time-of-flight imaging section and the analyzing section are stacked on each other.

[0165] (11) The time-of-flight imaging device according to any one of (9) or (10), wherein the predetermined pattern corresponds to an alternating arrangement of at least one first-type imaging element and at least one second-type imaging element.

[0166] (12) The time-of-flight imaging device according to any one of (9) to (11), wherein the predetermined pattern is a random pattern.

[0167] (13) The time-of-flight imaging device according to any one of (9) to (12), further including at least one third-type imaging element included in a predetermined pattern indicating color information.

[0168] (14) A method for controlling an analysis section for a time-of-flight imaging section, wherein the time-of-flight imaging section includes at least one imaging element of a first type and at least one imaging element of a second type, wherein the at least one imaging element of the first type and the at least one imaging element of the second type are arranged in a predetermined pattern, the method comprising:

[0169] First imaging data of at least one imaging element of the first type is constructed based on second imaging element data of at least one imaging element of the second type, wherein the first imaging data is constructed based on a machine learning algorithm.

[0170] (15) The method according to (14), further comprising:

[0171] Second imaging data of at least one imaging element of a second type are constructed based on first imaging element data of at least one imaging element of a first type, wherein

[0172] The first imaging element data corresponds to imaging data of a first modulation phase, and wherein,

[0173] The second imaging element data corresponds to imaging data of a second modulation phase.

[0174] (16) The method according to any one of (14) or (15), wherein the time-of-flight imaging section includes at least one imaging element of a third type, the at least one imaging element of the third type being included in a predetermined pattern, the method further comprising:

[0175] The third imaging data of at least one third type of imaging element is constructed based on either one of the first imaging element data and the second imaging element data, wherein at least

[0176] The third imaging data indicates color information, or wherein,

[0177] Any one of the first imaging data and the second imaging data is further constructed based on the third imaging element data.

[0178] (17) The method according to any one of (14) to (16), wherein the machine learning algorithm is applied to a neural network.

[0179] (18) The method according to any one of (14) to (17), wherein the neural network is trained based on a predetermined pattern.

[0180] (19) The method according to any one of (14) to (18), wherein a function for constructing the first imaging data and the second imaging data obtained by a machine learning algorithm is provided at the analysis section.

[0181] (20) The method according to any one of (14) to (19), wherein the first imaging data is constructed in response to one exposure of the time-of-flight imaging section.

[0182] (21) A computer program comprising a program code which, when executed on a computer, causes the computer to perform the method according to any one of (14) to (20).

[0183] (22) A non-transitory computer-readable recording medium storing a computer program product, which, when executed by a processor, causes the method according to any one of (14) to (20) to be performed.

[0184] (23) A circuit system for a time-of-flight imaging circuit system, wherein the time-of-flight imaging circuit system includes at least one imaging element of a first type and at least one imaging element of a second type, wherein the at least one imaging element of the first type and the at least one imaging element of the second type are arranged in a predetermined pattern, wherein the circuit system is configured to:

[0185] First imaging data of at least one imaging element of the first type is constructed based on second imaging element data of at least one imaging element of the second type, wherein the first imaging data is constructed based on a machine learning algorithm.

[0186] (24) The circuit system according to (23), further configured to:

[0187] Second imaging data of at least one second type of imaging element is constructed based on first imaging element data of at least one first type of imaging element, wherein the first imaging element data is based on a first modulation phase and the second imaging element data is based on a second modulation phase.

[0188] (25) The circuit system of any of (23) or (24), wherein the time-of-flight imaging circuit system includes at least one imaging element of the third type, the at least one imaging element of the third type being included in a predetermined pattern, and wherein the circuit system is further configured to:

[0189] Third imaging data of at least one third type of imaging element is constructed based on any one of the first imaging element data and the second imaging element data, wherein the third imaging data indicates color information.

[0190] (26) The circuit system according to any one of (23) to (25), wherein any one of the first imaging data and the second imaging data is further constructed based on third imaging element data.

[0191] (27) A circuit system according to any one of (23) to (26), wherein the machine learning algorithm is applied to the neural network.

[0192] (28) The circuit system of any one of (23) to (27), wherein the neural network is trained based on a predetermined pattern.

[0193] (29) A circuit system according to any one of (23) to (28), wherein a function obtained by a machine learning algorithm is provided at the circuit system.

[0194] (30) The circuit system of any one of (23) to (29), wherein the first imaging data is constructed in response to a single exposure of the time-of-flight imaging circuit system.

[0195] (31) A time-of-flight imaging device comprising:

[0196] a time-of-flight imaging circuit system comprising at least one imaging element of a first type and at least one imaging element of a second type, wherein the at least one imaging element of the first type and the at least one imaging element of the second type are arranged in a predetermined pattern; and

[0197] A circuit system for a time-of-flight imaging circuit system, in particular a circuit system according to any one of (23) to (29), configured to:

[0198] First imaging data of at least one imaging element of the first type is constructed based on second imaging element data of at least one imaging element of the second type, wherein the first imaging data is constructed based on a machine learning algorithm.

[0199] (32) The time-of-flight imaging device according to (31), wherein the time-of-flight imaging circuit system and the circuit system are stacked on each other.

[0200] (33) The time-of-flight imaging device according to any one of (31) or (32), wherein the predetermined pattern corresponds to an alternating arrangement of at least one first type imaging element and at least one second type imaging element.

[0201] (34) The time-of-flight imaging device according to any one of (31) to (33), wherein the predetermined pattern is a random pattern.

[0202] (35) The time-of-flight imaging device according to any one of (9) to (12), further including at least one third-type imaging element included in a predetermined pattern indicating color information.

[0203] (36) A method for controlling analysis circuitry for a time-of-flight imaging circuitry, wherein the time-of-flight imaging circuitry includes at least one imaging element of a first type and at least one imaging element of a second type, wherein the at least one imaging element of the first type and the at least one imaging element of the second type are arranged in a predetermined pattern, the method comprising:

[0204] First imaging data of at least one imaging element of the first type is constructed based on second imaging element data of at least one imaging element of the second type, wherein the first imaging data is constructed based on a machine learning algorithm.

[0205] (37) The method according to (36), further comprising:

[0206] Second imaging data of at least one imaging element of a second type are constructed based on first imaging element data of at least one imaging element of a first type, wherein

[0207] The first imaging element data corresponds to imaging data of a first modulation phase, and wherein,

[0208] The second imaging element data corresponds to imaging data of a second modulation phase.

[0209] (38) The method of any one of (36) or (37), wherein the time-of-flight imaging circuit system includes at least one imaging element of a third type, the at least one imaging element of the third type being included in a predetermined pattern, the method further comprising:

[0210] The third imaging data of at least one third type of imaging element is constructed based on either one of the first imaging element data and the second imaging element data, wherein at least

[0211] The third imaging data indicates color information, or wherein,

[0212] Any one of the first imaging data and the second imaging data is further constructed based on the third imaging element data.

[0213] (39) A method according to any one of (36) to (38), wherein the machine learning algorithm is applied to a neural network.

[0214] (40) The method of any one of (36) to (39), wherein the neural network is trained based on a predetermined pattern.

[0215] (41) A method according to any one of (36) to (40), wherein a function obtained by a machine learning algorithm for constructing the first imaging data and the second imaging data is provided at the analysis circuit system.

[0216] (42) The method of any one of (36) to (41), wherein the first imaging data is constructed in response to a single exposure by the time-of-flight imaging circuitry.

[0217] (43) A computer program comprising a program code which, when executed on a computer, causes the computer to perform the method according to any one of (36) to (42).

[0218] (44) A non-transitory computer-readable recording medium storing a computer program product, which, when executed by a processor, causes the method according to any one of (36) to (42) to be performed.

Claims

1. An analysis unit for a time-of-flight imaging unit, wherein: The time-of-flight imaging section includes at least one imaging element of a first type and at least one imaging element of a second type, wherein the at least one imaging element of the first type and the at least one imaging element of the second type are arranged in a predetermined pattern, wherein the analyzing section is configured to: obtaining, during a first measurement period, second imaging element data indicating second phase information or information from a spectrum from the at least one second type of imaging element when the at least one first type of imaging element is off; and Based on implementing a machine learning algorithm, first imaging data indicating first phase information of the at least one first type of imaging element for the first measurement period is constructed using the second imaging element data from the at least one second type of imaging element obtained during the first measurement period.

2. The analysis unit according to claim 1, further configured to: Second imaging data of the at least one imaging element of the second type is constructed based on the first imaging element data of the at least one imaging element of the first type.

3. The analysis unit according to claim 2, wherein: The time-of-flight imaging section includes at least one imaging element of a third type, the at least one imaging element of the third type being included in the predetermined pattern, and wherein the analyzing section is further configured to: Third imaging data of the at least one third-type imaging element is constructed based on either one of the first imaging element data and the second imaging element data, wherein the third imaging data indicates color information.

4. The analysis unit according to claim 3, wherein Either of the first imaging data and the second imaging data is further constructed based on third imaging element data.

5. The analysis unit according to claim 1, wherein The machine learning algorithm is applied to a neural network. The analysis unit according to claim 5 , wherein: The neural network is trained based on the predetermined pattern.

7. The analysis unit according to claim 1, wherein The function obtained by the machine learning algorithm is provided at the analysis section.

8. The analysis unit according to claim 1, wherein The first imaging data is constructed in response to one exposure of the time-of-flight imaging section.

9. A time-of-flight imaging device, comprising: a time-of-flight imaging section comprising at least one imaging element of a first type and at least one imaging element of a second type, wherein the at least one imaging element of the first type and the at least one imaging element of the second type are arranged in a predetermined pattern, and when the at least one imaging element of the first type is turned off, the at least one imaging element of the second type provides second imaging element data obtained during a first measurement period; and The analysis department is configured to: A machine learning algorithm is implemented to construct first imaging data indicating first phase information for the at least one first type of imaging element for the first measurement period using the second imaging element data from the at least one second type of imaging element obtained during the first measurement period.

10. The time-of-flight imaging device according to claim 9, wherein: The time-of-flight imaging section and the analyzing section are stacked on each other.

11. The time-of-flight imaging device according to claim 9, wherein: The predetermined pattern corresponds to an alternating arrangement of the at least one imaging element of the first type and the at least one imaging element of the second type.

12. The time-of-flight imaging device according to claim 9, wherein: The predetermined pattern is a random pattern. 13 . The time-of-flight imaging apparatus according to claim 9 , further comprising at least one third-type imaging element included in the predetermined pattern indicating color information.

14. A method for controlling an analysis section for a time-of-flight imaging section, wherein: The time-of-flight imaging section includes at least one imaging element of a first type and at least one imaging element of a second type, wherein the at least one imaging element of the first type and the at least one imaging element of the second type are arranged in a predetermined pattern, and the method includes: obtaining, during a first measurement period, second imaging element data indicating second phase information or information from a spectrum from the at least one second type of imaging element when the at least one first type of imaging element is off; and Based on implementing a machine learning algorithm, first imaging data indicating first phase information of the at least one first type of imaging element for the first measurement period is constructed using the second imaging element data from the at least one second type of imaging element obtained during the first measurement period.

15. The method according to claim 14, further comprising: Second imaging data of the at least one second type imaging element is constructed based on the first imaging element data of the at least one first type imaging element.

16. The method according to claim 15, wherein The time-of-flight imaging section includes at least one imaging element of a third type, the at least one imaging element of the third type being included in the predetermined pattern, the method further comprising: The third imaging data of the at least one third type of imaging element is constructed based on any one of the first imaging element data and the second imaging element data, wherein at least The third imaging data indicates color information, or wherein, Either one of the first imaging data and the second imaging data is further constructed based on third imaging element data.

17. The method according to claim 14, wherein: The machine learning algorithm is applied to a neural network.

18. The method according to claim 17, wherein The neural network is trained based on the predetermined pattern.

19. The method according to claim 14, wherein A function obtained by the machine learning algorithm and used to construct the first imaging data and the second imaging data is provided at the analyzing section.

20. The method according to claim 14, wherein The first imaging data is constructed in response to one exposure of the time-of-flight imaging section.

Images