Electronic device, system, method and computer program

Abstract

A time-of-flight sensor, comprising a processing unit (100), the processing unit comprising an encoding unit (120) to generate compressed time-of-flight histogram data from one or more time-of-flight histograms based on an encoding dictionary stored in an encoding dictionary unit (130); and wherein the one or more time-of-flight histograms are related to a current imaging frame obtained from a time-of-flight pixel array. The encoding unit (120) initially receives filtered histogram data (111) from a readout unit (110). The encoding unit (120) receives this filtered histogram data (111), and forwards the data to the encoding dictionary unit (130) which stores the encoding dictionary and which contains the specific codes and rules (131) used to convert the filtered histogram data into encoded histogram data (124). The encoding dictionary unit (130) may also receive updates from an external source, ensuring that the dictionary (131) is always optimized for the most efficient encoding possible. Encoded data (122) represents the histogram data transformed into its Huffman-encoded form, while the encoded data lengths (123) provide information about the size of each piece of encoded data. The encoding unit (120) forwards the encoded data (122) along with the output data lengths (123) to a packaging unit (140). Lower loss compression of histogram data, employing neighbor bin and neighbor histogram filtering combined with data encoding, in ToF image sensors addresses the increased volume and complexity of ToF data due to advancements in pixel resolution and frame rate.

Description

[0001] ELECTRONIC DEVICE, SYSTEM, METHOD AND COMPUTER PROGRAM

[0002] TECHNICAL FIELD

[0003] The present disclosure generally pertains to an electronic device, a system, a method and a computer program.

[0004] TECHNICAL BACKGROUND

[0005] In existing Time-of-Flight (ToF) sensors, the transmission of histogram data presents significant challenges due to the increased volume of data generated by higher numbers of histograms, bins, and the enhanced bit-depth of each bin. The concordant increase in data volume results in increased power consumption and transmission delays, as the sensor must halt data acquisition during the streamout process, leading to inefficiencies in both time and energy.

[0006] The current state of direct Time-of-Flight (TOF) image sensors lacks on-chip compression of histogram data, leading to high power consumption and temporal inefficiencies during full histogram data streaming. On-chip extraction of 3D depth points, while reducing data streaming, results in the loss of critical information such as noise, pulse shape, and pile-up effects. Enabling full availability of raw histogram data to the host could significantly enhance 3D depth map extraction. Accordingly, it is desirable to provide a ToF sensor with improved data transmission capabilities.

[0007] SUMMARY

[0008] According to a first aspect the present disclosure provides a time-of-flight sensor in accordance with independent claim 1. According to a second aspect, the present disclosure provides a time-of-flight imaging host device in accordance with independent claim 24. According to a third aspect, the present disclosure provides a time-of-flight imaging system in accordance with independent claim 26. According to a fourth aspect, the present disclosure provides a time-of-flight image readout method in accordance with independent claim 27. According to a fifth aspect, the present disclosure provides a computer program in accordance with independent claim 28.

[0009] Further aspects are set forth in the dependent claims, the drawings and the following description.

[0010] BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Embodiments are explained by way of example with respect to the accompanying drawings, in which: Fig. 1 is an abstract illustration of a ToF system according to the present disclosure; and

[0012] Fig. 2 is a block diagram illustrating a ToF system ; and

[0013] Fig. 3 is an explanatory diagram of a frame sequence; and

[0014] Fig. 4a and Fig. 4b are explanatory diagrams of median filtering applied to data; and

[0015] Fig. 5a, Fig. 5b and Fig. 5c are illustrations of obtaining ToF histograms from a pixel or a macropixel; and

[0016] Fig. 6a and Fig. 6b are an illustration of Paeth or median filtering applied to ToF histograms; and Fig. 7 is a block diagram of the interaction of an encoding unit and an encoding dictionary unit; and Fig. 8 is a detailed block diagram showing an example of the encoding process in one embodiment; and

[0017] Fig. 9 is a block diagram of the interaction of an encoding unit, an encoding dictionary unit and an offset dictionary unit; and

[0018] Fig. 10 is a detailed block diagram showing an example of the encoding process in a second embodiment; and

[0019] Fig. 11 is a block diagram illustrating a packing logic and a compression ratio monitor; and Fig. 12 is a more detailed explanatory diagram of the function of the packing logic; and

[0020] Fig. 13 is a block diagram showing an example of the function of the compression ratio monitor; and

[0021] Fig. 14 is an explanatory diagram of the function of a skip frame manager; and

[0022] Fig. 15 is an explanatory diagram of the function of a skip frame manager, illustrating smart skipping; and

[0023] Fig. 16 is a more detailed diagram illustrating the function of the encoding dictionary unit; and Fig. 17 is a diagram illustrating packaged data in one embodiment; and

[0024] Fig. 18 is a diagram illustrating packaged data in a second embodiment; and

[0025] Fig. 19 is a flow diagram of an ToF data transmission method according to the present disclosure; and

[0026] Fig. 20 is an illustration of an electronic device that may comprise the ToF system according to the present disclosure. DETAILED DESCRIPTION OF EMBODIMENTS

[0027] Before a detailed description of the embodiments under reference of Fig. 1 is given, general explanations are made.

[0028] The present disclosure provides a device, method and system for on-chip lossless compression of histogram data in ToF image sensors, thereby addressing the increased volume and complexity of ToF data due to, for example, advancements in pixel resolution and frame rate. The disclosed method employs neighbor bin and neighbor histogram filtering combined with data encoding, for example Huffman encoding. The method facilitates higher speed, lower loss compression directly on the imaging chip.

[0029] Despite efforts to reduce a physical area of histograms using advanced technology nodes and stacking beneath the light-sensitive layer, there remains a notable deficiency in the compression of histogram data that must be output from the image sensor to the host. Existing on-chip algorithms designed to extract depth information from histograms and minimize raw data streamout are not sufficiently versatile and may fail under various application environment scenarios influenced by factors such as pile-up, background light, reflectivity, and temporally correlated noise sources.

[0030] As the volume of histogram data increases, so do the power requirements and delays associated with data streamout, leading to inefficiencies. During data streamout, the sensor cannot utilize the histograms for ongoing data acquisition, resulting in interrupted exposure until the histograms are fully transmitted to the host. This interruption causes both temporal and power inefficiencies, as the sensor and Time-of-Flight transmitter remain active without gathering new depth information. Addressing this issue by merely increasing the number of streamout channels to enhance data transmission speed adversely impacts the power consumption, particularly at internet-of things (loT) edge nodes.

[0031] An alternative approach involves compressing the data as early as possible in the data pipeline to reduce the power consumed by all data elaboration units between the compression and decompression blocks. However, known on-chip implementable compression methods are lossy, preventing the recovery of original data, which may lead to resolution trade-offs and incompatibility with Image Signal Processor (ISP) blocks that require the original image data. Lossless compression algorithms, which require comprehensive information to be effective, pose integration challenges near the image sensing node, typically necessitating that the image data be first acquired and transferred to a dedicated ASIC or host before compression.

[0032] A time-of-flight sensor according to the present disclosure comprises a processing unit, the processing unit comprising an encoding unit configured to generate compressed time-of-flight histogram data from one or more time-of-flight histograms based on an encoding dictionary stored in an encoding dictionary unit; and wherein the one or more time-of-flight histograms are related to a current imaging frame obtained from a time-of-flight pixel array.

[0033] A time-of-flight sensor is a device designed to measure the distance to an object by calculating the time it takes for a light signal to travel from the sensor to the object and back. This sensor comprises a processing unit, which may be any component or set of components that manage data processing tasks within the sensor. Integral to the processing unit is an encoding unit, which is tasked with the function of compressing data. The encoding unit is configured to compress time-of-flight histogram data. A time of flight histogram is a data set representing the distribution of measured flight times of photons or light particles. Time-of-Flight histograms are an output of direct Time-of-Flight (dToF) sensors.

[0034] The compression process carried out by the encoding unit uses an encoding dictionary, which is a predefined set of rules or codes that dictate how the histogram data is to be compressed. This dictionary is stored within an encoding dictionary unit, a dedicated storage component within the sensor that maintains the encoding dictionary for ready access during data processing. The encoding unit may comprise a field-programmable-gate-array (FPGA).

[0035] The term "one or more time-of-flight histograms" refers to at least one histogram, where each histogram encapsulates data regarding a timing distribution of time-of-flight measurements captured during a specific measurement cycle. Each measurement cycle may be called a frame. The histograms are linked to, or associated with, a current imaging frame, which may be seen as a snapshot of time-of-flight data captured within a given exposure time. The frame data is obtained from a time-of-flight pixel array, which may be an arrangement of multiple sensor pixels, each capable of performing independent time-of-flight measurements.

[0036] There are embodiments wherein the processing unit and the time-of-flight pixel array are both provided on-chip.

[0037] The sensor may include the processing unit and the time-of-flight pixel array, such that the data compression is performed on-chip. The time-of-flight pixel array may consist of multiple pixels, each capable of independently performing ToF measurements.

[0038] There are embodiments wherein the sensor further comprises a readout unit configured to read out the time-of-flight histograms from the time-of-flight pixel array.

[0039] The sensor may include a readout unit. The readout unit is configured to retrieve or read out the time-of-flight histograms from the time-of-flight pixel array. The readout unit functions to extract these histograms from the pixel array, facilitating further processing such as encoding and compression.

[0040] There are embodiments wherein the sensor further comprises a readout unit configured to generate filtered time-of-flight histogram data by applying a filter to the time-of-flight histograms.

[0041] The sensor may include a readout unit. This readout unit is configured to read out time-of-flight histograms from a time-of-flight pixel array. The readout unit is further configured to generate filtered time-of-flight histogram data. The filtering comprises application of a filter to the time-of-flight histograms.

[0042] The filter applied by the readout unit filters the histograms to reduce the number of bit values required to represent the data.

[0043] There are embodiments wherein the sensor comprises a filter that is at least one of a Paeth filter or a median filter.

[0044] A Paeth filter is a predictive filtering method that selects the value (from adjacent data points) that is closest to the predicted value, minimizing the overall encoding size by reducing redundancy. A Paeth filter is used, for example, in image processing for reducing the error introduced during lossless compression.

[0045] A median filter operates by sorting the values in a neighborhood and choosing the median value, which is effective in removing noise from the data while preserving edges in the spatial domain. There are embodiments wherein the sensor comprises a plurality of time-of-flight pixels or time-of-flight macropixels that is a group of four time-of-flight pixels or time-of-flight macropixels, respectively, that are adjacent to each other.

[0046] The plurality of time-of-flight pixels or time-of-flight macropixels may consist of groups of four units that are adjacent to each other. Each macropixel is a plurality of pixels that are grouped together to produce a single ToF histogram. The maxropixel may comprise the plurality of pixels in an array configuration or in a configuration that is not an array.

[0047] In this setup, each time-of-flight pixel is an individual sensor element capable of independently generating ToF measurements. Similarly, a time-of-flight macropixel is composed of multiple time-of-flight pixels grouped together to collectively process the light data, producing a single histogram. The sensor comprises a Paeth or median filter configured to process multiple time-of-flight histograms, where each histogram is associated with either a distinct time-of-flight pixel or a distinct time-of-flight macropixel. There are embodiments wherein the sensor uses a histogram value of one bin of the plurality of the time-of-flight histograms as a reference value to filter the plurality of time-of-flight histograms. In the filtering, a histogram value from one bin of the plurality of time-of-flight histograms may be used as a reference value to filter the entire set of time-of-flight histograms. The reference bin may, for example, be a value of an n-th bin of a first histogram of a group of, for example, four histograms. The reference value may then be used to filter the n-th bins of the other three histograms.

[0048] As another example, a histogram bin 'n' can be filtered using one of the neighboring macropixel histograms by referencing the corresponding time-correlated bin 'n' of the neighboring pixel histograms or by using the previous bin 'n-1' of the same histogram.

[0049] The median and Paeth filters can be configured on-chip to determine which neighboring bin should be chosen in each case to optimize filtering efficiency and produce a value closest to zero when subtracted from the histogram bin.

[0050] There are embodiments wherein the sensor comprises a filter configured to filter bins of the same one of the time-of-flight histograms.

[0051] The sensor comprises a filter configured to process bins within a single time-of-flight histogram. Thus, a single bin, for example a first bin of a ToF histogram may be used to filter all remaining bins of the ToF histogram.

[0052] The filtering of bins of multiple histograms using a value of a bin of a single histogram as a reference value, as described hereinabove, may be performed in addition to the filtering of a single histogram. This results in all bins of all histograms in a group of histograms being filtered using a value of a single bin of one of the histograms as a reference value.

[0053] It should be noted that the bin having the reference value may not be filtered.

[0054] There are embodiments wherein the sensor comprises an encoding unit configured to generate the compressed time-of-flight histogram data by encoding the filtered time-of-flight histogram data based on the encoding dictionary.

[0055] The sensor comprises an encoding unit that is configured to generate compressed time-of-flight histogram data. The generating the compressed ToF histogram data comprises encoding the filtered time-of-flight histogram data.

[0056] The encoding is performed based on an encoding dictionary, which is a predefined set of rules or mappings that dictate how the data should be transformed. The mapping may be a predefined mapping of input values to output values. Both the input values and output values may be provided as binary numbers (which may also be called binary code) or bit-strings. According to the present disclosure, the mapping is performed such that, for example, rare input values, corresponding to bin values of the filtered ToF histograms, are mapped to longer binary numbers while frequent input values are mapped to shorter binary numbers. This reduces the number of bits required to represent the ToF histograms.

[0057] There are embodiments wherein the sensor comprises an encoding dictionary that is a Huffman encoding dictionary.

[0058] The Huffman encoding dictionary is a method used in data compression that involves creating a set of binary codes for different data elements, where the length of each code is inversely related to the frequency of the data element it represents. In the context of the sensor, the Huffman encoding dictionary is used to encode the time-of-flight histogram data.

[0059] There are embodiments wherein the sensor comprises an encoding dictionary unit further configured to update the encoding dictionary based on an updated encoding dictionary received from a host device.

[0060] The update sensor may receive an updated encoding dictionary from a host device. The host device may be a computer or a server that manages or processes the data collected by the sensor. By incorporating new rules or codes from the updated dictionary received from the host device, the encoding dictionary unit can adapt to changes in data patterns or requirements, ensuring that the compression remains optimal.

[0061] There are embodiments wherein the sensor comprises an encoding unit further configured to generate the compressed time-of-flight histogram data based on an offset dictionary stored in an offset dictionary unit.

[0062] The offset dictionary is stored within the offset dictionary unit within the sensor. The offset dictionary contains specific offsets or adjustments that are applied to the time-of-flight histogram data during the encoding process. These offsets are used to map the data values in the histograms before they are compressed, optimizing the data for better compression ratios and more efficient storage or transmission. By using an offset dictionary, the bit-length needed to encode the histograms can further be reduced.

[0063] There are embodiments wherein the sensor comprises an offset dictionary unit further configured to update the offset dictionary based on an updated offset dictionary received from a host device. The update to the offset dictionary may be facilitated through the receipt of an updated offset dictionary from the host device. The capability of updating the offset dictionary ensures that the compression adjustments remain optimal to the current data characteristics and requirements.

[0064] There are embodiments wherein the sensor further comprises a packaging module configured to generate packaged time-of-flight histogram data by concatenating significant digits of the compressed time-of-flight histogram data.

[0065] The packaging module is a module configured to handle the final preparation of time-of-flight histogram data for storage or transmission. This module is designed to generate packaged time-of-flight histogram data by concatenating significant digits of the compressed time-of-flight histogram data.

[0066] The process of concatenation performed by the packaging module involves aligning these significant digits into a continuous stream or structured format, which optimizes the data for subsequent processes such as transmission to a host device or further analytical processing.

[0067] There are embodiments wherein the sensor further comprises an output module configured to generate, from the packaged time-of-flight histogram data, a histogram output.

[0068] The output module is a module configured to manage the final stage of data handling by generating a histogram output from the packaged time-of-flight histogram data. This output module takes the data that has been previously processed and packaged by the packaging module — where significant digits of the compressed histogram data have been concatenated into a structured format — and transforms it into a histogram output. The histogram output is then streamed to the host device as a data stream (e.g. a bit stream). The streaming may use a data bus and / or a data bus protocol, such as MIPI.

[0069] There are embodiments wherein the sensor comprises a process for generating the histogram output that comprises formatting the packaged time-of-flight histogram data in a data block, wherein the bit length of each line of the data block is identical.

[0070] The formatting process involves organizing the data into lines within the data block, where each line has an identical bit length. This means that the data may, for example, be represented as a list of bit streams, each line of the list having an identical length.

[0071] There are embodiments wherein the sensor formats the packaged time-of-flight histograms by sequentially assigning them to a line of the data block and, if a remaining bit length of a line is insufficient for another histogram, assigning a first portion of the next histogram to the current line and the remainder to a subsequent line.

[0072] The formatting comprises placing each packaged histogram onto a line until the line reaches its capacity in terms of bit length. If the remaining bit length of a line is insufficient to accommodate another complete histogram, the process divides the histogram. The first portion of this histogram is assigned to the current line, filling it to capacity. The remainder of the histogram is then assigned to the next line in the data block. In the sense of the present disclosure, “assigning of a time-of-flight histogram” may be understood as generating a line of the data block such that a bin value of the histogram is set to be equal to a corresponding data stream segment in the data block.

[0073] There are embodiments wherein the sensor further comprises a skip frame manager, configured to determine, based on the streaming efficiency of the packaged time-of-flight histogram data, that the data related to the current imaging frame should not be streamed to a host device.

[0074] The skip frame manager is a component within the sensor tasked with evaluating a streaming efficiency of the packaged time-of-flight histogram data.

[0075] The streaming efficiency refers may be seen as the effectiveness of data transmission, assessed based on parameters such as transmission speed and resource consumption during data transfer. The streaming efficiency may be quantitatively determined and compared against a predetermined threshold. The threshold may either be set based on optimal operational parameters as would be expected if the packaged time-of-flight data is streamed from the sensor to the host device at an optimal streaming rate. The threshold may also be set based on a frame duration as described hereinabove.

[0076] The current imaging frame may be the specific set of data captured during a single imaging frame by the sensor during a current imaging cycle.

[0077] The skip frame manager is configured to make decisions about whether or not to transmit the current imaging frame's data to the host device based on the streaming efficiency. If the efficiency is below the predetermined threshold the manager will decide not to stream the data to avoid frame timing conflicts with subsequent imaging frames, where the frame rate would drop due to deterioration in compression ratio.

[0078] There are embodiments wherein the skip frame manager is further configured to relate a calculated output period of the packaged time-of-flight histogram data with an allocated output period to generate the streaming efficiency, where the calculated output period is the expected time required to output the data.

[0079] The streaming efficiency may be a target compression ratio. The data output may be, for example, one frame of histogram image data. The compression ratio indicates the efficiency of the data compression process. The compression ratio may, for example, be calculated by dividing the size of the data before compression by the size after compression. The calculated output period is the expected time required to output the packaged time-of-flight histogram data from the sensor to the host device. This period is determined based on the actual data processing and packaging speeds within the sensor.

[0080] Allocated output period is the time period designated or planned for the transmission of the packaged data. This period a predefined target period. The predefined period may correspond to the frame duration of the sensor or may be user-set. The predefined period may also be based on optimal transmission conditions based on a target compression ratio.

[0081] There are embodiments wherein the skip frame manager is further configured to output an event handling flag to a host device.

[0082] The event handling flag is a signal or data marker sent from the skip frame manager to the host device. This flag indicates to the host device that the calculated output period exceeds the allocated output period. Thus, the event handling flag is output if the output of the current frame were to continue, subsequent frames would be delayed and could violate the configured frame timing based on the target compression ratio.

[0083] This event handling flag serves to inform the host device of the specific action taken or needed, allowing the host device to adjust its operations accordingly.

[0084] The event handling flag may be sent to the host device via an Improved Inter-Integrated Circuit (I3C) or an Inter-Integrated Circuit (I2C) .

[0085] Alternatively, the event handling flag may be output to the host device via a Serial Peripheral Interface (SPI), a Universal Asynchronous Receiver / Transmitter (UART) or a Universal Serial Bus (USB).

[0086] There are embodiments wherein the sensor further comprises a compression ratio monitor, configured to monitor a compression ratio of the packaged time-of-flight histogram data.

[0087] The compression ratio monitor is a module within the sensor that continuously assesses the ratio of the original data size to the compressed data size.

[0088] If the compression ratio falls below a certain threshold, indicating less efficient compression, the monitor can trigger alerts or initiate processes to adjust the compression parameters or to investigate and rectify the underlying causes. Specifically, the compression ratio monitor may cause the skip frame manager to output an event handling flag or cause the host device to provide an updated encoding dictionary. There are embodiments wherein the compression ratio monitor is further configured to relate the bit-length of the compressed time-of-flight histogram data and the time-of-flight histograms to calculate the compression ratio.

[0089] The compression ratio may be calculated by the compression ratio monitor by dividing the bitlength of the original time-of-flight histograms by the bit-length of the compressed time-of-flight histogram data. This ratio provides a quantitative measure of the compression effectiveness, indicating how many times smaller the data becomes after compression.

[0090] By continuously calculating and monitoring the compression ratio, the monitor can detect any deviations from expected performance, which might suggest issues such as suboptimal compression settings or anomalies in the data. Monitoring the compression ratio allows for timely adjustments to the compression process, for example between two subsequently output frames.

[0091] A time-of-flight imaging host device according to the present disclosure comprises a decoding unit, configured to decode, based on an encoding dictionary, packaged time-of-flight histogram data to retrieve time-of-flight histogram data, and an encoding dictionary update unit, configured to generate an updated encoding dictionary based on the packaged and retrieved data.

[0092] Decoding unit is a module in the host device that is responsible for converting the packaged time-of-flight histogram data back into its original, uncompressed form.

[0093] The decoding is performed based on an encoding dictionary, which contains the rules and mappings used during the initial compression of the data. This dictionary ensures that the decoding process accurately reconstructs the original data from the compressed format.

[0094] The decoded data may then be used by other modules or processes not mentioned heretofore. Specifically, the decoded data may be used for further processing or analysis, for example to calculate a distance to an imaged object, for object recognition, for modelling a surrounding of the sensor, for simultaneous localizations and mapping (SLAM) or other applications. The decoded data may be forwarded to other devices, processing circuitry and the like, or be displayed to a user. The encoding dictionary used by the host device to decode received data is the same as the encoding dictionary used to encode the received data. The decoding is the inverse of the encoding.

[0095] The encoding dictionary update unit is a module on the host device that maintains and / or updates the encoding dictionary used for decoding the data by the host device and used for encoding the data by the sensor.

[0096] The encoding dictionary update unit generates an updated encoding dictionary based on the packaged data received and the data retrieved through the decoding process. The purpose of updating the dictionary is to adapt to any changes in data characteristics or to optimize the compression-decompression cycle based on observed encoding performance. The updated dictionary ensures that the encoding and decoding processes remain efficient across multiple frames. There are embodiments wherein the time-of-flight imaging host device is further configured to discard time-of-flight histogram data for which an associated event handling flag has been received. Event handling flag is a signal or marker transmitted from the sensor to the host device. The event handling flag serves, as discussed, as an indicator of specific conditions or events related to the data, such as anomalies detected during data transmission, data corruption, or other issues that might affect the quality or reliability of the data.

[0097] The host device comprises a mechanism or protocol configured to discard the time-of-flight histogram data when an event handling flag associated with that data is received. To discard in this context means that the host device actively chooses not to process, store, or utilize the flagged data for any further analysis, operations, or outputs.

[0098] As an alternative, the system may complete the transmission of the low compression ratio frame and compensate for the additional output streaming time by skipping the subsequent frame. This approach may aid in ensuring that future frames are processed efficiently even if the current dictionary proves inadequate and allow for updating the encoding dictionary using the low compression ratio frame data.

[0099] A time-of-flight imaging system according to the present disclosure comprises the time-of-flight sensor and the time-of-flight imaging host device, wherein the sensor is configured to output the packaged time-of-flight histogram data to the host device.

[0100] A time-of-flight image readout method according to the present disclosure comprises generating compressed time-of-flight histogram data from time-of-flight histogram data based on an encoding dictionary using a processing unit in a time-of-flight sensor, streaming the compressed data to a time-of-flight imaging host device, and decoding the compressed data based on an encoding dictionary to retrieve the time-of-flight histogram data.

[0101] A computer program according to the present disclosure is a computer program that, when executed by circuitry, causes the circuitry to apply the method of generating, streaming, and decoding compressed time-of-flight histogram data as described.

[0102] The methods, modules and units as described herein are also implemented in some embodiments as a computer program causing a computer and / or a processor to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer-readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be performed. Methods, modules and units may be implemented using a field-programmable-gate-array (FPGA) .

[0103] Fig. 1 depicts a Time-of-Flight (ToF) system 3 consisting of a ToF emitter 5 that emits light signals Rl, a ToF sensor 1 that captures and processes the reflected signals R2 to generate compressed data, and a host device 2 that receives and decompresses the data for analysis. The host device 2, in addition, provides control for the ToF sensor 1.

[0104] The ToF emitter 5 emits light signals Rl towards a target object (not illustrated). The light signals Rl may be light pulses Rl. The ToF sensor 1 captures the reflected light signals R2 and processes the incoming data. The data may, in particular, be ToF histograms as described hereinabove.

[0105] The ToF sensor 1 acquires, according to the process of known direct ToF sensors, data from the light signals R2. The data acquired from the reflected light signals R2 may be used to calculate a distance to the object. Operation of the ToF emitter 5 and the ToF sensor 1 may be synchronized, as is typical for direct ToF measurements.

[0106] The sensor 1 encodes and compresses this data and transmits it to the host device 2. The sensor 1 may also comprise mechanisms to monitor and maintain the efficiency of data compression and transmission.

[0107] The host device 2 receives the compressed data stream from the ToF sensor 1. The host device 2 decompresses the received data to retrieve the ToF histograms for further analysis.

[0108] Fig. 2 depicts a ToF sensor 1 with a processing unit 100 that includes various components for encoding, packaging, and managing data transmission to a host device 2 according to the present disclosure.

[0109] The processing unit 100 may comprise several units (which may also be called modules): a readout unit 110 responsible for retrieving histogram data 10 from the sensor 1, an encoding unit 120 that encodes the histogram data 10 using, for example Huffman coding, and an encoding dictionary unit 130 wherein the Huffman dictionary is stored. Additionally, a packaging unit 140 is provided to concatenate and compress the encoded data. An output unit 160 manages the transmission of the compressed data stream 20 to the host device 2. A skip frame manager 170 is provided to bypass frames. The skip frame manager may be configured to output an event handler flag 320, for example via a data bus, to the host device 2. The event handler flag may be output using a I2C or I3C connection, or any other data transfer interface known to the skilled person. On the host device 2 side, a decoding unit 210 receives and decompresses the data stream, reconstructing the original histogram data for further processing. The host device 2 also comprises an input bus 310, which may be a I3C Master, which facilitates communication between the host 2 and the processing unit 100. Specifically, the input bus may carry encoding dictionary updates. A dictionary update unit 220 comprised in the host device 2 is tasked with updating and sending the encoding dictionary to the processing unit’s 100 encoding dictionary unit 230 whenever necessary, based on output of the compression ratio monitor 150.

[0110] The device 1 integrates a system for on-chip lossless compression of direct Time-of-Flight (TOF) histogram data, designed to address challenges of increased data volume and associated power consumption and delays in data transmission. This system comprises filtering algorithms, such as median edge filtering and Paeth predictors, adapted from PNG compression techniques.

[0111] The sensor 1 transmits the histogram output stream 20 via an adequate output bus 330. The output bus 320 may be a Mobile Industry Processor Interface (MIPI) bus interface. However, there are embodiments wherein the output bus is a USB output, a Peripheral Component Interconnect Express (PCIe), a I2C or I3C, a Serial Peripheral Interface (SPI) or any other data transfer interface known to the skilled person.

[0112] The combination of the sensor 1 and the host device 2 may be seen as a time-of flight imaging system.

[0113] Fig. 3 illustrates a sequence of frames (Frame 1, Frame 2 ... Frame N) captured by the TOF sensor 1 and the corresponding frame data (output frame 1, output frame 2, output frame N) output by the sensor 1.

[0114] Fig. 3 shows a sequence of frames (Frame 1, Frame 2 ... Frame N). Each frame (Frame 1, Frame 2 . .. Frame N) in the sequence can be said to represent a snapshot taken at different time intervals along the time axis (as indicated), capturing depth information based on the light pulses R1 emitted by the TOF system 3 and their reflections R2 from objects within the scene.

[0115] Fig. 3 further shows corresponding frame data output (output frame 1, output frame 2, output frame N) by the TOF sensor 1 for each captured frame (Frame 1, Frame 2 ... Frame N).

[0116] This data output (output frame 1, output frame 2, output frame N) is output to the host device 2 in an output stream 20 as compressed and packaged data.

[0117] Fig. 4a and Fig. 4b show the effects of median filtering and Paeth filtering on the pixel code frequency. A non-limiting example of an input spectrum is shown in Fig. 4a. Fig. 4b shows the spectrum of Fig.

[0118] 4a the spectrum after the application of median filtering. As may be seen, median filtering leads to a clustering of the pixel data values of the input data around zero.

[0119] Such a tight clustering around the 0 data code is optimal for subsequent Huffman coding.

[0120] Median filtering, as depicted in the figure, utilizes the concept based on pixel neighbors, focusing on identifying the median value among the neighbors to adjust the histogram bin of interest. This method effectively reduces the range of data by centering the values closer to zero, as detailed, for example in the non-patent literature "Exclusive-or preprocessing and dictionary coding of continuous-tone images" from louisville.edu.

[0121] Faeth filtering, on the other hand, is demonstrated using a technique that selects the neighbor histogram bin closest in value to the target histogram bin and subtracts it, aiming to achieve a resultant value as close to zero as possible. This approach is well-documented in the PNG (Portable Network Graphics) standard, where Paeth filtering is used to optimize data before compression. Fig. 5a, Fig. 5b and Fig. 5c show a process of capturing and processing histogram data from a Time-of-Flight (TOF) image sensor.

[0122] Fig. 5a displays a group of four macropixels (30, 30 — NW, 30 — NE, 30 — SW, 30 — SE) as they appear on the TOF image sensor 1. This grouping represents a segment of the sensor array 21, where each macropixel 30 is responsible for capturing depth information (i.e. histograms) from the observed scene.

[0123] Fig. 5b is a more detailed view of one macropixel 30 shown in Fig. 5a. The macropixel 30 consists, for example, of four pixels 31.

[0124] Fig. 5c shows four histograms that are output by the four macropixels 30 depicted in Fig. 5a. The histograms represent the photon counting distribution (Bin value) across various bins (Bin number) for each macropixel 30 or pixel 31.

[0125] Fig. 6a and Fig. 6b show a more detailed view of the filtering process. Fig. 6a depicts the histograms acquired from a Time-of-Flight (TOF) image sensor before and Fig. 6b depicts the same histograms after the application of filtering processes. These histograms are derived from the corresponding TOF macropixels 30 on the sensor 1 and demonstrate the necessity and effectiveness of filtering in refining the data for further processing.

[0126] Fig. 6a displays four histograms as they are initially acquired on-chip from the TOF macropixels 30. Each histogram represents the bin counts that correspond to the depth information captured by the sensor 1. The histograms, shown in their raw form, in addition to capturing the target return light signal, also exhibit variations in bin counts which may include noise and other artifacts that can complicate data analysis and interpretation.

[0127] Fig. 6b shows the same four histograms following the application of filtering techniques. The filtering process involves using neighbor bins and histograms to adjust the histogram bin counts. This is achieved by selecting the most temporally correlated histogram bin from neighboring histograms and subtracting it from the selected histogram bin. The objective is to bring the histogram bin values closer to zero, thereby simplifying the data structure.

[0128] A comparison in the figure illustrates the impact of filtering on the histogram data.

[0129] For the filtering process, either a median or a Paeth filter as described with reference to Fig. 4a and Fig 4b is employed to select the most temporally correlated histogram bin from each histogram. This selected bin, which may be called a reference bin, is then subtracted from the histogram bin of interest, with the aim to encode the histogram data primarily in terms of shot noise, thereby achieving a significant reduction in the background bins' photon counting

[0130] Notably, even the first top-left histogram (30 - NW) in Fig. 6a can undergo filtering by comparing it with its internal neighbor bins, despite lacking neighboring histograms to the North (N), West (W), or Northwest (NW). The only exception is the first left-most bin RB, which remains unfiltered due to the absence of adjacent bins.

[0131] After the filtering stage, the processed histogram data is compressed using the Huffman coding pipeline, optimizing the data for transmission.

[0132] Fig. 7 illustrates the interaction between the encoding unit 120 and the encoding dictionary unit 130 within a processing unit 100.

[0133] The encoding unit 120 initially receives filtered histogram data 111 from the readout unit 110. This data has undergone filtering. The encoding unit 120 receives this filtered histogram data 111, and forwards the data to the encoding dictionary unit 130. The encoding dictionary unit 130 stores the encoding dictionary, which contains the specific codes and rules 131, which may be referred to as the encoding dictionary 131, used to convert the filtered histogram data 111 into a

[0134] Huffman / encoded format, namely encoded histogram data 124.

[0135] The encoding dictionary unit 130 may also receive updates, i.e., an updated encoding dictionary 132 for the encoding dictionary 131 from an external source, such as the host device 2, ensuring that the dictionary 131 is always optimized for the most efficient encoding possible based on the current data characteristics. After the filtered data 111 is processed using the encoding dictionary 131, the encoding dictionary unit 130 sends back encoded data 122 and the encoded data lengths 123 to the encoding unit 120.

[0136] The encoded data 122 represents the histogram data transformed into its Huffman-encodedform, while the encoded data lengths 123 provide information about the size (the number of significant bits) of each piece of encoded data. The encoding unit 120 then forwards the encoded data 122 along with the output data lengths 123 to the packaging unit 140.

[0137] The packaging unit 140 uses this information to properly compress and format the encoded histogram data into a structured output that can be efficiently transmitted to the next stage in the data processing pipeline or directly to the host device 2 for further use.

[0138] Fig. 8 provides a detailed depiction of the function of the encoding dictionary unit 130.

[0139] At the core of the encoding dictionary unit's 130 functionality is the Huffman encoding dictionary 131a. The Huffman encoding dictionary 131a transforms an input code ( left column in the table shown in fig. 8) into an output code (middle column in the table shown in Fig. 8) using a predefined dictionary.

[0140] The Huffman encoding dictionary 131a maps each possible input code to a corresponding output code (right column in the table in Fig. 8), which is typically shorter for more frequently occurring inputs, thereby achieving data compression.

[0141] The encoding process within the unit can be summarized by the formula:

[0142] Output code = dictionary(input code), [1] where the "dictionary" is a set of Huffman coding rules (i.e. a mathematical map) stored within the unit 130.

[0143] The encoding dictionary 131a is not static and can be updated to adapt to changes in data characteristics or to optimize compression efficiency. Updates to the dictionary 131a are received from the host device 2, for example via communication protocols such as I2C or I3C.

[0144] The updated encoding dictionaries 132 ensure that the dictionary 131a remains optimal for current data processing needs, allowing for better compression rates and more efficient data handling.

[0145] Fig. 8 also shows the function of the encoding dictionary unit 130 but also its connectivity with external devices. This connectivity allows the unit 130 to receive updated dictionaries 132, ensuring that the Huffman encoding process is continuously optimized for the best performance.

[0146] In general, Huffman encoding is a method for data compression that assigns variable-length codes to input values, with shorter codes assigned to more frequently occurring values. This technique leverages the frequency of occurrence of data items to create an efficient method of encoding information.

[0147] Generation of a Huffman dictionary 131a may be accomplished as follows:

[0148] The construction of the Huffman tree, a specific type of binary tree, begins by creating leaf nodes for each unique value in the data set, with the frequency of the value's occurrence as the associated weight. These nodes are initially placed in a priority queue or a similar structure based on their frequencies, with the least frequent nodes given higher priority for combination.

[0149] In the next step, two nodes with the lowest frequencies are repeatedly removed from the queue and combined into a new node. This new node’s frequency is the sum of the frequencies of the two nodes that were combined. The new node then becomes an internal node in the Huffman tree, with the two original nodes as its children — one on the left and one on the right. The new node is then reinserted into the priority queue.

[0150] This process of merging the two nodes with the lowest frequency continues until there is only one node left in the queue. This node acts as the root of the Huffman tree. The path from this root node to any leaf node defines the Huffman code for the value at that leaf node: traversing left in the tree adds a 'O' to the code, and traversing right adds a T'. Thus, the Huffman tree structure allows the generation of prefix-free binary codes (where no code is a prefix of any other), which are essential for effective data compression.

[0151] The role of the binary tree in this context is to organize the data values in such a way that the most frequently occurring values have the shortest codes, which significantly reduces the overall size of the encoded data when compared to using fixed-length codes for each value.

[0152] Fig. 9 illustrates, in a second embodiment, the interaction between the encoding unit 120, the encoding dictionary unit 130, and a global offset dictionary unit 133 within the processing unit 100. The encoding dictionary unit 130 is responsible for transforming the filtered data into a compressed format using a stored encoding dictionary 131. The encoding dictionary 131 contains predefined coding rules that convert the filtered histogram data into encoded data as described above with reference to Fig. 7 and Fig. 8. The encoding dictionary unit 130 processes the data and sends back two key pieces of information to the encoding unit 120: the encoded data 122 and the encoded data lengths 123, which indicate the size of each piece of encoded data.

[0153] Simultaneously, the encoded data lengths 123 are also sent to the global offset dictionary unit 133. The global offset dictionary unit 133 utilizes an offset dictionary 134 to determine the appropriate global offsets 126 based on the data lengths. The global offset dictionary unit 133 outputs these global offsets back to the encoding unit 120.

[0154] Back in the encoding unit 120, the global offsets are applied to the encoded data 122 to finalize the encoding process. The adjusted encoded data 122, along with the output data lengths 123, are then forwarded to the packaging unit 140. The packaging unit 140 packages the encoded histogram data into a structured format suitable for transmission or storage, completing the data processing cycle. Both the encoding dictionary unit 130 and the global offset dictionary unit 133 are capable of receiving updates from an external source, such as the host device 2. These updates may include an updated encoding dictionary 132 and an updated offset dictionary 135, respectively, ensuring that the data encoding process remains efficient and adapted to current data characteristics.

[0155] Fig. 10 provides a detailed view of the encoding dictionary unit 130 in the embodiment of Fig. 9 that utilizes both the encoding dictionary 131b and the global offset dictionary 134. This setup is designed to enhance the efficiency of the Huffman encoding process by optimizing the size of the dictionary and adjusting the code offsets dynamically.

[0156] The formula used by the encoder is: Code = global_offset(length) + huff_code_offset(input_code). Here, "global_offset(length)" refers to a value retrieved from the global offset dictionary 134, which provides an offset based on the code length. This offset is added to "huff_code_offset(input_code)", which is the specific offset for the given input code derived from the encoding dictionary 131b.

[0157] The global offset dictionary 134 stores offsets that are summed to the code offsets corresponding to each code length. This dictionary is optimized for size using techniques such as canonical and length-limited Huffman coding, which streamline the dictionary by standardizing the code lengths and limiting the maximum length of the codes. This can achieve a dictionary memory size reduction compared to a standard dictionary setup, with performances exceeding halving of the dictionary memory.

[0158] The present figure shows an example of a dictionary for 8-bit bins, where the bins take values ranging from 0 to 255 counts, with post-filtered possible values spanning from -255 to +255.

[0159] The figure also illustrates the flow of data through the encoding dictionary unit 130. Input pixel codes are fed into the unit, where they are processed to output the Huffman code offsets and the lengths of these codes. The global offset corresponding to each code length is then added to these Huffman code offsets to produce the final encoded output. Fig. 11 illustrates the interaction between the packaging unit 140 and the compression ratio monitor 150 within a data processing system, focusing on the final stages of data handling before transmission or storage.

[0160] The process begins with the packaging unit 140 receiving encoded histogram data 124 and corresponding data lengths 125 from the encoding unit 120. This data has already been processed through various stages of filtering and encoding to optimize its size and format for efficient transmission.

[0161] Within the packaging unit 140, a packing logic component takes the encoded histogram data 124 and data lengths 125 and organizes them into a structured format 142, which may be called frame output 142. This structured data can be accessed and decompressed by the receiving system, such as a host device 2.

[0162] Once the data is structured, the packaging unit 140 transfers a line synchronization signal 127 along with the data lengths 125 to the compression ratio monitor 150. The role of the compression ratio monitor 150 is to assess the efficiency of the data compression by calculating the compression ratio 151 for the frame. This ratio is determined by comparing the original data size to the compressed data size, providing a quantitative measure of the compression effectiveness. The line synchronization signal 127 indicates the streamout of a new line.

[0163] The compression ratio monitor 150 then outputs the calculated compression ratio 151, which serves as a feedback mechanism to evaluate the performance of the data compression process. If the compression ratio 151 is below a certain threshold, adjustments might be needed in earlier stages of data processing to improve the compression.

[0164] Finally, the packaging unit 140 takes the compressed frame data, now verified by the compression ratio monitor 150, and outputs it to the next destination, which could be a host device 2 or another component in the data processing pipeline. This output represents the culmination of the data processing workflow, where the data is now ready for transmission.

[0165] Fig. 12 provides a detailed depiction of the function of the packaging logic 141 within a data processing system, focusing on its role in preparing Huffman encoded data for transmission.

[0166] The core function of the packaging logic 141 is to concatenate Huffman encoded bins. This process involves removing the zero non-significant bits that were padded to the MSB of each code at the output of the encoder, based on the code length of each bin. By eliminating these unnecessary bits, the packaging logic 141 can reduce the size of the data, ensuring that only essential information is retained for transmission. Once the Huffman encoded bins are concatenated and optimized, the packaging logic 141 sends this streamlined data forward to be streamed out.

[0167] Additionally, a monitor component (i.e., the compression ratio monitor 150) may be integrated within this process to check that the compression ratio 151 is maintained above a certain threshold. This monitoring is essential to ensure that the Huffman dictionary used in the encoding process is still efficiently compressing the filtered codes. If the compression ratio 151 falls below the set threshold, it may indicate that the Huffman dictionary needs to be updated or that adjustments are required in the data processing to restore efficiency.

[0168] Fig. 13 illustrates the detailed function of the compression ratio monitor 150 within a data processing system, focusing on its role in evaluating the efficiency of data compression for histogram data.

[0169] The compression ratio monitor 150 receives two inputs: the encoded data length 125 and a line synchronization signal 127 indicating the streamout of a new line. These inputs are essential for calculating the compression ratio 151, which is a measure of the effectiveness of the data compression process. The data length 125 provides the size of this encoded data.

[0170] Within the compression ratio monitor 150, there are three main components that facilitate the calculation of the compression ratio 151: the code length adder, the line counter, and the compression ratio logic. The code length adder is responsible for summing up the lengths of the encoded data codes, providing a total that represents the amount of data after compression. The line counter tracks the number of lines or entries of histogram data processed, which is used as a denominator in the compression ratio calculation 151.

[0171] The compression ratio logic then takes the total encoded data length from the code length adder and the count from the line counter to compute the compression ratio 151.

[0172] The compression ratio (CR) 151 is a metric used to evaluate the efficiency of data compression processes, particularly in the context of histogram data in imaging systems. It is calculated using the formula:

[0173] CR = Total number of bits streamed for all histograms / (Number of histograms * Number of bins per histogram * Number of uncompressed bits per bin) [2] Formula [2] provides a quantitative measure of how effectively the data compression system reduces the size of the original data. The numerator, "total number of bits streamed for all histograms," represents the total amount of data, in bits, that is actually transmitted or stored after compression. The denominator is the product of three factors: the total number of histograms, the number of bins per histogram, and the number of uncompressed bits per bin, which together represent the total size of the data before compression.

[0174] Fig. 14 illustrates the operating principle of the skip frame manager 170 within a data processing system, focusing on its role in managing frame output under conditions of low compression ratios. The skip frame manager 170 operates by monitoring the compression ratio 151 of the current frame data. If the compression ratio 151 falls below a predetermined threshold, indicating inefficient compression, the manager 170 initiates a skip frame event. This decision is made at the end of the frame, allowing the system to complete processing of the current frame before any action is taken. Once a skip frame event is triggered, the skip frame manager 170 communicates with the host device 2, for example via the event handling flag 320 to signal the need for a new dictionary. The host device 2 is responsible for preparing and sending an updated dictionary that could potentially improve the compression ratio 151 for subsequent frames. However, if the updated dictionary is not prepared and transmitted back to the system in time for the next frame, the skip frame manager 170 might result in skipping multiple frames. This delay in updating the dictionary can lead to missed exposures, as frames that could not be efficiently compressed are not processed or transmitted. The process involves several key stages: readout of the frame data, compression of this data, and streaming of the compressed data. Concurrently, the dictionary update process is managed, where a decision to skip a frame due to an impending new frame start prompts the system to latch the new dictionary at the end of the dictionary update process.

[0175] Fig. 15 illustrates the operating principle of the skip frame manager 170 within a data processing system, focusing on its role in managing frame output under conditions of low compression ratios. The figure illustrates a smarter skipping strategy that enhances the efficiency of handling frames under varying compression conditions. This embodiment employs a Compression Ratio Monitor (CRM) 150 and introduces a method of row checkpoints to optimize the decision-making process regarding dictionary update timing.

[0176] The Compression Ratio Monitor (CRM) 150 plays a pivotal role in this system. It calculates the compression ratio 151 in real-time as the frame is being encoded line by line. This allows the CRM 150 to detect a "point of no return" — the point during the encoding process where it becomes evident that, even under the best compression scenario, there is too much data left to encode effectively within the desired compression ratio threshold 151. When this point is detected before the frame ends, the CRM 150 sends a signal to the host device 2 to initiate an update of the encoding dictionary, thereby reducing the lag time associated with dictionary updates. In addition to the CRM's 150 ongoing monitoring, this embodiment introduces a mode called "Row checkpoints." This mode involves setting conditions for a skip frame based on the progress of rows processed and the average compression ratio (CRav) achieved, compared against the compression ratio needed for the remaining rows (CRcomp) and a predefined compression ratio threshold (CRth). The condition can be expressed as Formula [3]:

[0177] Rows done * CRav + Rows remainCRcomp > CRth * total rows. [3] A predictor is inserted for the remaining rows based on the probability that processing these rows will recover the overall compression ratio to an acceptable level. If the calculated CRcomp becomes too large, indicating that the desired compression ratio is unlikely to be achieved, a signal is sent to the host device 2 to start building a new dictionary. This approach allows for adjustments to be made before the frame encoding and streaming completes, enhancing responsiveness and efficiency. The skip frame decision is made when a new frame is supposed to start, and if a new dictionary is needed, it is latched at the end of the dictionary update process. This method ensures that exposure is not missed unnecessarily and that readout, compression, and streaming processes are optimized for both speed and data integrity.

[0178] Fig. 16 provides a detailed depiction of the operating principle of the encoding dictionary unit 130 within a data processing system.

[0179] At the core of the encoding dictionary unit 130 is the encoding dictionary 131 itself, which is responsible for storing the output code and the code length corresponding to each input code. This dictionary 131 is typically received from an external source, such as the host device 2, which has the capability to update the dictionary 132 as needed. Updates 132 may occur in response to a detected drop in the compression ratio 151 or based on other algorithms running on the host device 2 that determine an update is necessary to maintain optimal compression efficiency.

[0180] The encoding dictionary 131 can be encoded by the host device 2 in a canonical form. This approach potentially reduces the size of the dictionary memory by only storing the code length and an offset, rather than the full output codes, thereby optimizing the use of memory resources within the unit 130.

[0181] Connected to the encoding dictionary 131 is a dictionary buffer, which consists of latches that temporarily store data during the encoding process. This buffer ensures that the data remains stable and accessible for the encoding operations.

[0182] A series of multiplex devices 137 are connected to this buffer. These multiplex devices 137 manage the flow of data within the unit 130. They receive the input code through a commensurate series of input buses 133. Each multiplex device 137 selects the appropriate data from the buffer based on the input code it receives and outputs the corresponding encoded data.

[0183] The input code is received through multiple channels, labeled as N channels, indicating the parallel processing capability of the unit 130 to handle multiple data streams simultaneously.

[0184] Correspondingly, the output from the multiplex devices 134 comprises both the output code and its length, also across N channels, ensuring that each piece of input data is processed efficiently and accurately.

[0185] Additionally, the encoding dictionary unit 130 comprises an I3C slave interface, which facilitates communication and data transfer with the host device 2, particularly for receiving updates 132 to the dictionary 131. The updated dictionary data 132 can be transmitted via I3C and stored in either flipflops (FFs) or static random-access memory (SRAM), depending on the system design and requirements.

[0186] Fig. 17 illustrates the operating principle of the output unit 160 within the processing unit 100 in one embodiment, focusing on the handling and transmission of histogram data. The output unit 160 is designed to manage the final stage of data processing, ensuring that the data is properly formatted and ready for external use or storage.

[0187] One function of the output unit 160 is the management of Standard Histogram Readout, which involves handling data that typically consists of M x N constant word-length data. This format is structured to maintain a uniform size for each data word, facilitating predictable and efficient data handling and transmission.

[0188] The process begins with the output unit 160 receiving encoded histogram data 124, which has been processed by earlier stages in the system to compress and encode the raw histogram data according to predefined criteria. This data arrives at the output unit 160 in a structured format, ready for the final processing steps.

[0189] Within the output unit 160, the primary task is to ensure that the histogram data is correctly aligned and formatted according to the standard readout specifications. This involves arranging the data into the M x N matrix format, where each element represents a constant word-length piece of data, for example a histogram 30-1 as shown in Fig. 6c.

[0190] Once the data is properly formatted, the output unit 160 then manages the transmission of this data to external systems or storage solutions. This step is vital as it represents the final output of the processed data from the system, making it available for further analysis, display, or long-term storage. Fig. 18 provides a more detailed depiction of the operating principle of the output unit 160 in another embodiment, focusing on the readout process with compressed histogram bins. This embodiment highlights the dynamic handling of compressed data and the management of line buffers within the output unit 160.

[0191] In this system, the output unit 160 manages the readout of compressed histogram bins, where the width of each bin box visually represents the compressed bin code length. This visualization underscores the variability in data size resulting from the compression process, with each bin potentially having a different code length depending on the compression efficiency achieved for that particular bin.

[0192] The process begins with the line buffer of the readout block, which is designed with a fixed length. As compressed histogram data 124 is read out from the processing system, it is loaded into this line buffer. The filling of the buffer is continuously monitored, and once it reaches its capacity, a new line is started to accommodate additional data.

[0193] A key challenge addressed in this embodiment is the handling of histogram bins that do not fit entirely within the remaining space of the current line buffer, known as spillover. For instance, if Histo 2 bin does not fit in the current line due to its compressed size, the remaining part of this bin spills over to the next line. This spillover management maintains data integrity and ensure that each piece of data is correctly aligned and stored.

[0194] At the end of each frame, if it is the last line and there is still space left in the line buffer after accommodating the spillover, the output unit 160 must decide how to utilize this remaining space. Two options are typically available: filling the rest of the line buffer with dummy code (DC) to maintain the fixed length of the line or, if there is sufficient space, starting the readout of histograms from Histo 0 of the next frame.

[0195] Additionally, the output unit 160 sends an End of Frame signal once the last line of the current frame is processed..

[0196] Fig. 19 shows a process for managing and transmitting time-of-flight histogram data. The process comprises four sequential steps: the Reception Step SI where data is captured from the sensor's pixel array, the Encoding Step S2 using an on-sensor encoding dictionary to compress the data, the Streaming Step S3 to transmit the encoded data to a host device 2, and the Decoding Step S4 where the host device 2 decodes the data using its matching dictionary to retrieve the original histogram information. The figure presents a flow diagram illustrating a process for managing and transmitting time-of-flight histogram data within a time-of-flight sensor system. This diagram outlines the sequential steps involved in encoding, streaming, and decoding histogram data, highlighting the interactions between the sensor 1 and the host device 2.

[0197] The process initiates with the Reception Step SI, where time-of-flight histogram data is received from a pixel array within a time-of-flight sensor 1. This step involves capturing raw histogram data that represents the depth information based on the time delay of light reflected from objects within the sensor's 1 field of view.

[0198] Following the reception, the Encoding Step S2 takes place, utilizing an encoding dictionary stored within the time-of-flight sensor 1. The encoding dictionary contains predefined rules and codes that convert the raw histogram data into a compressed format, referred to as encoded time-of-flight histogram data 124.

[0199] Subsequently, the Streaming Step S3 involves streaming the encoded time-of-flight histogram data 124 to a host device 2. This step ensures the transmission of the compressed data from the sensor 1 to the host 2, where it can be further processed, analyzed, or stored. The streaming is typically performed over a communication interface that ensures the data integrity and timely delivery to the host device 2.

[0200] The final step in the process is the Decoding Step S4, where the encoded time-of-flight histogram data 124 is decoded using the encoding dictionary stored in the host device 2. This step reverses the encoding process to retrieve the original time-of-flight histogram data from the encoded format. The host device 2 uses its stored encoding dictionary, which mirrors the dictionary in the sensor 1, to accurately decode and reconstruct the histogram data.

[0201] Fig. 20 illustrates the general configuration of an electronic device 1200, which may be the ToF imaging system 3 that integrates various components to support a comprehensive range of functionalities, including a Time-of-Flight (ToF) system 1213.

[0202] Included in the device is the CPU 1201, which orchestrates the operations of other components by processing data and executing instructions. The CPU 1201 interacts directly with the RAM 1203, facilitating rapid access to data and applications currently in use, enhancing the overall speed and responsiveness of the device.

[0203] For storage 1202, the device comprises a solid-state disk (SSD), which provides robust and fast storage capabilities for the operating system, applications, and user data. The device is equipped with an advanced Image sensor 1210, which captures high-quality images and videos. This sensor 1210 works in conjunction with the ToF system 1213, which is designed to capture depth information by measuring the time it takes for light to travel to objects and back to the sensor 1210.

[0204] Connectivity is a key feature of the device, supported by a cellular antenna 1211, Bluetooth 1204, and WLAN 1205. The cellular antenna 1211 allows the device to connect to mobile networks for voice and data communication. Bluetooth 1204 facilitates local wireless connections with other Bluetooth-enabled devices, such as headphones and wearable technology. WLAN 1205 connectivity enables access to the internet and local network resources via Wi-Fi.

[0205] The User interface 1212 of the device comprises components that allow users to interact with the device, such as touchscreens, buttons, and possibly voice recognition systems.

[0206] Additionally, the device's circuitry may include interfaces for additional sensors or systems that enhance its functionality. These can include interfaces for environmental sensing, health monitoring, or other specialized applications, depending on the device's intended use.

[0207] It should be noted that the methods of filtering, encoding and packaging described hereinabove may also be used on image data, such as acquired by an image sensor, instead of ToF data.

[0208] It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is however given for illustrative purposes only and should not be construed as binding.

[0209] Please note that the division of the control into units is only made for illustration purposes and that the present disclosure is not limited to any specific division of functions in specific units.

[0210] A method for controlling an electronic device, such as mobile terminal 1200 discussed above, is described in the following and under reference of Fig. 20. The method can also be implemented as a computer program causing a computer and / or a processor, such as processor XY discussed above, to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer-readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the method described to be performed.

[0211] All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software. In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure.

[0212] Note that the present technology can also be configured as described below.

[0213] (1) A time of flight sensor, comprising a processing unit, the processing unit comprising an encoding unit, configured to generate compressed time-of-flight histogram data from one or more time-of-flight histograms based on an encoding dictionary stored in an encoding dictionary unit; and whereinthe one or more time-of-flight histograms are related to a current imaging frame obtained from a time-of-flight pixel array.

[0214] (2) The time-of-flight sensor according to any one of (1), wherein the processing unit and the time-of-flight pixel array are both provided on-chip.

[0215] (3) The time-of-flight sensor according to any one of (1), further comprising a readout unit configured to read out the time-of-flight histograms from the time-of-flight pixel array.

[0216] (4) The time-of-flight sensor according to any one of (3), wherein the readout unit is further configured to generate filtered time-of-flight histogram data by applying a filter to the time-of-flight histograms.

[0217] (5) The time-of-flight sensor according to any one of (4), wherein the filter is at least one of a Paeth filter or a median filter.

[0218] (6) The time-of-flight sensor according to any one of (4), wherein the filter is configured to filter a plurality of the time-of-flight histograms, each of which associated with a separate time-of-flight pixel or associated with a separate time-of-flight macropixel, wherein the macropixel comprises a plurality of time-of-flight pixels.

[0219] (7) The time-of-flight sensor according to any one of (6), wherein the plurality of time-of-flight pixels or time-of-flight macropixels is a group of four time-of-flight pixels or time-of-flight macropixels, respectively, that are adjacent to each other.

[0220] (8) The time-of-flight sensor according to any one of (6), wherein a histogram value of one bin of the plurality of the time-of-flight histograms is used as a reference value to filter the plurality of time-of-flight histograms.

[0221] (9) The time-of-flight sensor according to any one of (4), wherein the filter is configured to filter bins of the same one of the time-of-flight histograms. (10) The time-of- flight sensor according to any one of (4), wherein the encoding unit is configured to generate the compressed time-of-flight histogram data by encoding the filtered time-of-flight histogram data based on the encoding dictionary.

[0222] (11) The time-of-flight sensor according to any one of (1), wherein the encoding dictionary is a Huffman encoding dictionary.

[0223] (12) The time-of-flight sensor according to any one of (1), wherein the encoding dictionary unit is further configured to update the encoding dictionary based on an updated encoding dictionary received from a host device.

[0224] (13) The time-of-flight sensor according to any one of (1), wherein the encoding unit is further configured to generate the compressed time-of-flight histogram data based on an offset dictionary stored in an offset dictionary unit.

[0225] (14) The time-of-flight sensor according to any one of (13), wherein the offset dictionary unit is further configured to update the offset dictionary based on an updated offset dictionary received from a host device.

[0226] (15) The time-of-flight sensor according to any one of (1), further comprising a packaging module, configured to generate packaged time-of-flight histogram data by concatenating significant digits of the compressed time-of-flight histogram data.

[0227] (16) The time-of-flight sensor according to any one of (15), further comprising an output module, configured to generate, from the packaged time-of-flight histogram data, a histogram output.

[0228] (17) The time-of-flight sensor according to any one of (16), wherein the generating of the histogram output comprises formatting the packaged time-of-flight histogram data in a data block, wherein the bit length of each line of the data block is identical.

[0229] (18) The time-of-flight sensor according to any one of (17), wherein the formatting comprises sequentially assigning the packaged time of flight histograms to a line of the data block and, if a remaining bit length of a line of the data block is insufficient to assign another time-of-flight histogram, assigning a first portion of the another time-of-flight histogram to the line and assigning a remainder of the another time-of-flight histogram to a subsequent line of the data block.

[0230] (19) The time-of-flight sensor according to any one of (15), further comprising a skip frame manager, configured to determine, based on a streaming efficiency of the packaged time-of-flight histogram data, that the packaged time-of-flight histogram data related to the current imaging frame should not be streamed to a host device. (20) The time-of-flight sensor according to any one of (19), wherein the skip frame manager is further configured to relate a calculated output period of the packaged time-of-flight histogram data with an allocated output period to generate the streaming efficiency, wherein the calculated output period is an expected time required to output the packaged time-of-flight histogram data.

[0231] (21) The time-of-flight sensor according to any one of (20), wherein the skip frame manager is further configured to output an event handling flag to a host device.

[0232] (22) The time-of-flight sensor according to any one of (15), further comprising a compression ratio monitor, configured to monitor a compression ratio of the packaged time-of-flight histogram data. (23) The time-of-flight sensor according to any one of (22), wherein the compression ratio monitor is further configured to relate a calculated output timing of the packaged time-of-flight histogram data with a time slot allocated for outputting the packaged time-of-flight histogram data.

[0233] (24) A time-of-flight imaging host device comprising a decoding unit, configured to decode, based on an encoding dictionary, packaged time-of-flight histogram data to retrieve time-of-flight histogram data, the time-of-flight histogram data comprising one or more time-of-flight histograms related to a current imaging frame of a time-of-flight pixel array; and an encoding dictionary update unit; configured to generate, based on the packaged time-of-flight histogram data and the time-of-flight histogram data, an updated encoding dictionary.

[0234] (25) The time-of-flight imaging host device according to any one of (24) further configured to discard data time-of-flight histogram data associated for which an associated event handling flag has been received.

[0235] (26) A time-of-flight imaging system comprising the time-of-flight sensor according to any one of (1) to (23); and the time-of-flight imaging host device according to any one of (24) and (25), wherein the time-of-flight sensor is configured to output the packaged time-of-flight histogram data to the time-of-flight host device.

[0236] (27) A time-of-flight image readout method comprising generate, using a processing unit comprised in a time-of-flight sensor, compressed time-of-flight histogram data from time-of-flight histogram data based on an encoding dictionary; and stream the compressed time-of-flight histogram data to a time-of-flight imaging host device; and decode, based on an encoding dictionary, the compressed time-of-flight histogram data to retrieve the time-of-flight histogram data.

[0237] (28) A computer program that, when executed by circuitry, causes the circuitry to apply the method of (27). (29) A non-transitory computer-readable recording medium that stores therein a computer program product according to (28).

[0238] LIST OF REFERENCE SIGNS

[0239] 1 ToF sensor

[0240] 2 Host device

[0241] 3 ToF system

[0242] 5 ToF emitter

[0243] 10 Histogram data

[0244] 20 Compressed data stream / Output stream

[0245] 21 Sensor array

[0246] 30 Macropixel

[0247] 30 - NW Northwest macropixel

[0248] 30 - NE Northeast macropixel

[0249] 30 - SW Southwest macropixel

[0250] 30 - SE Southeast macropixel

[0251] 31 Pixel

[0252] 100 Processing unit

[0253] 110 Readout unit

[0254] 111 Filtered histogram data

[0255] 120 Encoding unit

[0256] 122 Encoded data

[0257] 123 Encoded data lengths

[0258] 124 Encoded histogram data

[0259] 125 Data lengths

[0260] 126 Global offset

[0261] 127 Line synchronization signal

[0262] 130 Encoding dictionary unit 131 Encoding dictionary

[0263] 131a Huffman encoding dictionary

[0264] 131b Encoding dictionary (with global offset) 132 Updated encoding dictionary

[0265] 133 Global offset dictionary unit

[0266] 134 Offset dictionary

[0267] 135 Updated offset dictionary

[0268] 137 Multiplex devices

[0269] 140 Packaging unit

[0270] 141 Packaging logic

[0271] 142 Structured format / Frame output 150 Compression ratio monitor

[0272] 151 Compression ratio

[0273] 160 Output unit

[0274] 170 Skip frame manager

[0275] 210 Decoding unit

[0276] 220 Dictionary update unit

[0277] 310 Input bus (I3C Master)

[0278] 320 Event handling flag

[0279] 330 Output bus (MIPI or other interface) 1200 Electronic device

[0280] 1201 CPU

[0281] 1202 Storage (SSD)

[0282] 1203 RAM

[0283] 1204 Bluetooth

[0284] 1205 WLAN

[0285] 1210 Image sensor 1211 Cellular antenna 1212 User interface 1213 ToF system

Claims

CLAIMS1. A time-of-flight sensor, comprisinga processing unit, the processing unit comprisingan encoding unit, configured to generate compressed time-of-flight histogram data from one or more time-of-flight histograms based on an encoding dictionary stored in an encoding dictionary unit; and whereinthe one or more time-of-flight histograms are related to a current imaging frame obtained from a time-of-flight pixel array.

2. The time-of-flight sensor according to claim 1, wherein the processing unit and the time-of-flight pixel array are both provided on-chip.

3. The time-of-flight sensor according to claim 1, further comprisinga readout unit configured to read out the time-of-flight histograms from the time-of-flight pixel array.

4. The time-of-flight sensor according to claim 3, whereinthe readout unit is further configured to generate filtered time-of-flight histogram data by applying a filter to the time-of-flight histograms.

5. The time-of-flight sensor according to claim 4, whereinthe filter is at least one of a Paeth filter or a median filter.

6. The time-of-flight sensor according to claim 4, whereinthe filter is configured to filter a plurality of the time-of-flight histograms, each of which associated with an separate time-of-flight pixel or associated with an separate time-of-flight macropixel, wherein the macropixel comprises a plurality of time-of-flight pixels.

7. The time-of-flight sensor according to claim 6, whereinthe plurality of time-of-flight pixels or time-of-flight macropixels is a group of four time-of-flight pixels or time-of-flight macropixels, respectively, that are adjacent to each other.

8. The time-of-flight sensor according to claim 6, whereina histogram value of one bin of the plurality of the time-of-flight histograms is used as a reference value to filter the plurality of time-of-flight histograms.

9. The time-of-flight sensor according to claim 4, whereinthe filter is configured to filter bins of a same one of the time-of-flight histogram.

10. The time-of-flight sensor according to claim 4, whereinthe encoding unit is configured to generate the compressed time-of-flight histogram data by encoding the filtered time-of-flight histogram data based on the encoding dictionary.

11. The time-of-flight sensor according to claim 1, whereinthe encoding dictionary is a Huffman encoding dictionary.

12. The time-of-flight sensor according to claim 1, whereinthe encoding dictionary unit is further configured to update the encoding dictionary based on an updated encoding dictionary received from a host device.

13. The time-of-flight sensor according to claim 1, whereinthe encoding unit is further configured to generate the compressed time-of-flight histogram data based on an offset dictionary stored in an offset dictionary unit.

14. The time-of-flight sensor according to claim 13, whereinthe offset dictionary unit is further configured to update the offset dictionary based on an updated offset dictionary received from a host device.

15. The time-of-flight sensor according to claim 1, further comprisinga packaging module, configured to generate packaged time-of-flight histogram data concatenating significant digits of the compressed time-of-flight histogram data.

16. The time-of-flight sensor according to claim 15, further comprisingan output module, configured to generate, from the packaged time-of-flight histogram data a histogram output.

17. The time-of-flight sensor according to claim 16, whereinthe generating of the histogram output comprises formatting the packaged time-of-flight histogram data in a data block, wherein the bit length of each line of the data block is identical.

18. The time-of-flight sensor according to claim 17, whereinthe formatting comprises sequentially assigning the packaged time of flight histograms to a line of the data block and, if a remaining bit length of a line of the data block is insufficient to assign another time of flight histogram, assigning a first portion of the another time-of-flight histogram to the line and assigning a remainder of the the another time-of-flight histogram to a subsequent line of the data block.

19. The time-of-flight sensor according to claim 15, further comprisinga skip frame manager, configured to determine, based on a streaming efficiency of the packaged time-of-flight histogram data, that a the packaged time-of-flight histogram data related to the current imaging frame should not be streamed to a host device.

20. The time-of-flight sensor according to claim 19, whereinthe skip frame manager is further configured to relate a calculated output period of the packaged time-of-flight histogram data with an allocated output period to generate the streaming efficiency, wherein the calculated output period is an expected time required to output the packaged time-of-flight histogram data.

21. The time-of-flight sensor according to claim 20, whereinthe skip frame manager is further configured to output an event handling flag to a host device.

22. The time-of-flight sensor according to claim 15, further comprisinga compression ratio monitor, configured to monitor a compression ratio of the packaged time-of-flight histogram data.

23. The time-of-flight sensor according to claim 22, whereinthe compression ratio monitor is further configured to relate a calculated output timing of the packaged time-of-flight histogram data with a time slot allocated for outputting the packaged time-of-flight histogram data.

24. A time-of flight imaging host device comprisinga decoding unit, configured to decode, based on an encoding dictionary, packaged time-of-flight histogram data to retrieve time-of-flight histogram data, the time-of-flight histogram data comprising one or more time-of-flight histograms related to a current imaging frame of a time-of-flight pixel array; anda encoding dictionary update unit; configured to generate, based on the packaged time-of-flight histogram data and the time-of-flight histogram data, an updated encoding dictionary.

25. The time-of flight imaging host device according to claim 24 further configured to discard data time-of-flight histogram data associated for which an associated event handling flag has been received.

26. A time-of flight imaging system comprisingthe time-of-flight sensor according to any of claims 1 to 23; andthe time-of flight imaging host device according to any of claims 24 and 25, wherein the time-of-flight sensor is configured to output the packaged time-of-flight histogram data to the time-of-flight host device.

27. A time-of-flight image readout method comprisinggenerate, using a processing unit comprised in a time-of-flight sensor, compressed time-of-flight histogram data from time-of-flight histogram data based on an encoding dictionary; and stream the compressed time-of-flight histogram data to a time-of flight imaging host device; anddecode, based on an encoding dictionary, the compressed time-of-flight histogram data to retrieve the time-of-flight histogram data.

28. A computer program that, when executed by circuitry, causes the circuitry to apply the method of claim 27.

Images