Time-of-flight sensing method

Abstract

A method of time-of-flight sensing. The method includes emitting a pulse of radiation using an emitter and detecting the radiation reflected from an object using an array of photodetectors. For a given group of photodetectors in the array, the method determines, based on the measured time-of-flight of the radiation, whether to use a first operating mode in which the outputs from the individual photodetectors in the group are combined or a second operating mode in which the outputs from the individual photodetectors are processed individually. The array of photodetectors includes multiple photodetector groups. One or more photodetector groups operate in the first mode, while in parallel, one or more photodetector groups operate in the second mode.

Description

Technical Field

[0001] The present disclosure relates to a method of time-of-flight sensing and to a time-of-flight sensor system. Background Art

[0002] The present disclosure relates to a method for time-of-flight sensing. Time-of-flight sensing uses the known speed of light to determine the distance of an object from a sensor. In one example, a light pulse (e.g., an infrared wavelength) is emitted from a light source, and the light reflected from the object toward the sensor is detected. The source and sensor can be positioned adjacent to each other (e.g., as part of the same integrated system). The distance between the sensor and the object is determined based on the time elapsed between the emission of the light pulse and its detection by the sensor.

[0003] A time-of-flight sensor may be an imaging array comprising an array of pixels. The pixels may be, for example, single-photon avalanche photodiodes (or some other form of photodetector). The imaging array may provide a "depth map" that indicates, in the form of an image, the measured distance of an object from the sensor.

[0004] After the light reflects from an object, the intensity of the light received by the time-of-flight sensor decreases as a function of the square of the distance between the object and the sensor. Therefore, when the light reflects from a distant object (for example, at a distance of 5 meters or more), the signal-to-noise ratio at the sensor may be lower. In one known time-of-flight sensing method, in order to improve the signal-to-noise ratio for distant objects, the signals output from multiple pixels of the sensor are combined together. The multiple pixels can be arranged in a square or rectangle and can be referred to as macropixels. Because a macropixel has a larger surface area than a single pixel, it receives more photons of light, thereby providing an output with a better signal-to-noise ratio. However, grouping pixels together in this way is disadvantageous because it reduces the resolution of the depth map obtained.

[0005] In one known approach, this problem is addressed by obtaining one depth map from a time-of-flight sensor at low resolution (using macropixels) and another depth map from the time-of-flight sensor at high resolution (using individual pixels). The two depth maps are then added together by a processor. The resulting combined depth map consists of high-resolution areas where nearby objects exist and low-resolution areas where distant objects exist.

[0006] A disadvantage associated with the above approach is that the light source must be operated twice to obtain two depth maps. This uses a significant amount of power. This is particularly disadvantageous if the sensor forms part of a handheld device (e.g., a mobile phone), as the device's battery life is crucial to the user and operating the light source will drain the battery.

[0007] It is therefore an object of the present disclosure to provide a time-of-flight sensing method that addresses one or more of the above-mentioned problems or at least provides a useful alternative. Summary of the Invention

[0008] In general, the present disclosure proposes to overcome the above problems by operating groups of photodetectors in a first mode in which the outputs from the photodetectors are combined together, while simultaneously operating other groups of photodetectors in a second mode in which the outputs from the photodetectors are not combined together. The first mode can be used for distant objects and the second mode can be used for close objects. This arrangement advantageously provides a low resolution depth map for distant objects (while providing an acceptable signal-to-noise ratio) and simultaneously provides a high resolution depth map for close objects. This is advantageous because it avoids the requirement of the prior art to obtain a full low resolution depth map and then obtain a full high resolution depth map. Obtaining two full depth maps according to the prior art requires that more light pulses be emitted from the light source and therefore results in faster battery consumption.

[0009] According to one aspect of the present disclosure, a method of time-of-flight sensing is provided, the method comprising emitting a radiation pulse using an emitter, detecting radiation reflected from an object using an array of photodetectors, and determining, for a given photodetector group of the array, based on the measured time-of-flight of the radiation, whether to use a first operating mode in which the outputs from the individual photodetectors of the group are combined together or to use a second operating mode in which the outputs from the individual photodetectors are processed individually, wherein the array of photodetectors includes a plurality of photodetector groups, and wherein one or more photodetector groups operate in the first mode, while in parallel one or more photodetector groups operate in the second mode.

[0010] Thus, embodiments of the present disclosure advantageously allow distance data of different resolutions to be obtained in parallel from different portions of the same photodetector array.

[0011] When the method starts, the photodetector group may initially operate in a first operating mode.

[0012] When the method starts, all photodetector groups may initially operate in a first operating mode.

[0013] If the measured time of flight for a photodetector group indicates the presence of an object at a distance below a threshold distance, the method may switch from the first operating mode to the second operating mode for the photodetector group.

[0014] The method may delay switching to the second mode of operation until sufficient measured time-of-flight has been received at the pixel group in the first mode to provide a desired signal-to-noise ratio.

[0015] The method may immediately switch to the second operating mode.

[0016] If the measured time of flight for each photodetector does not indicate the presence of an object at the distance identified during the first operating mode, the method may switch back to the first operating mode.

[0017] The method may further include determining, for a given photodetector group of the array, based on a measured time-of-flight of the radiation, whether to use a third operating mode in which outputs from subgroups of photodetectors are combined, if the measured time-of-flight of the photodetector group indicates the presence of an object at a distance below a first threshold distance but above a second threshold distance, the method switching from the first operating mode to a third operating mode for the photodetector group, wherein one or more photodetector groups operate in the first mode and, in parallel, one or more photodetector groups operate in the third mode.

[0018] The method may further include operating one or more photodetector groups in a second mode.

[0019] Switching between operating modes for a given photodetector group may be controlled by logic circuitry associated with that photodetector group.

[0020] The logic circuit may form part of a circuit that is associated with the group of photodetectors and forms part of the same integrated circuit as the photodetectors.

[0021] The method can be restarted each time a light pulse is emitted.

[0022] The photodetector may be a single photon avalanche photodiode.

[0023] According to a second aspect of the invention, there is provided a time-of-flight sensor system comprising a transmitter configured to transmit radiation pulses, and a sensor module comprising a sensor and sensor electronics, wherein the sensor comprises an array of photodetectors, the photodetectors being arranged in groups, and wherein the sensor electronics comprises a plurality of circuits, the circuit being associated with each sensor group, and wherein each circuit comprises a logic circuit configured to determine, based on the measured time-of-flight of the radiation, whether to use a first operating mode in which the outputs from the individual photodetectors of the group are combined together or a second operating mode in which the outputs from the individual photodetectors are not combined together.

[0024] The logic circuitry may be configured to switch the circuitry from the first mode of operation to the second mode of operation for a group of photodetectors if a measured time of flight for the group of photodetectors indicates the presence of an object at a distance below a threshold distance.

[0025] The logic circuit can be configured to determine whether to use a third operating mode in which the outputs from the photodetector subgroups are combined based on the measured flight time of the radiation, if the measured flight time for the photodetector group indicates the presence of an object at a distance below the first threshold distance but above a second threshold distance, then switching from the first operating mode to the third operating mode occurs for the photodetector group.

[0026] The circuitry associated with the photodetector group may form part of the same integrated circuit as the photodetectors.

[0027] The circuitry associated with the photodetector group may also include a front end, a time-to-digital value converter, and memory.

[0028] The memory may be a histogram memory.

[0029] The photodetector may be a single photon avalanche photodiode.

[0030] Finally, the present time-of-flight sensing method disclosed herein utilizes a novel approach, at least in that higher resolution and lower resolution data are captured in parallel using a single sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Some embodiments of the present disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0032] Figure 1 schematically depicts a time-of-flight sensor system that may be operated using a method according to an embodiment of the present invention;

[0033] Figure 2 A method of time-of-flight sensing according to an embodiment of the present invention is schematically depicted;

[0034] Figure 3 A method of time-of-flight sensing according to another embodiment of the present invention is schematically depicted; and

[0035] Figure 4 is a flow chart depicting a method of operating a time-of-flight sensor according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] Generally speaking, the present disclosure provides a method of time-of-flight sensing in which higher resolution data and lower resolution data are captured in parallel.

[0037] Some examples of solutions are given in the accompanying drawings.

[0038] Figure 1A time-of-flight sensor system 100 that may operate in accordance with an embodiment of the present invention is schematically depicted. The time-of-flight sensor system 100 includes a light source 102, a sensor module 104, and an image processor 106. The sensor module includes a sensor 122 and sensor electronics 110. The light source 102 is configured to emit pulses of light (e.g., infrared radiation).

[0039] The sensor 122 is an array of photodetectors (e.g., single photon avalanche diodes). These may be referred to as pixels. The pixels may be arranged in groups. Figure 1 , nine pixel groups 131 - 139 are schematically depicted (other numbers of pixel groups may be provided). Each pixel group may be provided with its own associated electronics.

[0040] The sensor electronics 110 may be configured as a plurality of independently operating circuits. A circuit may be associated with each pixel group. Figure 1 In FIG. 1 , there are nine circuits 141 - 149 (one for each pixel group 131 - 39). Other numbers of circuits may be provided. Each circuit 141 - 149 of the sensor electronics 110 may be provided directly below the associated pixel 131 - 139. However, for ease of illustration, in FIG. Figure 1 , the sensor electronics 110 are depicted separately from the sensor 122. The sensor 122 and the sensor electronics 110 may be provided as a single integrated circuit.

[0041] exist Figure 1 The right-hand side of FIG depicts one of the circuits 143 in more detail. Circuit 143 is associated with one pixel group 133. Pixel group 133 may be referred to as a macropixel. Circuit 143 includes a so-called "front end" 112 that receives an analog voltage from each pixel of the macropixel and provides an output signal. In one example, the pixel is a single-photon avalanche photodiode. When a photon is incident on the photodiode, an avalanche effect occurs in the photodiode, and an analog voltage pulse is output from the photodiode. The analog voltage pulse may have a generally triangular shape. The analog voltage pulse is received by front end 112, and the front end outputs a digital pulse, i.e., a pulse having a generally rectangular shape.

[0042] The circuit 143 also includes a time to digital value converter 114. Figure 1 As depicted, the output from the light source 102 is connected to the sensor electronics 110. When a light pulse is emitted, this output starts the operation of a timer of a time-to-digital converter 114. When a pixel receives a reflected photon and the front end 112 outputs a signal, this enables the reading of an elapsed time value from the time-to-digital converter 114. This can be referred to as a timestamp.

[0043] The time values are stored in memory 116. Memory 116 may be a histogram memory. The histogram memory includes bins representing different elapsed times. When a photon is detected, the elapsed time value for that photon causes the bin corresponding to that time value to increment. Over time, many photons are detected and many associated elapsed times are measured. The data in the histogram memory represents the number of detected photons as a function of elapsed time. Peaks in the data indicate objects that have reflected photons. The times associated with the peaks indicate the distances to those objects. The distance resolution provided by the histogram memory (which may be referred to as depth resolution) will depend on the duration of each bin.

[0044] Circuitry 143 also includes logic circuitry 118. The logic circuitry determines whether the outputs from macropixels 133 should be used individually or combined (as explained further below). The logic circuitry associated with different macropixels can operate independently of each other. The logic circuitry can be, for example, a processor that performs this determination. The logic circuitry can be, for example, a hardwired circuit that performs this determination.

[0045] Circuitry 143 also includes a time-to-distance converter 120 that receives the histogram memory output in the form of digital representations of time and converts these times into distances. The outputs from time-to-distance converter 120 are passed to image processor 106, which combines the outputs to form an image depicting the distances to objects in the field of view of time-of-flight sensor system 100. This image can be referred to as a depth map.

[0046] Figure 2 A method of time-of-flight sensing according to an embodiment of the present invention is schematically depicted. Figure 2 In the schematic depiction of FIG, the sensor 222 includes an array of 12×12 single-photon avalanche diodes (which may be referred to as pixels). In other embodiments, the sensor array may be implemented as a CCD array or an array of PPD photodiodes. In the case of using a CCD or PPD, a memory other than a histogram memory may be used.

[0047] Pixels are grouped together, and each group is called a macropixel. There are nine macropixels 231-239. Each macropixel has 16 pixels ap (not all of which are labeled to simplify the illustration). It should be understood that this is merely an illustrative example, and in practice, sensor array 222 may consist of more than nine macropixels. Similarly, a macropixel may consist of more or fewer than sixteen pixels.

[0048] In a method according to an embodiment of the present invention, five of the macropixels 231-234, 239 operate in a first mode, in which the outputs of the individual pixels ap are combined. This combining of the outputs occurs before a time value is assigned to a detected photon. When a time value is assigned to a detected photon (which may be referred to as a timestamp), no information is recorded regarding which of the individual pixels ap detected the photon. Therefore, the data stored in the histogram memory relates to the macropixel as a whole, rather than to individual pixels.

[0049] Four of the macropixels 235-238 operate in a second mode, in which the outputs of the individual pixels ap are processed individually and not combined. When a time value is assigned to a detected photon (i.e., a timestamp), the identity of the pixel that detected the photon is recorded along with the time value. Thus, the data stored in the histogram memory relates to each individual pixel.

[0050] The first mode and the second mode operate in parallel.

[0051] Now combine Figure 1 and Figure 2 The first mode and the second mode are described. In the first mode, when the front end 112 associated with a macropixel (e.g., macropixel 231) receives an output from any one of the pixel aps of the macropixel, it causes the time value to be read out from the time-to-digital value converter 114. The histogram memory 116 is incremented accordingly. No data is recorded about which of the pixels of the macropixel 231 detected the photon. Over time, data is recorded in the histogram memory, which indicates the photons received at any pixel ap on the macropixel 231. This is relatively low-resolution data, but has a relatively high signal-to-noise ratio. For example, thousands of pulses can be emitted by the light source 102, and photons from at least some of the pulses are detected by the macropixel 231. As a result, timestamps are accumulated in the histogram memory. Peaks can be seen in the data in the histogram memory, indicating the presence of an object at a specific distance. When the peak has an acceptable signal-to-noise ratio, further accumulation of data in the histogram memory may add little or no useful information. The accumulation of timestamps may be stopped and data from the histogram memory may be transferred to the image processor 106. Accumulation of timestamps using the macropixels may then be restarted. Typically, data may be transferred from the macropixels periodically (i.e., after a predetermined time interval) or after a threshold level of peak signal-to-noise ratio has been reached.

[0052] The logic circuit 118 monitors the data held in the histogram memory to identify peaks in the data. The identified peaks are compared by the logic circuit 118 to a threshold time value indicating a predetermined object distance. In an embodiment, the threshold time value may correspond to an object distance of 2 meters. In terms of time values, this may be expressed as 1.33×10 -8 seconds (time = speed of light / distance). If a -8 If the threshold value in this example is a time value associated with 2 m, other time values may be used.

[0053] Macropixels 235-238 are schematically depicted as operating in a second operating mode. Taking macropixel 235 as an example, in the second operating mode, the signal output from each pixel 235a-p is processed separately. Therefore, when a photon is incident on a given pixel 235a, the front end 112 causes the time value to be read out from the time-to-digital value converter 114 and stored in a portion of the histogram memory specifically allocated for that pixel. Therefore, instead of storing a single data set in the histogram memory, nine data sets are stored in the histogram memory. This is relatively high-resolution data. The signal-to-noise ratio can be sufficiently high because the object reflecting the light is relatively close (e.g., within 2m).

[0054] The time-to-digital converter 114 may be capable of simultaneously receiving outputs from multiple pixels and converting these outputs into digital values for storage in a histogram memory. In an alternative approach, multiplexing may be used so that the time-to-digital converter 114 receives signals serially from each pixel AP of the macropixel 235. In one example, a raster scan of the pixel APs may be used so that a signal is received from pixel a, then from pixel b, and so on. In this case, the histogram memory may record data from all pixel APs and an identifier identifying the pixel from which the data was received.

[0055] Data is periodically transferred to the image processor 106. Transfer for a given pixel can occur when a peak in the data for that pixel has an acceptable signal-to-noise ratio (e.g., above a predetermined threshold). Alternatively, data transfer for a given pixel can occur after a predetermined time interval. Data transfer for all pixels 235a-p can occur simultaneously, or can occur at different times for different pixels. The timing of data transfer can depend on, for example, the size of the histogram memory and the degree to which multiplexing is used. Data transfer should occur before the histogram memory becomes too full.

[0056] The logic circuit 118 monitors the data held in the histogram memory to determine if there is a peak in the data. The identified peak is compared by the logic circuit 118 to a threshold time value (e.g., 1.33×10 -8 seconds) for comparison. If the identified -8 On the other hand, no peak with a time value less than 1.33×10 -8 If a peak in the time value of 100 ms is detected, the logic circuitry may instruct the macropixel to switch to operating in the first mode.

[0057] The image processor 106 generates a depth map composed of a combination of individual pixel outputs and macropixel outputs.Time of flight sensing is performed with high resolution data and low resolution data captured in parallel.

[0058] Embodiments of the present invention are advantageous because they provide a relatively high signal-to-noise ratio but low-resolution output when the sensed object is relatively far away, but automatically switch to a higher-resolution output when a closer object is present. The switch from low resolution to high resolution is performed instantly. That is, it does not rely on receiving instructions from the image processor 106 after the depth map (or other image) has been generated. Instead, the switch is performed by the logic circuit 118 based on the data in the histogram memory 116 (and can therefore be performed quickly). Similarly, the switch from high resolution to low resolution is also performed instantly via instructions from the logic circuit 118 and does not rely on receiving instructions from the image processor 106.

[0059] Embodiments of the present invention provide lower resolution data for the background of an image and higher resolution data for the foreground without the need to obtain two complete images. As further noted above, obtaining two complete images is disadvantageous because it requires the use of more energy to generate illumination for both images.

[0060] exist Figure 3 An alternative embodiment of the present invention is schematically depicted in FIG. Figure 2 , but instead of a single threshold value that determines the mode of operation, there are now two threshold values. It will be appreciated that in other embodiments, more than two threshold values may be used. Sensor 322 again includes nine macropixels 331-339, each having sixteen pixels ap (not all of which are labeled to simplify the illustration).

[0061] Combined with reference Figure 1 and Figure 3When logic circuit 118 determines that the distance to the object is less than 2 meters, the outputs from each pixel ap are processed individually. If logic circuit 118 determines that the object distance is between 2 meters and 5 meters, the outputs from groups of four pixels are combined. If the logic circuit determines that the object distance is greater than 5 meters, the outputs from the macropixels are used. Although the thresholds are expressed here in units of distance, they can equivalently be expressed in units of time. Other thresholds may also be used.

[0062] exist Figure 3 , four macropixels 331-333, 339 of the macropixels operate in a first mode, in which the outputs of all pixel ap from each macropixel are combined together. Three macropixels 336-338 of the macropixels operate in a second mode, in which the output of each pixel ap from the macropixel is processed separately. Two macropixels 334, 335 of the macropixels operate in a third mode, in which the outputs from groups of four pixels a, b, e, f; c, d, g, h; i, j, m, n; k, l, o, p are combined together. A group of four pixels may be referred to as a subgroup. In this third mode, the histogram memory 116 has four data sets, each data set representing a different group of four pixels. In other embodiments, the subgroups of pixels may have different numbers of pixels.

[0063] and Figure 2 As with the method shown, logic circuit 118 monitors the data held in the histogram memory to determine if there are peaks in the data. The identified peaks are compared to thresholds by logic circuit 118, and the macropixels are switched between the first, second, and third modes of operation based on the results of these comparisons. Figure 2 As with the method described in , switching is performed on the fly. Switching between modes does not depend on receiving instructions from the image processor 106.

[0064] Figure 3 The embodiment of is advantageous because it provides time-of-flight measurements with three different resolutions (and associated signal-to-noise levels) in parallel from the same sensor 322 .

[0065] Figure 4 is a flow chart depicting a time-of-flight sensing method according to an embodiment of the present invention. The method may be used, for example, Figure 3 For a macro pixel of the sensor array, the method starts at step 402 (eg, when a light pulse is emitted from the light source 102 - see Figure 1 ). At step 404, a low resolution acquisition is performed (e.g., the outputs from the sixteen pixels that make up the macro pixel are combined together). This may be referred to as a first mode of operation. At step 406, the logic circuit 118 (see Figure 1 ) The acquired data is checked to see if there are peaks indicating an object at a distance below a first threshold or an object at a distance below a second threshold. The thresholds in this example correspond to 2 meters and 5 meters. If the data indicates that there is no object at a distance less than 5 meters, a further low-resolution acquisition is performed at step 408. The low-resolution acquisition step is repeated multiple times to obtain more low-resolution data. The data is transmitted to the image processor 106 and the method begins again.

[0066] If a peak indicating an object at a distance of less than 2 meters is identified at step 406, the macropixel's operation switches at step 410 so that the output from a single pixel is acquired and stored individually in a histogram memory. This may be referred to as the second mode of operation. This high-resolution acquisition step is repeated multiple times to acquire more high-resolution data. The data is transferred to the image processor 106 and the method begins again. The low-resolution data acquired before switching to the second mode may be transferred to the image processor 106 as well as the high-resolution data.

[0067] If a peak indicating an object at a distance between 2 and 5 meters is identified at step 406, the outputs from the four-pixel subset of macropixels are combined and stored separately in a histogram memory. This is step 412 and may be referred to as the third mode of operation. This intermediate resolution acquisition step is repeated multiple times to obtain more intermediate resolution data. Periodically, the data is transferred to the image processor 106 and the method begins again. The low-resolution data acquired before switching to the second mode, as well as the intermediate resolution data, may be transferred to the image processor 106.

[0068] In one embodiment, once a switch to the second operating mode or the third operating mode occurs, the logic circuit 118 (see Figure 1 ) can examine the acquired data to monitor for peaks corresponding to those seen when operating in the first operating mode. That is, the data is examined to see if an object at the expected distance is seen. If the object is not seen, the logic circuitry can switch operation back to the first operating mode. The advantage of this approach is that it can provide data with a sufficiently high signal-to-noise ratio for use by the image processor if sufficiently strong signal-to-noise data is not seen in the second or third operating modes.

[0069] In an embodiment, data acquisition in the first operating mode may continue even after a peak corresponding to an object less than 5 m or less than 2 m has been identified. Data acquisition in the first operating mode may continue until a desired signal-to-noise ratio has been achieved. At this point, operation may switch to the second mode or the third mode. An advantage of this approach is that even if the second or third modes do not generate usable data (i.e., data with a sufficiently high signal-to-noise ratio), data with a sufficiently high signal-to-noise ratio will always be available for use by the image processor.

[0070] The above method can be modified. For example, instead of always starting with a low-resolution acquisition for each macropixel, the initial acquisition for a macropixel can be at any resolution previously used for the macropixel. In one example, if a video stream is being generated, the previously recorded distance data for that macropixel can be used to predict the optimal acquisition resolution for the next initial acquisition for that macropixel.

[0071] Although the described embodiments of the present invention use a threshold distance of 2 meters and optionally 5 meters, in other embodiments, other threshold distances may be used. Methods according to embodiments of the present invention may use a single threshold distance, may use two threshold distances, or may use more than two threshold distances. For example, three or more threshold distances may be used.

[0072] In the described embodiments, the threshold is primarily expressed in terms of distance. However, the threshold can be applied in terms of time rather than distance. For example, the threshold can be expressed as 1.33×10 -8 seconds (equivalent to 2 meters) and 3.325×10 -8 Second.

[0073] In the described embodiment of the present invention, each macropixel is composed of 16 individual pixels. In other embodiments, each macropixel may have a different number of individual pixels. Different groupings of these pixels may be used. For example, subgroups of pixels consisting of more than four pixels may be used.

[0074] Embodiments of the present invention allow for the parallel capture of a higher resolution foreground with more spatial details and a lower resolution background at an acceptable signal-to-noise level.

[0075] Applications of the present invention include image segmentation, for example for video conference calls. During a video conference call, the image can be segmented between a foreground represented at a higher resolution and a background represented at a lower resolution. This can be done, for example, if the background is not relevant to the call and the caller wishes to suppress it for privacy reasons (e.g., a work-related call being conducted from home). To achieve this in current systems, two-dimensional spatial images are obtained (as opposed to a depth map) and image processing is used to determine which part of those 2D spatial images is the foreground and which is the background. This approach uses a lot of power processing and will therefore consume a lot of energy from the battery of the mobile device. As explained elsewhere, energy consumption in mobile devices is undesirable. Embodiments of the present invention avoid this intensive processing.

[0076] By providing background and foreground measurements in parallel, embodiments of the present invention allow for more accurate perspective and detail to be provided in augmented reality systems. This can also be applied to virtual reality or mixed reality systems.

[0077] Embodiments of the present invention may advantageously allow for more accurate tracking of objects. The movement of an object across the sensor, measured in pixels per second, will be faster for relatively close objects and slower for relatively far objects (if the objects are traveling at the same speed). Because embodiments of the present invention scale the resolution based on the distance to the object, the same object traveling at the same speed can be tracked at different distances at different resolutions (effectively giving the same tracking information). Embodiments of the present invention allow for efficient tracking of objects even when the objects move through different distances from the sensor array. Embodiments of the present invention advantageously provide three-dimensional tracking (which is more useful than conventional two-dimensional tracking). Embodiments of the present invention may provide tracking over a wider range of distances from the sensor array (which may be referred to as depth) because switching between different resolutions may occur automatically.

[0078] exist Figure 1 , front end 112, time-to-digital converter 114, memory 116, logic circuitry 118, and time-to-distance converter 120 are depicted as circuitry 143, which is formed within the same integrated circuit as its associated macropixel 133. Circuitry 141-149 can be located below sensor 122. Alternatively, the circuitry can be located around the periphery of sensor 122. Providing the circuitry below sensor 122 may be preferred because it can provide scalability and superior performance.

[0079] It is not necessary for all components of a circuit to be within an integrated circuit. One or more of the components can be located remotely from the integrated circuit. For example, a time-to-distance converter can form part of a different integrated circuit. However, providing all components within the integrated circuit may be the most efficient configuration.

[0080] Typically, logic circuitry 118 may be provided for each group of pixels (which may be referred to as a macropixel).

[0081] Embodiments of the present disclosure may be employed in many different applications, including, for example, in mobile phones.

[0082] List of reference numerals:

[0083] 100 - Time of Flight Sensor System

[0084] 102——Light Source

[0085] 104——Sensor module

[0086] 106——Image Processor

[0087] 110 - Sensor electronics

[0088] 112 - Front-end

[0089] 114 - Time to Digital Converter

[0090] 116 - Memory

[0091] 118——Logic Circuits

[0092] 120——Time to Distance Converter

[0093] 122, 222, 322 - Sensors

[0094] 131-139 - sensor pixel groups (macro pixels)

[0095] 141-149——Circuit

[0096] 231-239 - sensor pixel groups (macro pixels)

[0097] 331-339 - sensor pixel groups (macro pixels)

[0098] ap - pixel of macro pixel

[0099] 402 - Start of method

[0100] 404 - Low-resolution retrieval

[0101] 406 - Data obtained by checking the logic circuit

[0102] 408 - Further low-resolution retrieval

[0103] 410 - Get from a single pixel

[0104] 412 - Get from group of four pixels

[0105] Those skilled in the art will understand that in the foregoing description and the appended claims, positional terms such as "above," "along," and "to the side" are made with reference to conceptual illustrations such as those shown in the accompanying drawings. These terms are used for ease of reference and are not intended to be limiting. Therefore, these terms should be understood to refer to an object when in the orientation shown in the accompanying drawings.

[0106] Although the present disclosure has been described in terms of the preferred embodiments described above, it should be understood that these embodiments are merely illustrative and that the claims are not limited to those embodiments. In light of this disclosure, those skilled in the art will be able to make modifications and substitutions, which are considered to fall within the scope of the appended claims. Each feature disclosed or shown in this specification may be incorporated in any embodiment, either alone or in any appropriate combination with any other features disclosed or shown herein.

Claims

1. A time-of-flight sensing method, the method comprising: Using a transmitter to emit a radiation pulse; using an array of photodetectors to detect radiation reflected from the object; for a given group of photodetectors of the array, determining based on the measured time of flight of the radiation whether to use a first operating mode in which the outputs from the individual photodetectors of the group are combined together or a second operating mode in which the outputs from the individual photodetectors are processed individually; characterised in that the array of photodetectors is an array of single photon avalanche diodes of photodetectors and comprises a plurality of photodetector groups, and wherein one or more photodetector groups operate in the first operating mode and in parallel one or more photodetector groups operate in the second operating mode, Switching between operating modes for a given photodetector group is controlled by independently operated logic circuitry associated with the group, wherein the independently operated logic circuitry is configured to determine the operating mode based on data in a histogram memory, wherein the data in the histogram memory is derived from measured flight times collected by the photodetector group. 2 . The method of claim 1 , wherein when the method starts, the photodetector group initially operates in the first operating mode. 3 . The method of claim 2 , wherein when the method starts, all photodetector groups are initially operated in the first operating mode.

4. The method of any one of claims 1 , wherein the method switches from the first operating mode to the second operating mode for a group of photodetectors if a measured time of flight for the group of photodetectors indicates the presence of an object at a distance below a threshold distance.

5. The method of claim 4, wherein the method delays switching to the second operating mode until sufficient measured time-of-flight has been received at the pixel group in the first operating mode to provide a desired signal-to-noise ratio. The method of claim 4 , wherein the method immediately switches to the second operating mode.

7. The method of claim 6, wherein the method switches back to the first mode of operation if the measured time of flight for the respective photodetectors does not indicate the presence of an object at the distance identified during the first mode of operation.

8. The method according to claim 1, comprising: For the given photodetector group of the array, determining whether to use a third operating mode in which the outputs from the photodetector subgroups are combined together is based on the measured flight time of the radiation, and if the measured flight time of the photodetector group indicates the presence of an object at a distance below a first threshold distance but above a second threshold distance, the method switches from the first operating mode to the third operating mode for the photodetector group, wherein one or more photodetector groups operate in the first operating mode and, in parallel, one or more photodetector groups operate in the third operating mode.

9. The method of claim 8, wherein one or more photodetector groups operate in the second operating mode.

10. A method according to claim 1, wherein the logic circuit forms part of a circuit associated with the set of photodetectors and forming part of the same integrated circuit as the photodetectors. The method of claim 1 , wherein the method restarts each time a light pulse is emitted.

12. A time-of-flight sensor system comprising a transmitter configured to transmit radiation pulses, and a sensor module comprising a sensor and sensor electronics; wherein the sensor comprises an array of photodetectors, the photodetectors being single photon avalanche photodiodes, the photodetectors being arranged in groups, and wherein the sensor electronics comprises a plurality of circuits, a circuit being associated with each photodetector group; And wherein each circuit includes logic circuitry configured to determine, based on the measured time-of-flight of the radiation, whether to use a first operating mode in which the outputs from the individual photodetectors of the group are combined together or a second operating mode in which the outputs from the individual photodetectors are not combined together, wherein switching between the operating modes for a given photodetector group is controlled by an independently operating logic circuit associated with the photodetector group, wherein the independently operating logic circuitry is configured to determine the operating mode based on data in a histogram memory, wherein the data in the histogram memory is derived from the measured time-of-flight collected by the photodetector group.

13. The system of claim 12, wherein the logic circuit is configured to switch the circuit from the first mode of operation to the second mode of operation for a group of photodetectors if a measured time of flight for the group of photodetectors indicates the presence of an object at a distance below a threshold distance.

14. The system of claim 12 , wherein the logic circuit is configured to determine whether to use a third operating mode in which outputs from a subset of photodetectors are combined based on a measured time-of-flight of the radiation, switching from the first operating mode to the third operating mode occurring for the group of photodetectors if the measured time-of-flight for the group of photodetectors indicates the presence of an object at a distance below a first threshold distance but above a second threshold distance.

15. The system of claim 12, wherein the circuitry associated with the set of photodetectors forms part of the same integrated circuit as the photodetectors.

16. The system of claim 12, wherein the circuitry associated with the photodetector group further comprises a front end, a time-to-digital value converter, and a memory. The system of claim 16 , wherein the memory is a histogram memory.

Images