Depth event camera

Abstract

Systems and techniques for determining a change in distance are described herein. For example, a device for determining a change in distance is provided. The device may include an EM emission emitter configured to emit electromagnetic (EM) radiation toward the environment; a detector configured to receive reflected EM radiation from the environment; a phase calculation circuit configured to calculate a phase difference value indicating the difference between the phase of the emitted EM radiation and the phase of the reflected EM radiation; and a difference circuit configured to trigger an event when the difference between the current phase difference value and the previous phase difference value exceeds a threshold.

Description

Technical Field

[0001] This disclosure generally relates to depth event cameras. For example, aspects of the present disclosure include systems and techniques for using a depth event camera to detect depth changes at points in an environment.

Background Art

[0002] An indirect time of flight (iToF) depth camera can emit light pulses and receive the light pulses reflected by various points in an environment. The iToF depth camera can calculate the time of flight for various points in the environment. The iToF depth camera can generate depth information indicating the distance between the iToF depth camera and various points in the environment.

Summary of the Invention

[0003] The following presents a simplified summary of one or more aspects disclosed herein. Thus, the following summary should not be considered an extensive overview of all contemplated aspects, nor should the following summary be regarded as identifying key or critical elements of all contemplated aspects or as delineating the scope associated with any particular aspect. Accordingly, the following summary presents in simplified form a particular conception of one or more aspects related to the mechanisms disclosed herein, prior to the "Detailed Description of the Invention" presented below.

[0004] Systems and techniques for determining a change in distance are described herein. According to at least one example, an apparatus for determining a change in distance is provided. The apparatus includes an EM emission emitter configured to emit electromagnetic (EM) radiation toward the environment; a detector configured to receive reflected EM radiation from the environment; a phase calculation circuit configured to calculate a phase difference value indicating the difference between the phase of the emitted EM radiation and the phase of the reflected EM radiation; and a difference circuit configured to trigger an event when the difference between the current phase difference value and the previous phase difference value exceeds a threshold.

[0005] Systems and techniques for determining a change in distance are described herein. According to at least one example, an apparatus for determining a change in distance is provided. The apparatus includes a memory and one or more processors coupled to the memory. The one or more processors are configured to cause at least one electromagnetic (EM) emitter to emit a first EM radiation toward the environment and, upon receiving the first EM radiation reflected from the environment, to calculate a first phase difference value indicating the difference between the phase of the emitted first EM radiation and the phase of the reflected first EM radiation; to cause at least one EM emitter to emit a second EM radiation toward the environment and, upon receiving the second EM radiation reflected from the environment, to calculate a second phase difference value indicating the difference between the phase of the emitted second EM radiation and the phase of the reflected second EM radiation; and to trigger an event when the difference between the second phase difference value and the first phase difference value exceeds a threshold.

[0006] Another example provides a method for determining a change in distance. The method includes emitting a first electromagnetic (EM) radiation toward the environment, receiving the first EM radiation reflected from the environment, calculating a first phase difference value indicating the difference between the phase of the emitted first EM radiation and the phase of the reflected first EM radiation, emitting a second EM radiation toward the environment, receiving the second EM radiation reflected from the environment, calculating a second phase difference value indicating the difference between the phase of the emitted second EM radiation and the phase of the reflected second EM radiation, and triggering an event when the difference between the second phase difference value and the first phase difference value exceeds a threshold.

[0007] In another example, a non-temporary computer-readable medium storing instructions is provided, and the instructions are configured such that, when executed by at least one processor, at least one processor causes at least one electromagnetic (EM) emitter to emit a first EM radiation toward the environment and, upon receiving the first EM radiation reflected from the environment, calculates a first phase difference value indicating the difference between the phase of the emitted first EM radiation and the phase of the reflected first EM radiation; at least one EM emitter to emit a second EM radiation toward the environment and, upon receiving the second EM radiation reflected from the environment, calculates a second phase difference value indicating the difference between the phase of the emitted second EM radiation and the phase of the reflected second EM radiation; and triggers an event when the difference between the second phase difference value and the first phase difference value exceeds a threshold.

[0008] In another example, a device for determining a change in distance is provided. The device includes means for emitting a first electromagnetic (EM) radiation toward an environment; means for receiving the first EM radiation reflected from the environment; means for calculating a first phase difference value indicating the difference between the phase of the emitted first EM radiation and the phase of the reflected first EM radiation; means for emitting a second EM radiation toward an environment; means for receiving the second EM radiation reflected from the environment; means for calculating a second phase difference value indicating the difference between the phase of the emitted second EM radiation and the phase of the reflected second EM radiation; and means for triggering an event in response to the difference between the second phase difference value and the first phase difference value exceeding a threshold.

[0009] In some embodiments, one or more of the devices described herein may be, be part of, or include, a mobile device (e.g., a mobile phone or so-called “smartphone”, a tablet computer, or other type of mobile device), an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a vehicle (or a computing device or system for a vehicle), a smart device or connected device (e.g., an Internet-of-Things (IoT) device), a wearable device, a personal computer, a laptop computer, a video server, a television (e.g., a network-attached television), a robotics device or system, or other devices. In some embodiments, each device may include an image sensor (e.g., a camera) or multiple image sensors (e.g., multiple cameras) for capturing one or more images. In some embodiments, each device may include one or more displays for displaying one or more images, notifications, and / or other displayable data. In some embodiments, each device may include one or more speakers, one or more light-emitting devices, and / or one or more microphones. In some embodiments, each device may include one or more sensors. In some cases, one or more sensors may be used to determine the location of the device, the status of the device (e.g., tracking status, operating status, temperature, humidity level, and / or other status), and / or for other purposes.

[0010] This summary is not intended to identify the main or essential features of the claimed subject matter, nor is it intended to be used independently to determine the scope of the claimed subject matter. The subject matter should be understood by referring to the entire specification of this patent, any or all of the drawings, and the appropriate portions of each claim.

[0011] The above, along with other features and embodiments, will become clearer with reference to the following specification, claims, and accompanying drawings.

[0012] An example for the purpose of explaining this application will be described in detail below with reference to the following figures. [Brief explanation of the drawing]

[0013] [Figure 1A] This figure illustrates a system including a device capable of detecting changes in distance within an environment, according to various aspects of this disclosure. [Figure 1B] This figure includes perspective views of a portion of the environment 116 and the device 102 for detecting changes in distance within the environment 116, according to various aspects of the present disclosure. [Figure 2] This figure illustrates a device capable of detecting changes in distance within an environment, according to various aspects of this disclosure. [Figure 3] This disclosure includes various aspects of scene images and scene depth event representations. [Figure 4] This includes images of objects on a mirrored surface and nearby transparent surfaces, a representation of depth change information corresponding to the images, and a representation of confidence values ​​corresponding to the images. [Figure 5] This includes images of the surface and surface edges, a representation of depth change information corresponding to the images, and a representation of confidence values ​​corresponding to the images. [Figure 6] This figure shows a vehicle including a depth event camera, according to some aspects of the present disclosure. [Figure 7] This is a flowchart illustrating the process for determining the change in distance according to the aspects of this disclosure. [Figure 8] This is a block diagram representing an exemplary computing device architecture for an exemplary computing device capable of implementing the various techniques described herein. [Modes for carrying out the invention]

[0014] Specific aspects of this disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects may be applied independently, and some may be applied in combination. The following specification provides specific details for illustrative purposes to give a complete understanding of the aspects of this application. However, it will be apparent that various aspects may be practiced without these specific details. The figures and specification are not intended to be restrictive.

[0015] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the following description of exemplary embodiments provides a feasible explanation for implementing the exemplary embodiments for those skilled in the art. It should be understood that various modifications can be made to the function and arrangement of the elements without departing from the spirit and scope of the application as set forth in the appended claims.

[0016] The terms “exemplary” and / or “example” are used herein to mean “serving as an example, case, or illustration.” No aspect described herein as “exemplary” and / or “example” should be construed as necessarily preferable or advantageous to any other aspect. Similarly, the term “aspects of the disclosure” does not require that all aspects of the disclosure include the described features, advantages, or modes of operation.

[0017] An indirect time-of-flight (iToF) depth camera may measure the phase difference between an emitted light pulse and a light pulse received by the iToF depth camera after the light pulse has been reflected by a point in the environment. Based on the speed of light and the frequency of the light pulse, the iToF depth camera may relate the phase difference to the time of flight of the light pulse between emission and reception. Based on the time of flight and the speed of light, the iToF depth camera may calculate the distance between the iToF depth camera and a point in the environment. In this disclosure, the term “depth” may refer to the distance between a point in the environment and a detector of a sensor (e.g., an iToF depth camera). An iToF depth camera may determine the depth for all points within the iToF depth camera’s field of view. For example, an iToF depth camera may include a detector array. Each detector may correspond to a portion of reflected EM radiation (e.g., rays between the detector and the environment, based on the reflected pulse being focused onto the array).

[0018] An iToF depth camera may require time to accumulate reflected light in the corresponding detector. Furthermore, an iToF depth camera may capture multiple pulses and average the phase difference across the multiple pulses. In addition, an iToF depth camera may operate frame by frame to determine the depth for all of its detectors, for example, and generate a representation of the depth measurement for points within the iToF depth camera's field of view.

[0019] Systems, apparatus, methods (also called processes), and computer-readable media (collectively referred to herein as “systems and techniques”) for determining changes in distance are described herein. For example, systems and techniques may be implemented in or by depth event cameras. Such depth event cameras may generate events for each detector when the distance between the depth event camera and a point in the environment changes.

[0020] In some aspects, a depth event camera can include an EM radiation emitter configured to emit electromagnetic (EM) radiation towards an environment. In some cases, the EM radiation emitter can modulate (e.g., amplitude modulate) the EM radiation based on a reference modulation signal. For example, the EM radiation emitter can multiply a carrier signal (e.g., having a carrier frequency) by the reference modulation signal. The reference modulation signal can define the envelope of the EM radiation.

[0021] The depth event camera can also include a detector array. Each detector of the array can correspond to a corresponding portion of the reflected EM radiation (e.g., a ray (in a particular direction) between the array and various points in the environment). For example, it may be possible to trace the corresponding ray outward from each detector of the detector array into the environment. In some cases, the system and technique can include a lens for focusing the reflected EM radiation from the environment onto the array. The EM radiation from the environment propagating along the ray towards the lens can be focused onto the detectors of the detector array by the lens.

[0022] The system and technique can include a phase calculation engine (e.g., a circuit). Each of the phase calculation engines can be connected to a corresponding detector of the detector array. For example, the phase calculation engine can calculate a phase difference value indicating the difference between the phase of the reference modulation signal and the phase of the modulated EM radiation received by the detector connected to the phase calculation engine.

[0023] In contrast to iToF depth cameras, the systems and techniques described herein can independently compare the phase difference values corresponding to each detector to determine whether a change in distance has occurred at the corresponding points in the environment corresponding to each detector. For example, the systems and techniques can include a difference engine (e.g., a difference circuit). Each of the difference engines can be connected to a corresponding phase calculation engine. Each difference engine can trigger an event in response to the difference between the current phase difference value and the previous phase difference value exceeding a threshold. The detector array, phase calculation engines, and difference engines can detect independently and asynchronously and, when a change is detected, can trigger an event in response to the detected change in distance. In some cases, the event can be output as a data stream.

[0024] As described above, a depth event camera can trigger events asynchronously for each detector, while an iToF depth camera operates on a per-frame basis. For example, an iToF depth camera can capture a depth-based representation of a scene, while a depth event camera can trigger events for each detector based on depth changes within the scene. For example, the systems and techniques can continuously emit EM radiation. When the depth of a point in the scene changes (e.g., based on something moving within the scene), the detector corresponding to that point can receive the EM radiation reflected from that point. The corresponding phase calculation engine can calculate a phase difference value different from the previous phase difference value for that point, and the corresponding difference engine can trigger an event based on the different phase difference value. In some cases, the event can be correlated with that detector and can be independent of other detectors in the detector array. In some examples, the event can be triggered when the difference engine detects the difference between the previous phase difference value and the latest phase difference value.

[0025] In this disclosure, the term “engine” may refer to a circuit, one or more circuits, logic, and / or one or more processors and / or operations performed by a circuit. In this disclosure, the term “circuit” may refer to an analog circuit comprising one or more analog circuits and / or one or more analog circuit elements (e.g., resistors, capacitors, inductors, diodes, transistors, and / or operational amplifiers), and / or a digital circuit comprising one or more digital circuits and / or circuit elements (e.g., transistors, logic gates, microcontrollers, and / or processors). In this disclosure, the term “modulate” may refer to amplitude modulation of a signal (e.g., a carrier signal) by a modulated signal, unless otherwise stated. Pulsing a signal may be an example of modulating a signal according to a square wave modulation frequency. In this disclosure, the terms “electromagnetic radiation,” “EM radiation,” and similar terms may refer to electromagnetic radiation of any frequency, including ultraviolet light, visible light, infrared light, and / or radio waves.

[0026] Various aspects of this application will be described with reference to the following figures.

[0027] Figure 1A is a diagram showing a system 100 including a device 102 capable of detecting changes in distance within an environment 116, according to various embodiments of the present disclosure. Figure 1B is a diagram including a perspective view of the environment 116 and a portion of the device 102 for detecting changes in distance within the environment 116, according to various embodiments of the present disclosure. For example, the device 102 may be, a portion thereof, or include a depth event camera.

[0028] As shown in Figure 1A, the apparatus 102 includes an electromagnetic (EM) emission emitter 104. As shown in Figure 1B, the EM emission emitter 104 may be configured to emit EM radiation 112 toward the environment 116. The EM emission emitter 104 can modulate the EM radiation 112 according to a reference modulation signal. For example, in some cases, the EM emission emitter 104 may modulate the EM radiation 112 using one or more square waves. In some cases, the EM emission emitter 104 may modulate the EM radiation 112 according to a polyphase modulation scheme (e.g., a four-phase modulation scheme). Additional details regarding polyphase modulation schemes are provided below.

[0029] As shown in Figure 1A, the apparatus 102 further includes one or more detectors 106. As shown in Figure 1B, one or more detectors 106 may be configured to receive EM radiation 114 from the environment. For example, EM radiation 112 may be reflected by objects in the environment 116, return, and be received and detected by the detectors 106. The detectors 106 may include a detector array 117 (e.g., an array of independent detectors). The detectors 106 may include a lens 120 for directing EM radiation 114 associated with a given ray from the environment 116 onto the corresponding detector of the detector array. For example, EM radiation 114 propagating along a ray 115 between the environment 116 and the lens 120 may be focused onto a detector 118a of the detector array 118. Thus, each detector in the detector array may correspond to a ray from the environment 116. For illustrative purposes only, Figure 1B is not drawn to scale. Furthermore, unlike what is shown in Figure 1B, the EM emission emitter 104 and the detector(s) 106 may be positioned close to each other.

[0030] As shown in Figure 1A, the apparatus 102 includes one or more phase calculation engines 108 (e.g., phase calculation circuits) configured to calculate a phase difference value representing the difference between the phase of the emitted EM radiation 112 and the phase of the reflected EM radiation 114. In some embodiments, the apparatus 102 may include a phase calculation engine (from the phase calculation engines 108) for each detector in the detector array. The phase calculation engines 108 may compare the phase of the envelope rather than the phase of the carrier signal. For example, an EM radiation emitter 104 may emit EM radiation 112 having a phase corresponding to the phase of a reference modulated signal (e.g., a square wave). A point in the environment 116 may reflect the EM radiation 114 back towards the detectors 106. The detectors 106 may receive the EM radiation 114. The phase of the reference modulation signal may be advanced between the time it is emitted from the EM emission emitter 104 and the time it is received by the detector(s) 106. However, the phase of the reflected EM emission 114 can be aligned with the phase of the reference modulation signal when the EM emission 112 was emitted. Therefore, the phase difference between the reference modulation signal and the EM emission 114 may indicate the time of flight of the EM emission 112 and EM emission 114 between emission and reception.

[0031] As shown in Figure 1A, the device 102 also includes one or more difference engines 110 (e.g., difference circuits) configured to trigger an event when the difference between the current phase difference value and the previous phase difference value exceeds a threshold. In some cases, the device 102 may include one difference engine of the difference engines 110 for each phase calculation engine of the phase calculation engines 108 (as previously stated, each phase calculation engine 108 may correspond to one detector of the detector array). The phase calculation engines 108 may determine the phase difference value when EM radiation 114 is received. In some cases, the phase calculation engines 108 may determine the phase difference value continuously. For example, an EM emission emitter 104 may continuously emit EM emission 112, and a detector(s) 106 may continuously detect EM emission 114 (for example, the detector's sensor may sample at a certain sampling rate). A phase calculation engine(s) 108 may continuously compare the phase of EM emission 112 (or the reference modulation signal) with the phase of EM emission 114 (for example, according to the sensor's sampling rate). A difference engine(s) 110 may continuously compare a previous phase difference value with the current phase difference value (for example, the most recently received phase difference value) (for example, as soon as the current phase difference value is received). For example, a difference engine(s) 110 may include a comparator that holds a previous phase difference value and can compare the previous phase difference value with the current phase difference value.

[0032] The differential engine(s) 110 may trigger an event whenever the difference between the current phase difference value and the previous phase difference value exceeds a threshold. In some cases, the phase calculation engine(s) 108 and the differential engine(s) 110 may operate asynchronously. In such cases, by operating the phase calculation engine(s) 108 and the differential engine(s) 110 asynchronously, the device 102 may trigger an event when a distance change is detected, rather than at a specific frame rate. In some cases, the differential engine(s) 110 may include one differential engine for each phase calculation engine(s) 108 and one phase calculation engine(s) 108 for each detector array of the detector, so that the device 102 may detect distance changes for each detector.

[0033] In some cases, the device 102 may output the event as a data stream (e.g., a variable bandwidth data stream). For example, the data stream may include the event (e.g., in the form of a tuple containing location and direction information). For example, the data stream may include x and y coordinates indicating the detector's location or the environment corresponding to the event. Furthermore, the data stream may include an indication of the direction corresponding to the event (e.g., an indication of what threshold was exceeded to trigger the event). For example, the data stream may include an indication of whether the event responds to a change based on something approaching the detector or a change due to something in the environment moving away from the detector. Furthermore, in some cases, the data stream may include a timestamp of the event or another indication of when the event occurred.

[0034] In other cases, the device 102 may generate a depth event map. The depth event map may include pixels corresponding to each detector. The depth event map may include a representation of events triggered by each detector; for example, an event at a detector may correspond to a pixel value, and the absence of an event at a detector may correspond to a different pixel value. The depth event map may show points in the environment where the depth measurement changed between the time the previous phase difference value was determined and the time the current phase difference value was determined.

[0035] Figure 2 is a diagram illustrating a device 200 capable of detecting changes in distance within an environment, according to various embodiments of the present disclosure. For example, the device 200 may be included in a depth event camera. The device 200 may be an example of a single detector, a single phase calculation engine (e.g., a phase calculation circuit), and a single difference engine (e.g., a difference circuit) of the device 102 in Figure 1A.

[0036] In some embodiments, the device 200 may include an EM emission emitter (not shown in Figure 2), such as the EM emission emitter 104 in Figures 1A and 1B. The device 200 may modulate the emitted EM radiation according to a reference modulation signal. For example, the EM emission emitter may multiply a carrier signal (e.g., having a carrier frequency) by the reference modulation signal. The reference modulation signal may define the envelope of the EM radiation. In some cases, the device 200 may emit a plurality of modulated EM radiation signals modulated in different phases (e.g., according to a polyphase modulation scheme). For example, the iToF depth pixel 202 may emit four EM radiation signals, including a first EM radiation signal, a second EM radiation signal delayed by 90° from the first EM radiation signal, a third EM radiation signal delayed by 180° from the first EM radiation signal, and a fourth EM radiation signal delayed by 270° from the first EM radiation signal. One or more of the reference modulation signals may be square waves.

[0037] The device 200 includes an iToF depth pixel 202. The iToF depth pixel 202 receives reflected EM radiation and may provide an indication of the phase of the EM radiation that has been received as reflected EM radiation to a phase calculation engine 204 (e.g., a phase calculation circuit). The phase may be the phase of the modulated signal (e.g., the envelope) and not the phase of the carrier signal. The iToF depth pixel 202 may include a detector (such as one of the detectors 106, one or more, in Figures 1A and 1B).

[0038] The device 200 includes a phase calculation engine 204. The phase calculation engine 204 can calculate a phase difference value based on the difference between the phase of the emitted EM radiation and the phase of the reflected EM radiation received by the iToF depth pixel 202. The phase may be the phase of the modulation signal (e.g., the envelope of the reference modulation signal and the envelope of the reflected EM radiation) and not the phase of the carrier signal. For example, the phase calculation engine 204 may receive an indication of the phase of the reflected EM radiation from the iToF depth pixel 202. The phase calculation engine 204 may receive an indication of the phase of the reference modulation signal 216 (e.g., the reference modulation signal 216 used to modulate the EM radiation emitted by the iToF depth pixel 202). The phase calculation engine 204 may compare the phases to determine a phase difference value 210.

[0039] The phase calculation engine 204 includes a differential amplifier 206. The differential amplifier 206 can amplify a voltage difference. The voltage difference may represent the difference between the phase of emitted EM radiation and the phase of reflected EM radiation. For example, the voltage difference may be the difference between two voltages. Each of the two voltages may represent the correlation between one of four control signals (e.g., a reference signal) and the reflected signal. Each of the four control signals may have a different phase. The differential amplifier 206 may be an analog circuit. The differential amplifier 206 may include an amplifier and one or more resistors configured so that the amplifier amplifies the difference between two input voltages.

[0040] The phase calculation engine 204 includes an arctangent engine 208. The arctangent engine 208 can calculate the phase angle based on the difference between the phase of the emitted EM radiation and the phase of the reflected EM radiation. The arctangent engine 208 may operate on the output of the differential amplifier 206 (e.g., an amplified signal showing the phase difference between the emitted EM radiation and the reflected EM radiation). The arctangent engine 208 may calculate the phase angle using a geometric function (e.g., arctangent or its equivalent). When determining the arctangent, the arctangent engine 208 may perform demodulation, multiplication, inversion, addition, squaring, integration, filtering, and / or buffering operations. The arctangent engine 208 may be an analog circuit.

[0041] The phase calculation engine 204 may generate a phase difference value 210. The phase difference value 210 may represent the phase difference between reflected EM radiation and emitted EM radiation. In one exemplary example, the current phase difference value 210 may be a phase angle.

[0042] The device 200 includes a difference engine 212 (e.g., a difference circuit). The difference engine 212 can compare the current phase difference value 210 with a previous phase difference value 210. For example, the difference engine 212 receives the current phase difference value 210. The difference engine 212 can compare the current phase difference value 210 with a previously received phase difference value 210. Depending on whether the difference between the current phase difference value 210 and the previously received phase difference value 210 exceeds a threshold, the difference engine 212 may generate an event 214. Furthermore, the difference engine 212 can store the current phase difference value 210 and compare the stored phase difference value 210 with a subsequently received phase difference value 210. For example, the current phase difference value may be a previously received difference value, which can be compared with a new current phase difference value. The difference engine 212 may be an analog circuit. For example, the differential engine 212 may include a capacitor (for example, for storing phase difference values ​​210), an inverter, and one or more comparators (for example, for comparing the difference between the phase difference values ​​210 with one or more thresholds).

[0043] In some cases, the difference engine 212 may compare the difference between phase difference values ​​210 with several thresholds and trigger different events 214 based on which threshold is exceeded. For example, the difference engine 212 may subtract the current phase difference value 210 from the previous phase difference value 210 and compare the difference with a larger positive threshold (indicating that the previous phase difference value 210 was much larger than the current phase difference value 210), a smaller positive threshold (indicating that the previous phase difference value 210 was larger than the current phase difference value 210), a smaller negative threshold (indicating that the current phase difference value 210 was larger than the previous phase difference value 210), and a larger negative threshold (indicating that the current phase difference value 210 was much larger than the previous phase difference value 210). If the difference exceeds a larger positive threshold, a larger positive event may be generated; if the difference exceeds a smaller positive threshold, a smaller positive event may be generated; if the difference exceeds a smaller negative threshold, a smaller negative event may be generated; and if the difference exceeds a larger negative threshold, a larger negative event may be generated.

[0044] Figure 3 includes a depth event representation 300 of a scene according to various aspects of the present disclosure. The depth event representation 300 represents a depth event in the scene (e.g., a change in depth over a certain duration). The depth event representation 300 is based on detectors that detected the change in depth over that duration. For example, the depth event representation 300 includes one pixel corresponding to each detector in a detector array. For each detector that detected an increase in depth over a certain duration, the depth event representation 300 includes a black pixel; for each detector that detected a decrease in depth over a certain duration, the depth event representation 300 includes a white pixel; and for each detector that did not detect a change in depth over a certain duration, the depth event representation 300 includes a gray pixel. The scene includes a person moving towards the depth event sensor and is therefore mostly white. In the depth event representation 300, the floor is noisy (e.g., including white, black, and gray pixels), which indicates depth uncertainty due to infrared (IR) absorption by the carpet. For example, a carpet may randomly absorb or scatter IR pulses, and these IR pulses may be detected as random depth events. Depth event representation 300 may be an example of a depth event map.

[0045] The system and technique may generate depth events asynchronously (for example, the system and technique may generate an event whenever the depth changes). The system and technique may output depth events as a data stream rather than according to duration. Since events may be generated asynchronously, the duration may be a portion of the frame rate of other sensors (e.g., a camera and / or an iToF depth camera). The depth event representation 300 may be a representation of events generated over a certain duration (e.g., a duration specifically selected to generate the depth event representation 300). In some cases, the system and technique may not generate a depth event representation, but rather output the depth events in a data stream. The depth event representation 300 may be generated by another system that can aggregate events generated asynchronously over a certain duration. Alternatively, in some cases, the system and technique may aggregate depth events over a duration and generate and / or output a depth event representation.

[0046] In some cases, the system and technique may aggregate depth measurements over multiple discrete durations. In such cases, the depth event representation may include pixel values ​​representing multiple changes in the same direction over multiple discrete durations. For example, for a detector that detected movement away from the detector in two of the two durations, the depth event representation may include black pixels; for a detector that detected movement away from the detector in one of the two durations, the depth event representation may include dark gray pixels; for a detector that did not detect movement away from the detector in either of the two durations, the depth event representation may include gray pixels; for a detector that detected movement towards the detector in one of the two durations, the depth event representation may include light gray pixels; and for a detector that detected movement towards the detector in two of the two durations, the depth event representation may include white pixels.

[0047] Some iToF depth representations exhibit motion blur. For example, an iToF depth camera may determine depth information that includes the depth of a point in a scene (e.g., the distance between the iToF depth camera and the point in the scene). For example, an iToF depth camera may generate depth information at a certain frame rate (e.g., 30 frames per second (fps)). Such an iToF depth camera may integrate iToF depth information over time (e.g., according to the frame rate) to generate depth information. For example, such an iToF depth camera may receive reflected EM radiation over a duration (e.g., over 1 / 30 of a second) and integrate all the phase difference values ​​of all reflected EM radiation received over that duration to generate a single depth value for that duration. However, objects in the scene may move during that duration. Motion can cause motion blur, which can result in inaccurate depth information based on the fact that a point in the scene has more than one depth during that duration.

[0048] The system and techniques can be used to remove blur from depth information. For example, the system and techniques can be used with or in conjunction with an iToF depth camera. Depth change information generated by the system and techniques can be used to correct motion blur in iToF depth information. For example, the system and techniques can capture depth changes at a fine-grained temporal level (and / or asynchronously). Depth change-based information can be used to resolve ambiguity in iToF depth information. For example, to correct motion blur, the system and techniques can sum positive and negative depth events for a single pixel over a period of time. For example, the pixel may have approached an amount exceeding a threshold N times during that period. In this case, the system and techniques do not ignore phase difference values. Instead, the system and techniques sample each of the many phase difference values ​​with a very short integration time and sum their contributions. In contrast, conventional iToF techniques can average frames over time (e.g., over long integration times, such as the Q1-Q4 phases in a four-phase modulation scheme), potentially outputting inaccurate phase difference values.

[0049] Figure 4 includes an image 400 of an object 402 located on a mirrored surface 404 next to a transparent surface 406. It may be difficult for the iToF depth camera to accurately determine the depth of points on the transparent surface and points on the mirrored surface. For example, a mirrored or transparent surface may generate multiple reflections (e.g., multipath reflections) that can arrive at different times (e.g., based on moving along different paths). Such reflections may be difficult for the iToF depth camera to distinguish. Systems and techniques may be used to generate confidence values ​​related to depth information. For example, systems and techniques may be used with or in parallel with an iToF depth camera to generate confidence values ​​corresponding to depth information generated by the iToF depth camera.

[0050] For example, depth change information can be determined based on the number of depth events using a depth event camera. Figure 4 includes a representation 410 of depth events generated over a period of time. Transparent surfaces 406 and specular surfaces 404 may generate a higher proportion of depth changes than objects 402. For example, transparent surfaces 406 and specular surfaces 404 may cause the system and technique to generate depth events more frequently. Therefore, in representation 410, objects 402 may appear gray (indicating no depth events during the duration), while transparent surfaces 406 and specular surfaces 404 are represented in white and black (representing depth events indicating movement towards and away from the depth event camera). As an example, a depth event camera may generate two events (indicating depth changes) per iToF depth camera frame. Based on the depth change information, the system and technique may determine confidence values ​​for each detector (and / or for each pixel of iToF depth camera depth information corresponding to the pixel). For example, Figure 4 includes a confidence value representation 420 corresponding to image 400. In representation 420, object 402 appears as white, indicating high confidence. Transparent surface 406 and specular surface 404 appear as gray and black, indicating low confidence.

[0051] Figure 5 includes an image 500 of surface 502 and its edge 504. It can be difficult for an iToF depth camera to accurately determine the depth of a point at an object's edge. For example, based on how emitted EM radiation is reflected, the phase difference value from a single point at an object's edge can change between a first time and a second time, even if the distance remains constant. For instance, the iToF depth camera's detector may receive reflections from both foreground and background surfaces. Systems and techniques can be used to generate confidence values ​​related to depth information. For example, systems and techniques can be used with or in conjunction with an iToF depth camera to generate confidence values ​​corresponding to the depth information generated by the iToF depth camera.

[0052] For example, depth change information can be determined using a depth event camera. Figure 5 includes a representation 510 of depth events generated over a period of time. Points at the edge 504 of surface 502 may frequently generate depth changes (for example, the edge 504 of surface 502 may cause the system and technique to frequently generate depth events). As an example, the depth event camera may generate two events (indicating depth changes) at the point of edge 504 for each iToF depth camera frame. Based on the depth change information, the system and technique may determine confidence values ​​for each detector (and / or for each pixel of the iToF depth camera depth information corresponding to the pixel). For example, Figure 5 includes a representation 520 of confidence values ​​corresponding to image 500. In representation 520, surface 502 is shown as white, indicating high confidence. Edge 504 appears as gray and black, indicating low confidence.

[0053] Figure 6 shows a vehicle 602 including a depth event camera 606 according to various aspects of the present disclosure. For example, the vehicle 602 may include a depth event camera 606 that can be used to determine when a change in depth has occurred between the vehicle 602 and various points in the vehicle 602's environment. There may be various applications for determining changes in depth within the environment of a vehicle, robot, drone, or automated machine. The vehicle 602 is given as an example. For example, the change in depth may be used to determine the time until a collision between the vehicle 602 and an object in the environment (e.g., vehicle 604). As another example, the change in depth may be used to determine how fast an object in the environment (e.g., vehicle 604) is moving (or whether it is moving). Determining depth changes using depth event cameras (e.g., device 102 in Figure 1, device 200 in Figure 2, and / or depth event camera 606 in Figure 6) may be superior to other techniques for determining depth and / or depth changes, at least because depth event camera 606 may be able to determine depth changes asynchronously and / or more quickly than other techniques. For example, light detection and ranging (LIDAR) techniques may be used to determine the depth to a point in the environment. A LIDAR scan takes more time than a depth event camera to scan the environment and detect and report changes in the environment 606. In addition, depth events (such as those generated by a depth event camera) may be able to represent changes in the environment with higher spatial and / or temporal resolution than LIDAR-based depth information. Furthermore, depth events can be communicated or stored using fewer bits of data despite representing higher spatial and / or temporal resolution than LIDAR-based depth information. For example, depth events may asynchronously represent all depth changes within a large field of view of a depth camera in a data stream containing tuples. In contrast, LIDAR-based depth information may include values ​​for each of multiple pixels. As another example, depth changes may be determined by comparing the size of objects in the environment in a sequence of images captured by the camera.A depth event camera (e.g., device 102 in Figure 1, device 200 in Figure 2, and / or depth event camera 606 in Figure 6) 606 may be able to determine depth changes at a speed faster than the image comparison technique can determine depth changes.

[0054] Figure 7 is a flowchart representing a process 700 for determining a change in distance, according to an aspect of this disclosure. One or more processes 700 may be performed by a computing device (or apparatus) or a component of a computing device (e.g., a chipset, a codec, etc.). A computing device may be a vehicle or a component or system of a vehicle, a mobile device (e.g., a mobile phone) or a component or system of a mobile device, a network-connected wearable such as a watch or a component or system of a wearable, an extended reality (XR) device such as a virtual reality (VR) device or an augmented reality (AR) device or a component or system of an XR device, a desktop computing device or a component or system of a desktop computing device, a tablet computing device or a component or system of a tablet computing device, a server computer or a component or system of a server computer, a robotic device or a component or system of a robotic device, and / or any other computing device having the resource capacity to perform process 700. One or more operations of process 700 may be implemented as software components running on and operating on one or more processors.

[0055] In block 702, a computing device (or one or more components thereof) may cause at least one electromagnetic (EM) emission emitter to emit a first EM emission toward the environment. For example, the EM emission emitter 104 of device 102 may emit EM emission 112 toward the environment 116.

[0056] In block 704, a computing device (or one or more of its components) may receive first EM radiation reflected from the environment. For example, one or more detectors 106 of device 102 may receive EM radiation 114 (e.g., reflected first EM radiation) reflected from the environment 116.

[0057] In block 706, a computing device (or one or more components thereof) can calculate a first phase difference value that represents the difference between the phase of the emitted first EM radiation and the phase of the reflected first EM radiation. For example, the phase calculation engine 108 of device 102 can calculate a phase difference value (e.g., a first phase difference value) that represents the difference between the phase of the EM radiation 112 and the phase of the EM radiation 114. As another example, the phase calculation engine 204 in Figure 2 can calculate the current phase difference value 210.

[0058] In block 708, a computing device (or one or more components thereof) may cause at least one EM emission emitter to emit a second EM emission into the environment. For example, the EM emission emitter 104 of device 102 may emit EM emission 112 into the environment 116.

[0059] In block 710, a computing device (or one or more of its components) may receive second EM radiation reflected from the environment. For example, one or more detectors 106 of device 102 may receive EM radiation 114 (e.g., reflected second EM radiation) reflected from the environment 116.

[0060] In block 712, a computing device (or one or more of its components) may calculate a second phase difference value that represents the difference between the phase of the emitted second EM radiation and the phase of the reflected second EM radiation. For example, the phase calculation engine 108 of device 102 may calculate a phase difference value (e.g., a second phase difference value) that represents the difference between the phase of EM radiation 112 and the phase of EM radiation 114. As another example, the phase calculation engine 204 in Figure 2 may calculate the current phase difference value 210.

[0061] In block 714, a computing device (or one or more of its components) may trigger an event when the difference between a second phase difference value and a first phase difference value exceeds a threshold. For example, the differential engine(s) 110 of device 102 may trigger an event when the difference between a second phase difference value (calculated, e.g., in block 712) and a first phase difference value (calculated, e.g., in block 706) exceeds a threshold. As another example, the differential engine 212 in Figure 2 may trigger event 214 when the difference between a second phase difference value 210 (calculated, e.g., in block 712) and a first phase difference value 210 (calculated, e.g., in block 706) exceeds a threshold.

[0062] In some embodiments, receiving the reflected first EM radiation may be the reception of the reflected first EM radiation in a detector of a detector array. The detector may correspond to a portion of the reflected EM radiation. Furthermore, receiving the reflected second EM radiation may be the reception of the reflected second EM radiation in a detector of a detector array. Furthermore, the event may correspond to a detector of a detector array. For example, the reflected first and second radiations may propagate along ray 115 and both may be received by detector 118a. Detector 118a may be part of iToF depth pixel 202. Depending on the reflected first and second radiations, the device 200 may generate event 214.

[0063] In some embodiments, a detector of a detector array may be a first detector of the detector array, and an event may be a first event. A computing device (or one or more components thereof) may receive a third EM emission reflected from the environment at a second detector of the detector array, calculate a third phase difference value indicating the difference between the phase of the emitted first EM emission and the phase of the reflected third EM emission, receive a fourth EM emission reflected from the environment at a second detector of the detector array, calculate a fourth phase difference value indicating the difference between the phase of the emitted second EM emission and the phase of the reflected fourth EM emission, and trigger a second event if the difference between the fourth phase difference value and the third phase difference value exceeds a threshold. The second event may correspond to a second detector of the detector array. For example, a detector 106 of device 102 may receive EM emission 114 (e.g., reflected third emission) reflected from the environment 116. The reflected third radiation may be a reflected version of the emitted first EM radiation. The reflected third radiation may be received by a detector in a detector array separate from the detector that received the reflected first and second radiations. The phase calculation engine 108 of the device 102 may calculate a phase difference value that indicates the difference between the phase of the EM radiation 112 and the phase of the EM radiation 114 (e.g., a third phase difference value). The phase difference value between the reflected third EM radiation and the emitted first radiation may be calculated by a phase calculation engine separate from the one used to calculate the phase difference value between the reflected first EM radiation and the emitted first EM radiation. The detector 106 of the device 102 may receive EM radiation 114 reflected from the environment 116 (e.g., reflected fourth radiation). The reflected fourth radiation may be a reflected version of the emitted second EM radiation. The reflected fourth radiation may be received by a detector in a detector array that received the reflected third radiation. The phase calculation engine(s) 108 of the device 102 may calculate a phase difference value that represents the difference between the phase of the EM radiation 112 and the phase of the EM radiation 114 (e.g., a fourth phase difference value).The phase difference between the reflected fourth EM radiation and the emitted second radiation can be calculated by the same phase calculation engine used to calculate the phase difference between the reflected third EM radiation and the emitted first EM radiation. The differential engine(s) 110 of the device 102 may trigger an event (e.g., a second event) when the difference between the fourth phase difference value and the third phase difference value exceeds a threshold. The second event may be triggered by a different differential engine than the one used to trigger the first event. As another example, the phase calculation engine 204 in Figure 2 may calculate the current phase difference value 210 (e.g., the third phase difference value). The phase calculation engine 204 in Figure 2 may calculate the current phase difference value 210 (e.g., the fourth phase difference value). The differential engine 212 in Figure 2 may trigger an event 214 (e.g., a second event) when the difference between the fourth phase difference value 210 and the third phase difference value 210 exceeds a threshold.

[0064] In some embodiments, a first phase difference value may be determined by a first phase calculation engine (e.g., a first analog differential amplifier and a first analog inc-tangent calculation circuit). A second phase difference value may be determined by a first phase calculation engine (e.g., an analog differential amplifier and a first analog inc-tangent calculation circuit). A third phase difference value may be determined by a second phase calculation engine (e.g., a second analog differential amplifier and a second analog inc-tangent calculation circuit). A fourth phase difference value may be determined by a second phase calculation engine (e.g., a second analog differential amplifier and a second analog inc-tangent calculation circuit). For example, reflected first and second radiation may propagate along the ray 115 and be received by a detector 118a of the detector array 117. Detector 118a may be part of an iToF depth pixel 202, which may be coupled with a phase calculation engine 204 and a difference engine 212 to generate an event 214. The reflected third and fourth radiations may propagate along another ray and be received by another detector in the detector array 117. The other detector may be part of another iToF depth pixel 202, which may be coupled with another phase calculation engine 204 and another difference engine 212 to generate another event 214.

[0065] In some embodiments, the first phase difference value may be calculated asynchronously, and / or the second phase difference value may be calculated asynchronously. For example, the phase calculation engine(s) 108 may operate asynchronously. As another example, the phase calculation engine 204 may operate asynchronously. In some embodiments, the first phase difference value may be determined by an analog phase calculation engine (e.g., an analog differential amplifier and an analog inverse tangent calculation circuit), and the second phase difference value may be determined by an analog phase calculation engine (e.g., an analog differential amplifier and an analog inverse tangent calculation circuit). For example, the phase calculation engine 108 may be or include one or more analog circuits. As another example, the phase calculation engine 204 may be or include one or more analog circuits.

[0066] In some embodiments, a computing device (or one or more components thereof) may asynchronously determine whether the difference between a second phase difference value and a first phase difference value exceeds a threshold. For example, a differential engine(s) 110 may operate asynchronously. As another example, a differential engine 212 may operate asynchronously. In some embodiments, whether the difference between a second phase difference value and a first phase difference value exceeds a threshold may be determined by an analog differential circuit. For example, a differential engine(s) 110 may be or include one or more analog circuits. As another example, a differential engine 212 may be or include one or more analog circuits.

[0067] In some embodiments, whether the difference between a second phase difference value and a first phase difference value exceeds a threshold can be determined by a first difference engine (e.g., a first analog difference circuit). Whether the difference between a fourth phase difference value and a third phase difference value exceeds a threshold can be determined by a second difference engine (e.g., a second analog circuit). For example, reflected first and second radiation may propagate along ray 115 and be received by detector 118a of detector array 117. Detector 118a may be part of iToF depth pixel 202, which can be coupled with phase calculation engine 204 and difference engine 212 to generate event 214. Reflected third and fourth radiation may propagate along another ray and be received by another detector of detector array 117. The other detector may be part of another iToF depth pixel 202, which can be coupled with another phase calculation engine 204 and another difference engine 212 to generate another event 214.

[0068] In some embodiments, a computing device (or one or more of its components) may generate and / or output a map showing a first event and a second event. The map may show points in the environment where the depth measurement changed and / or the direction in which the depth measurement changed. For example, device 102 may generate a map showing the event. In some embodiments, a computing device (or one or more of its components) may generate and / or output a data stream showing the first event and the second event. The data stream may show points in the environment where the depth measurement changed and / or the direction in which the depth measurement changed. For example, device 102 may generate a data stream showing the event.

[0069] In some embodiments, a computing device (or one or more components thereof) may modulate at least one emitter to emit a first EM emission according to a reference modulation signal and to modulate an emitted second EM emission according to the reference modulation signal. Calculating a first phase difference value may be based on the difference between the phase of the reference modulation signal and the phase of the envelope of the reflected first EM emission. Calculating a second phase difference value may be based on the difference between the phase of the reference modulation signal and the phase of the envelope of the reflected second EM emission. For example, device 102 may modulate an EM emission emitter 104 to emit an EM emission 112, and a phase difference engine(s) 108 may determine a phase difference value based on the envelope of the EM emission 114 (e.g., not based on the carrier frequency). In some embodiments, the modulation signal may be a square wave (e.g., so that the EM emission 112 is pulsed).

[0070] In some examples, as described above, the processes described herein (e.g., process 700 in Figure 7, and / or other methods described herein) can be implemented entirely or partially by a computing device or apparatus. In one example, one or more of the methods can be implemented by apparatus 102 in Figure 1A, apparatus 200 in Figure 2, depth event camera 606 in Figure 6, or by another system or device. In another example, one or more of the methods (e.g., process 700 in Figure 7, and / or other methods described herein) can be implemented entirely or partially by the computing device architecture 800 shown in Figure 8. For example, a computing device having the computing device architecture 800 shown in Figure 8 may include or be included in components of apparatus 102, apparatus 200, and / or depth event camera 606, and may implement the operation of process 700 and / or other processes described herein. In some cases, a computing device or apparatus may include a variety of components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other components (one or more) configured to perform the steps of the process described herein. In some examples, a computing device may include a display, a network interface configured to communicate and / or receive data, any combination thereof, and / or other components (one or more). The network interface may be configured to communicate and / or receive Internet Protocol (IP) based data or other types of data.

[0071] Components of a computing device can be implemented in a circuit configuration. For example, components may include and / or be implemented using one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), electronic circuits, or other electronic hardware, and / or may include and / or be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0072] Process 700 and / or other processes described herein are illustrated as logical flow diagrams, and their operations represent sequences of operations that can be implemented by hardware, computer instructions, or a combination thereof. In the context of computer instructions, an operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the operation described. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or realize a particular data type. The order in which the operations are described is not intended to be interpreted as limiting, and any number of the described operations can be combined in any order and / or in parallel to implement a process.

[0073] Furthermore, process 700 and / or other processes described herein may be carried out under the control of one or more computer systems comprising executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that are executed collectively on one or more processors by hardware or a combination of hardware. As described above, the code may be stored in a computer-readable or machine-readable storage medium in the form of a computer program containing multiple instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-temporary.

[0074] Figure 8 shows an exemplary computing device architecture 800 of an exemplary computing device that can implement various techniques described herein. Some examples include a vehicle (or a computing device in a vehicle), a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, or other device. For example, the computing device architecture 800 may include, implement, or include any or all of the device 102 in Figure 1A, the device 200 in Figure 2, and the depth event camera 606 in Figure 6.

[0075] The components of the computing device architecture 800 are shown as communicating electrically with one another using connections 812 such as buses. An exemplary computing device architecture 800 includes a processing unit (CPU or processor) 802 and computing device connections 812 that connect various computing device components to the processor 802, including computing device memories 810 such as read-only memory (ROM) 808 and random-access memory (RAM) 806.

[0076] The computing device architecture 800 may include a high-speed memory cache that is directly connected to the processor 802, adjacent to the processor 802, or integrated as part of the processor 802. The computing device architecture 800 may copy data from memory 810 and / or storage device 814 to cache 804 for rapid access by the processor 802. In this way, the cache can provide a performance improvement that avoids delays for the processor 802 while waiting for data. These and other modules can control the processor 802 to perform various actions, or can be configured to control the processor 802. Other computing device memories 810 may also be available for use. Memory 810 may include multiple different types of memory with different performance characteristics. The processor 802 may include an arbitrary general-purpose processor, hardware or software services such as service 1 816, service 2 818, and service 3 820 stored in storage device 814 and configured to control the processor 802, and a dedicated processor with software instructions incorporated into the processor design. Processor 802 may be a self-contained system including multiple cores or processors, a bus, a memory controller, a cache, etc. A multi-core processor may be symmetrical or asymmetrical.

[0077] To enable user interaction with the computing device architecture 800, the input device 822 can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, or speech. The output device 824 may also be one or more of several output mechanisms known to those skilled in the art, such as a display, projector, television, or speaker device. In some cases, a multimodal computing device may allow the user to provide multiple types of input to communicate with the computing device architecture 800. The communication interface 826 can generally control and manage user input and computing device output. There are no restrictions on operating on any particular hardware configuration, and therefore, as improved hardware or firmware configurations are developed, the basic functions here can be easily replaced with them.

[0078] The storage device 814 is a non-volatile memory and may be a hard disk or other type of computer-readable medium capable of storing computer-accessible data, such as a magnetic cassette, flash memory card, solid-state memory device, digital versatile disk, cartridge, random access memory (RAMs) 806, read-only memory (ROM) 808, and hybrids thereof. The storage device 814 may include services 816, 818, and 820 for controlling the processor 802. Other hardware or software modules are intended. The storage device 814 may be connected to the computing device connection unit 812. In one embodiment, a hardware module performing a particular function may include software components stored on a computer-readable medium in relation to necessary hardware components such as the processor 802, the connection unit 812, and the output device 824 in order to perform that function.

[0079] The term “substantially” with respect to a given parameter, characteristic, or condition may mean that a person skilled in the art would understand that the given parameter, characteristic, or condition is met with a small degree of variation, such as within acceptable manufacturing tolerances. For example, depending on the specific parameter, characteristic, or condition that is substantially met, the parameter, characteristic, or condition may be met at least 90%, at least 95%, or even at least 99%.

[0080] Aspects of this disclosure are applicable to any suitable electronic device (such as a security system, smartphone, tablet, laptop computer, vehicle, drone, or other device) that includes or is coupled to one or more active depth sensing systems. While the following describes devices having or coupled to one optical projector, aspects of this disclosure are applicable to devices having any number of optical projectors and are therefore not limited to any particular device.

[0081] The term “device” is not limited to one or a specific number of physical objects (such as one smartphone, one controller, or one processing system). As used herein, a device may be any electronic device having one or more parts that can implement at least some parts of this disclosure. The following descriptions and examples use the term “device” to describe various aspects of this disclosure, but the term “device” is not limited to a specific configuration, type, or number of objects. Additionally, the term “system” is not limited to multiple components or a specific configuration. For example, a system may be mounted on one or more printed circuit boards or other boards and may have movable or static components. The following descriptions and examples use the term “system” to describe various aspects of this disclosure, but the term “system” is not limited to a specific configuration, type, or number of objects.

[0082] Specific details are provided in the above specification to provide a complete understanding of the embodiments and examples provided herein. However, those skilled in the art will understand that the embodiments can be put into practice without these specific details. For the sake of clarity, in some cases the technology may be presented as including individual functional blocks, including devices, device components, steps or routines in the way they are embodied in software, or combinations of hardware and software. Additional components other than those shown in the figures and / or described herein may also be used. For example, circuits, systems, networks, processes, and other components may be shown as components in the form of block diagrams, so as not to obscure the embodiments with unnecessary details. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details, so as not to obscure the embodiments.

[0083] Individual embodiments may be described above as processes or methods represented as flowcharts, flow diagrams, data flow diagrams, structural diagrams, or block diagrams. While flowcharts can describe operations as sequential processes, many of these operations can also be performed in parallel or simultaneously. Furthermore, the order of operations can be rearranged. A process terminates when its operations are complete, but it may have additional steps not included in the diagram. A process can correspond to a method, function, procedure, subroutine, subprogram, etc. If a process corresponds to a function, its termination may correspond to the function returning to the calling function or main function.

[0084] The processes and methods described in the above examples can be implemented using computer-executable instructions stored in or available from computer-readable media. Such instructions may include instructions and data that cause, for example, a general-purpose computer, a dedicated computer, or a processing device to perform a certain function or group of functions, or otherwise configure them to perform them. The portion of computer resources used may be accessible over a network. Computer-executable instructions may be, for example, binary, or intermediate format instructions such as assembly language, firmware, or source code.

[0085] The term “computer-readable media” includes, but is not limited to, portable or nonportable storage devices, optical storage devices, and various other media capable of storing, containing, or transporting instructions (one or more) and / or data. Computer-readable media may include non-temporary media capable of storing data and that do not contain carrier waves and / or transient electronic signals propagating wirelessly or via wired connections. Examples of non-temporary media include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, magnetic or optical disks, USB devices with non-volatile memory, network storage devices, and any suitable combination thereof. Code and / or machine-executable instructions may be stored on computer-readable media, which may represent procedures, functions, subprograms, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. A code segment can be coupled to another code segment or hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory content. Information, arguments, parameters, data, etc., can be passed, transferred, or transmitted via any preferred means, including memory sharing, message passing, token passing, or network transmission.

[0086] In some embodiments, computer-readable storage devices, media, and memory may include cables or wireless signals, such as bitstreams. However, non-transient computer-readable storage media, as referred to, expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0087] Devices implementing processes and methods in accordance with these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take on any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segment (e.g., a computer program product) that performs the required tasks may be stored in computer-readable or machine-readable media. One or more processors may perform the required tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices, or other small form factor personal computers, digital assistants, rack-mount devices, and standalone devices. The functionalities described herein may also be embodied in peripherals or add-in cards. Such functionalities may also, as a further example, be implemented between different chips on a circuit board or between different processes running within a single device.

[0088] Instructions, a medium for transmitting such instructions, computing resources for executing those instructions, and other structures supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0089] While the above description refers to specific embodiments of this application, those skilled in the art will recognize that this application is not limited thereto. Therefore, although illustrative embodiments of this application are described in detail herein, it should be understood that the concepts of the present invention can be embodied and employed in various other ways, and that, unless limited by the prior art, the appended claims are intended to be interpreted as including such variations. The various features and embodiments of this application described above may be used individually or in combination. Furthermore, embodiments can be used in any number of environments and applications other than those described herein without departing from the broader spirit and scope of this specification. Therefore, this specification and the drawings should be considered illustrative and not limiting. For illustrative purposes, the methods are described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in an order different from the order described.

[0090] Those skilled in the art will understand that the symbols or terms less than ("<") and greater than (">") used herein may be replaced with the symbols less than or equal to ("≦") and greater than or equal to ("≧"), respectively, without departing from the scope of this specification.

[0091] Where a component is described as "configured to perform a particular operation," such configuration can be achieved, for example, by designing an electronic circuit or other hardware to perform that operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuit) to perform that operation, or by any combination thereof.

[0092] The phrase "connected to" refers to any component that is physically connected to another component, either directly or indirectly, and / or communicates with another component, either directly or indirectly (for example, connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0093] The claim language or other language that states "at least one of a set" and / or "one or more of a set" indicates that one member of that set, or multiple members of that set (in any combination), satisfy the claim. For example, the claim language that states "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, the claim language that states "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language that states "at least one of a set" and / or "one or more of a set" does not limit the set to the items listed within it. For example, the wording of a claim that states "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0094] Claim language that includes phrases such as "at least one processor configured to perform X, Y, and Z" or other similar phrases indicates that one processor or multiple processors (in any combination) can perform the associated operations (one or more). For example, claim language that includes "at least one processor configured to perform X, Y, and Z" means that a single processor can perform operations X, Y, and Z, or that multiple processors can each task a subset of operations X, Y, and Z, or that a group of multiple processors can work together to perform operations X, Y, and Z, so that multiple processors perform X, Y, and Z together. In another example, claim language that includes "at least one processor configured to perform X, Y, and Z" may mean that any single processor can perform only at least one subset of operations X, Y, and Z.

[0095] The various illustrative logic blocks, modules, circuits, and algorithmic steps described in relation to the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly demonstrate this hardware-software compatibility, various exemplary components, blocks, modules, circuits, and steps have been outlined above in relation to their functions. Whether such functions are implemented as hardware or executed as software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionalities in various ways for each specific application, but such implementation decisions should not be construed as causing a departure from the scope of this application.

[0096] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device handsets, or integrated circuit devices having multiple applications, including applications in wireless communication device handsets and other devices. Any feature described as a module or component may be implemented together in an integrated logic device, or separately as individual but interoperable logic devices. When implemented in software, the techniques may be at least partially implemented by a computer-readable data storage medium containing program code that, when executed, includes instructions to perform one or more of the methods described above. The computer-readable data storage medium may also form part of a computer program product, which may include packaging materials. Computer-readable media may include random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, and magnetic or optical data storage media. These techniques may be implemented, at least in part, by computer-readable communication media, such as propagating signals or waves, that carry or communicate program code in the form of instructions or data structures that can be accessed, read, and / or executed by a computer.

[0097] The program code may be executed by a processor, which may include one or more processors such as digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor may be configured to implement any of the techniques described herein. The general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, as used herein, the term “processor” may refer to any of the structures described herein, any combination thereof, or any other structure or device suitable for implementing the techniques described herein.

[0098] Claim language that includes phrases such as "at least one processor configured to perform X, Y, and Z" or other similar phrases indicates that one processor or multiple processors (in any combination) can perform the associated operations (one or more). For example, claim language that includes "at least one processor configured to perform X, Y, and Z" means that a single processor can perform operations X, Y, and Z, or that multiple processors can each task a subset of operations X, Y, and Z, or that a group of multiple processors can work together to perform operations X, Y, and Z, so that multiple processors perform X, Y, and Z together. In another example, claim language that includes "at least one processor configured to perform X, Y, and Z" may mean that any single processor can perform only at least one subset of operations X, Y, and Z.

[0099] Exemplary embodiments of this disclosure include the following:

[0100] Embodiment 1. An apparatus for determining a change in distance, comprising: an EM emission emitter configured to emit electromagnetic (EM) radiation toward the environment; a detector configured to receive reflected EM radiation from the environment; a phase calculation circuit configured to calculate a phase difference value indicating the difference between the phase of emitted EM radiation and the phase of reflected EM radiation; and a difference circuit configured to trigger an event when the difference between the current phase difference value and the previous phase difference value exceeds a threshold.

[0101] Embodiment 2. The apparatus according to Embodiment 1, wherein the detector comprises a detector array, and each detector in the detector array corresponds to a corresponding portion of the reflected EM radiation.

[0102] Embodiment 3. The apparatus according to Embodiment 2, further comprising a plurality of phase calculation circuits, each of which is connected to a corresponding detector of a detector array, and each of which is configured to calculate a corresponding phase difference value indicating a corresponding difference between the phase of emitted EM radiation and the corresponding phase of reflected EM radiation received by the corresponding detector.

[0103] Embodiment 4. The apparatus according to Embodiment 3, further comprising a plurality of difference circuits, each of which difference circuits is connected to a corresponding phase calculation circuit of a plurality of phase calculation circuits, and each difference circuit of the plurality of difference circuits is configured to trigger an event when the difference between the current phase difference value and the previous phase difference value exceeds a threshold.

[0104] Embodiment 5. The apparatus according to Embodiment 4, further comprising at least one processor configured to output a map showing events triggered by corresponding difference circuits of a plurality of difference circuits.

[0105] Embodiment 6. The apparatus according to Embodiment 5, wherein the map shows points in the environment where the depth measurement value changed between the time the previous phase difference value was determined and the time the current phase difference value was determined.

[0106] Embodiment 7. The apparatus according to any one of Embodiments 4 to 6, wherein multiple phase calculation circuits operate asynchronously and multiple difference circuits operate asynchronously.

[0107] Embodiment 8. The apparatus according to any one of Embodiments 4 to 7, wherein multiple difference circuits are configured to output data streams representing events.

[0108] Embodiment 9. The apparatus according to Embodiment 8, wherein the data stream includes location information and direction information.

[0109] Embodiment 10. The apparatus according to any one of Embodiments 1 to 9, wherein the differential circuit is configured to output a data stream representing an event.

[0110] Embodiment 11. The apparatus according to Embodiment 10, wherein the data stream includes location information and direction information.

[0111] Embodiment 12. The apparatus according to any one of Embodiments 1 to 11, wherein the EM emission emitter is further configured to modulate emitted EM radiation based on a reference modulation signal, and the phase difference value indicates the difference between the phase of the reference modulation signal and the phase of the envelope of reflected EM radiation.

[0112] Embodiment 13. The apparatus according to Embodiment 12, wherein the reference modulated signal is a square wave.

[0113] Embodiment 14. The apparatus according to any one of Embodiments 1 to 13, wherein the phase calculation circuit comprises an analog differential amplifier and an analog inverse tangent calculation circuit.

[0114] Embodiment 15. The apparatus according to any one of Embodiments 1 to 14, wherein the difference circuit comprises an analog difference circuit.

[0115] Embodiment 16. A method for determining a change in distance, comprising: emitting a first electromagnetic (EM) radiation toward an environment; receiving the first EM radiation reflected from the environment; calculating a first phase difference value indicating the difference between the phase of the emitted first EM radiation and the phase of the reflected first EM radiation; emitting a second EM radiation toward an environment; receiving the second EM radiation reflected from the environment; calculating a second phase difference value indicating the difference between the phase of the emitted second EM radiation and the phase of the reflected second EM radiation; and triggering an event in response to the difference between the second phase difference value and the first phase difference value exceeding a threshold.

[0116] Embodiment 17. The method according to Embodiment 16, wherein receiving a reflected first EM radiation includes receiving the reflected first EM radiation in a detector of a detector array, the detector corresponding to a portion of the reflected first EM radiation, and receiving a reflected second EM radiation includes receiving the reflected second EM radiation in a detector of a detector array, the event corresponding to a detector of a detector array.

[0117] Embodiment 18. The method of Embodiment 17, further comprising: the detector of the detector array comprises a first detector of the detector array, an event includes a first event, and a method includes: receiving a third EM emission reflected from the environment at a second detector of the detector array; calculating a third phase difference value indicating the difference between the phase of the emitted first EM emission and the phase of the reflected third EM emission; receiving a fourth EM emission reflected from the environment at a second detector of the detector array; calculating a fourth phase difference value indicating the difference between the phase of the emitted second EM emission and the phase of the reflected fourth EM emission; and triggering a second event in response to the difference between the fourth phase difference value and the third phase difference value exceeding a threshold, wherein the second event corresponds to the second detector of the detector array.

[0118] Embodiment 19. The method according to Embodiment 18, wherein a first phase difference value is determined by a first analog differential amplifier and a first analog inverse tangent calculation circuit, a second phase difference value is determined by a first analog differential amplifier and a first analog inverse tangent calculation circuit, a third phase difference value is determined by a second analog differential amplifier and a second analog inverse tangent calculation circuit, and a fourth phase difference value is determined by a second analog differential amplifier and a second analog inverse tangent calculation circuit.

[0119] Embodiment 20. The method according to any one of Embodiments 16 to 18, wherein the first analog difference circuit determines whether the difference between the second phase difference value and the first phase difference value exceeds a threshold, and the second analog difference circuit determines whether the difference between the fourth phase difference value and the third phase difference value exceeds a threshold.

[0120] Embodiment 21. The method according to any one of embodiments 16 to 20, further comprising outputting maps showing a first event and a second event.

[0121] Embodiment 22. The method according to Embodiment 21, wherein the map shows points in the environment where the depth measurement value changed, and the direction in which the depth measurement value changed.

[0122] Embodiment 23. The method according to any one of embodiments 18 to 22, further comprising outputting data streams indicating a first event and a second event.

[0123] Embodiment 24. The method according to Embodiment 23, wherein the data stream indicates points in the environment where the depth measurement changed, and the direction in which the depth measurement changed.

[0124] Embodiment 25. The method according to any one of Embodiments 16 to 24, further comprising modulating an emitted first EM radiation according to a reference modulation signal and modulating an emitted second EM radiation according to a reference modulation signal, wherein the first phase difference value is calculated based on the difference between the phase of the reference modulation signal and the phase of the envelope of the reflected first EM radiation, and the second phase difference value is calculated based on the difference between the phase of the reference modulation signal and the phase of the envelope of the reflected second EM radiation.

[0125] Embodiment 26. The method according to Embodiment 25, wherein the reference modulated signal is a square wave.

[0126] Embodiment 27. The method according to any one of Embodiments 16 to 26, wherein a first phase difference value is calculated asynchronously and a second phase difference value is calculated asynchronously.

[0127] Embodiment 28. The method according to any one of Embodiments 16 to 27, wherein a first phase difference value is determined by an analog differential amplifier and an analog inverse tangent calculation circuit, and a second phase difference value is determined by an analog differential amplifier and an analog inverse tangent calculation circuit.

[0128] Embodiment 29. The method according to any one of Embodiments 16 to 28, further comprising asynchronously determining whether the difference between a second phase difference value and a first phase difference value exceeds a threshold.

[0129] Embodiment 30. The method according to any one of Embodiments 16 to 29, wherein an analog difference circuit determines whether the difference between a second phase difference value and a first phase difference value exceeds a threshold.

Claims

1. A device for determining the change in distance, An EM emission emitter configured to emit electromagnetic (EM) radiation toward the environment, A detector configured to receive EM radiation reflected from the aforementioned environment, A phase calculation circuit configured to calculate a phase difference value indicating the difference between the phase of the emitted EM radiation and the phase of the reflected EM radiation, A device comprising: a difference circuit configured to trigger an event when the difference between the current phase difference value and the previous phase difference value exceeds a threshold;

2. The detector comprises a detector array, The apparatus according to claim 1, wherein each detector in the detector array corresponds to a corresponding portion of the reflected EM radiation.

3. The apparatus according to claim 2, further comprising a plurality of phase calculation circuits, each of which is connected to a corresponding detector of the detector array, and each of which is configured to calculate a corresponding phase difference value indicating a corresponding difference between the emitted EM radiation and the corresponding phase of the reflected EM radiation received by the corresponding detector.

4. The apparatus according to claim 3, further comprising a plurality of difference circuits, each of which difference circuits is connected to a corresponding phase calculation circuit of the plurality of phase calculation circuits, and each difference circuit of the plurality of difference circuits is configured to trigger an event when the difference between the current phase difference value and the previous phase difference value exceeds a threshold.

5. The apparatus according to claim 4, further comprising at least one processor configured to output a map showing events triggered by corresponding difference circuits of the plurality of difference circuits.

6. The apparatus according to claim 5, wherein the map indicates points in the environment where the depth measurement value changed between the time the previous phase difference value was determined and the time the current phase difference value was determined.

7. The aforementioned multiple phase calculation circuits operate asynchronously, The apparatus according to claim 4, wherein the plurality of difference circuits operate asynchronously.

8. The apparatus according to claim 4, wherein the plurality of difference circuits are configured to output a data stream representing the event.

9. The apparatus according to claim 8, wherein the data stream includes location information and direction information.

10. The apparatus according to claim 1, wherein the difference circuit is configured to output a data stream representing the event.

11. The apparatus according to claim 10, wherein the data stream includes location information and direction information.

12. The EM emission emitter is further configured to modulate the emitted EM radiation based on a reference modulation signal, The apparatus according to claim 1, wherein the phase difference value represents the difference between the phase of the reference modulation signal and the phase of the envelope of the reflected EM radiation.

13. The apparatus according to claim 12, wherein the reference modulation signal is a square wave.

14. The apparatus according to claim 1, wherein the phase calculation circuit comprises an analog differential amplifier and an analog inverse tangent calculation circuit.

15. The apparatus according to claim 1, wherein the difference circuit comprises an analog difference circuit.

16. A method for determining the change in distance, Emitting primary electromagnetic (EM) radiation into the environment, Receiving the first EM radiation reflected from the aforementioned environment, Calculating a first phase difference value that indicates the difference between the phase of the emitted first EM radiation and the phase of the reflected first EM radiation, Releasing a second wave of EM radiation into the environment, Receiving the second EM radiation reflected from the aforementioned environment, Calculating a second phase difference value that indicates the difference between the phase of the emitted second EM radiation and the phase of the reflected second EM radiation, A method comprising triggering an event in response to the difference between the second phase difference value and the first phase difference value exceeding a threshold.

17. Receiving the reflected first EM radiation includes receiving the reflected first EM radiation in a detector of a detector array, wherein the detector corresponds to a portion of the reflected first EM radiation. Receiving the reflected second EM radiation includes receiving the reflected second EM radiation in the detector of the detector array, The method according to claim 16, wherein the event corresponds to the detector of the detector array.

18. The detector of the detector array comprises the first detector of the detector array, The aforementioned event includes the first event, The method described above is The third EM radiation reflected from the aforementioned environment is received by the second detector of the detector array, Calculating a third phase difference value that indicates the difference between the phase of the emitted first EM radiation and the phase of the reflected third EM radiation, The fourth EM radiation reflected from the environment is received by the second detector of the detector array, Calculate a fourth phase difference value that indicates the difference between the phase of the emitted second EM radiation and the phase of the reflected fourth EM radiation, The second event is triggered when the difference between the fourth phase difference value and the third phase difference value exceeds a threshold, The method according to claim 17, further comprising the second event corresponding to the second detector of the detector array.

19. The first phase difference value is determined by the first analog differential amplifier and the first analog inverse tangent calculation circuit. The second phase difference value is determined by the first analog differential amplifier and the first analog inverse tangent calculation circuit. The third phase difference value is determined by the second analog differential amplifier and the second analog inverse tangent calculation circuit. The method according to claim 18, wherein the fourth phase difference value is determined by the second analog differential amplifier and the second analog inverse tangent calculation circuit.

20. The first analog difference circuit determines whether the difference between the second phase difference value and the first phase difference value exceeds the threshold. The method according to claim 18, wherein the second analog difference circuit determines whether the difference between the fourth phase difference value and the third phase difference value exceeds the threshold.

21. The method according to claim 18, further comprising outputting a map showing the first event and the second event.

22. The method according to claim 21, wherein the map shows points in the environment where the depth measurement value changed, and the direction in which the depth measurement value changed.

23. The method according to claim 18, further comprising outputting data streams indicating the first event and the second event.

24. The method according to claim 23, wherein the data stream indicates points in the environment where the depth measurement value changed, and the direction in which the depth measurement value changed.

25. The first EM radiation emitted is modulated according to a reference modulation signal, The method further includes modulating the emitted second EM radiation according to the reference modulation signal, The calculation of the first phase difference value is based on the difference between the phase of the reference modulation signal and the phase of the envelope of the reflected first EM radiation. The method according to claim 16, wherein the calculation of the second phase difference value is based on the difference between the phase of the reference modulation signal and the phase of the envelope of the reflected second EM radiation.

26. The method according to claim 25, wherein the reference modulated signal is a square wave.

27. The method according to claim 16, wherein the first phase difference value is calculated asynchronously, and the second phase difference value is calculated asynchronously.

28. The first phase difference value is determined by an analog differential amplifier and an analog inverse tangent calculation circuit. The method according to claim 16, wherein the second phase difference value is determined by the analog differential amplifier and the analog inverse tangent calculation circuit.

29. The method according to claim 16, further comprising asynchronously determining whether the difference between the second phase difference value and the first phase difference value exceeds the threshold.

30. The method according to claim 16, wherein an analog difference circuit determines whether the difference between the second phase difference value and the first phase difference value exceeds the threshold.

Images