Time-of-flight motion misalignment artifact correction

Abstract

Various techniques described herein relate to mitigating motion misalignment of a time-of-flight sensor system and / or generating lateral velocity estimation data using a time-of-flight sensor system. A stream of frames output by a sensor of a time-of-flight sensor system is received. A pair of non-adjacent frames in the frame stream is identified. The calculated optical flow data is calculated based on a pair of non-adjacent frames in the frame stream. Estimated optical flow data for the at least one different frame may be generated based on the calculated optical flow data, and the at least one different frame may be realigned based on the estimated optical flow data. Further, lateral velocity estimation data of the object may be generated based on the calculated optical flow data.

Description

BACKGROUND ART

[0001] In the aspect of environmental navigation, an autonomous vehicle senses objects around the autonomous vehicle based on sensor signals generated by a sensor system of the autonomous vehicle. For example, the autonomous vehicle may include various sensor systems, such as a radio detection and ranging (radar) sensor system, a camera sensor system, and / or a light detection and ranging (lidar) sensor system, for generating sensor signals. The autonomous vehicle also includes a centralized processing device that receives data based on the sensor signals generated by the sensor system and performs various different tasks, such as detecting vehicles, pedestrians, and other objects. Based on the output of the processing device, the autonomous vehicle can perform driving maneuvers.

[0002] Recently, a time-of-flight sensor system has been developed for autonomous vehicles. A time-of-flight sensor system is a device for measuring the distance to an object(s) in the environment. The time-of-flight sensor system can continuously capture multiple frames and combine these frames to form a point cloud. When the time-of-flight sensor system and the object(s) in the scene imaged by the time-of-flight sensor system are in relative motion with respect to each other, these frames (or certain sections of the frames) are no longer pixel-aligned. For example, the relative motion may depend on the distance of the object from the time-of-flight sensor system and the speed of movement of the time-of-flight sensor system (e.g., the speed of the autonomous vehicle incorporating the time-of-flight sensor system). As a result, errors are introduced in the estimation of the pixel depth of the pixels exhibiting relative motion. In addition, relative motion may also occur when the object(s) within the field of view of the time-of-flight sensor system undergoes motion independent of the time-of-flight sensor system (e.g., a pedestrian crossing the street).

[0003] The time-of-flight sensor system captures multiple discrete frames during a period of time. However, the scene may not be stationary during this period of time. By way of example, if the time-of-flight sensor is moving at a relatively high speed while capturing frames of a scene that includes parked cars and pedestrians moving at a relatively low speed, this relative motion causes the pixels of successive frames to shift between frames, which introduces errors when attempting to estimate depth.

[0004] Some conventional methods attempt to mitigate pixel misalignment in a time-of-flight sensor system by minimizing the amount of time from the start to the end of a frame capture sequence. For example, these methods attempt to compress frame capture in time to minimize the effects of relative motion. However, as the time period in which multiple discrete frames reside is compressed, the integration time of the frames is also shortened, resulting in less signal being collected for each frame. However, there are fundamental limits in terms of the amount of signal required for each frame captured by the time-of-flight sensor system, and thus, the integration time cannot be arbitrarily shortened, or else the required amount of signal cannot be collected. Additionally, it takes a limited amount of time to read information from the sensor (e.g., imager) of the time-of-flight sensor system; this time may depend on the design of the sensor and the speed of the analog-to-digital converter (ADC) of the sensor. This inter-measurement time between integration times is used to read, reset, and restart integration on the sensor. Thus, the design of the time-of-flight sensor system itself limits the degree to which the amount of time from the start to the end of a frame capture sequence can be compressed. Summary of the Invention

[0005] The following is a brief overview of the subject matter described in greater detail herein. This overview is not intended to limit the scope of the claims.

[0006] Various techniques for mitigating motion misalignment in a time-of-flight sensor system are described herein. A frame stream output by a sensor of the time-of-flight sensor system may be received (e.g., by a computing system). The frame stream includes a sequence of frames. The sequence of frames includes a set of frames, where the frames in the set have different frame types. The frame type of a frame represents the sensor parameters of the time-of-flight sensor system when the time-of-flight sensor system captures the frame, and thus different frame types represent different sensor parameters. For example, the sensor parameters of the time-of-flight sensor system may include the illumination state of the time-of-flight sensor system (e.g., whether the time-of-flight sensor system is emitting light or is prohibited from emitting light for the frame), the relative phase delay between the emitter system and the receiver system of the time-of-flight sensor system for the frame, and / or the integration time of the sensor of the time-of-flight sensor system for the frame. Additionally, a pair of non-adjacent frames in the frame stream may be identified. For example, the pair of non-adjacent frames may include successive frames of the same frame type. According to another example, the pair of non-adjacent frames may include successive frames with a relative phase delay having a 180-degree phase difference. Additionally, computed optical flow data may be calculated based on the pair of non-adjacent frames in the frame stream.

[0007] According to various embodiments, estimated optical flow data for at least one different frame in the frame stream other than the pair of non-adjacent frames may be generated based on the computed optical flow data. Additionally, the at least one different frame may be realigned based on the estimated optical flow data. Additionally, object depth data may be calculated based on the realigned frames in the sequence of frames, and a point cloud including the object depth data may be output.

[0008] In various embodiments, the estimated optical flow data for at least one different frame in the stream other than the pair of non-adjacent frames can be generated by: interpolating the estimated optical flow data for at least one intermediate frame between the pair of non-adjacent frames based on the calculated optical flow data. According to an example, the estimated optical flow data for an intermediate frame between the pair of non-adjacent frames can be calculated based on the optical flow data. According to other embodiments, the estimated optical flow data for at least one different frame in the stream other than the pair of non-adjacent frames can be generated by: extrapolating the estimated optical flow data for at least one successive frame after the pair of non-adjacent frames based on the calculated optical flow data.

[0009] In addition, according to other embodiments, the lateral velocity estimation data of an object detected in the frame stream can be generated. The pair of non-adjacent frames in the frame stream can be identified. In addition, the calculated optical flow data can be calculated based on the pair of non-adjacent frames in the frame stream. In addition, the lateral velocity estimation data of the object in the non-adjacent frames can be generated based on the calculated optical flow data. The lateral velocity estimation data of the object can also be generated based on the region in the environment of the time-of-flight sensor system included in the field of view of the frame and / or the depth data of the object.

[0010] Because the frames in the frame sequence can be realigned, the techniques described herein achieve motion artifact reduction. Therefore, the measurements at a single pixel between the realigned frames in the frame sequence are more likely to correspond to a common object at a relatively same depth in the scene. In addition, overlapping the realigned frames can improve the signal-to-noise ratio of a given pixel; thus, frame alignment improves the signal-to-noise ratio of the combined image. In addition, the depth accuracy of the realigned points can be enhanced.

[0011] The above Summary of the Invention presents a simplified summary in order to provide a basic understanding of some aspects of the systems and / or methods discussed herein. This Summary of the Invention is not an exhaustive overview of the systems and / or methods discussed herein. It is not intended to identify key / critical elements or to delineate the scope of such systems and / or methods. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A functional block diagram of an exemplary time-of-flight sensor system is illustrated.

[0013] Figure 2 An exemplary frame stream of the sensor output of a time-of-flight sensor system is illustrated.

[0014] Figure 3 Exemplary techniques that a motion analysis component can employ to interpolate the estimated optical flow data for frames in a frame stream in various embodiments are illustrated.

[0015] Figure 4 Illustrates an example of successive passive frames in a frame stream.

[0016] Figure 5 Illustrates another exemplary technique that a motion analysis component can employ to interpolate estimated optical flow data for frames in a frame stream in various embodiments.

[0017] Figure 6 Illustrates an exemplary technique that a motion analysis component can employ to extrapolate estimated optical flow data for frames in a frame stream in various embodiments.

[0018] Figure 7 Illustrates a functional block diagram of another exemplary time-of-flight sensor system.

[0019] Figure 8 Illustrates a functional block diagram of another exemplary time-of-flight sensor system.

[0020] Figure 9 Illustrates a functional block diagram of an exemplary autonomous vehicle including a time-of-flight sensor system.

[0021] Figure 10 Is a flowchart illustrating an exemplary method for mitigating motion misalignment of a time-of-flight sensor system.

[0022] Figure 11 Is a flowchart illustrating an exemplary method performed by a time-of-flight sensor system.

[0023] Figure 12 Illustrates an exemplary computing device. Detailed Description

[0024] Now refer to the accompanying drawings to describe various techniques related to mitigating motion misalignment of a time-of-flight sensor system and / or generating lateral velocity estimation data of an object detected by the time-of-flight sensor system, where like reference numerals are always used to refer to like elements. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. However, it may be apparent that such (one or more) aspects may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing one or more aspects. Additionally, it should be understood that functions described as being performed by certain system components may be executed by multiple components. Similarly, for example, one component may be configured to perform functions described as being executed by multiple components.

[0025] In addition, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise stated or the context clearly indicates otherwise, the phrase "X employs A or B" is intended to mean any natural inclusive permutation. That is, the phrase "X employs A or B" is satisfied by any of the following: X employs A; X employs B; or X employs both A and B. Further, the articles "a" and "an" as used in this application and the appended claims are generally to be construed to mean "one or more" unless otherwise specified or the context clearly indicates that they refer to the singular form.

[0026] As used herein, the terms "component" and "system" are intended to encompass a computer-readable data storage device configured with computer-executable instructions that, when executed by a processor, implement a particular function. The computer-executable instructions can include routines, functions, and the like. It should also be understood that a component or system can be located on a single device or distributed across multiple devices. Further, as used herein, the term "exemplary" is intended to mean "serving as an illustration or example of something".

[0027] As described herein, one aspect of the present technology is to collect and use available data from various sources to improve quality and experience. The present disclosure contemplates that, in some cases, the collected data may include personal information. The present disclosure contemplates that entities involved with such personal information respect and value privacy policies and practices.

[0028] The examples set forth herein relate to an autonomous vehicle that includes a time-of-flight sensor system that utilizes the techniques set forth herein to mitigate motion misalignment and / or generate lateral velocity estimation data. However, it should be understood that the time-of-flight sensor systems described herein can be used in a variety of different scenarios, such as flight, drone technology, monitoring technology (e.g., security technology), augmented reality (AR) or virtual reality (VR) technology, and so on. The autonomous vehicle is set forth herein as one possible use case, and the features of the claims are not limited to autonomous vehicles unless those claims expressly recite an autonomous vehicle.

[0029] Now referring to the drawings, Figure 1 illustrates an exemplary time-of-flight sensor system 100. The time-of-flight sensor system 100 includes a transmitter system 102 and a receiver system 104. In Figure 1 an example, the time-of-flight sensor system 100 can also include a computing system 106. However, in other embodiments, it is contemplated that the computing system 106 can be separate from, but in communication with, the time-of-flight sensor system 100.

[0030] The transmitter system 102 of the time-of-flight sensor system 100 can be configured to send modulated light into the environment of the time-of-flight sensor system 100. The light can propagate outward from the time-of-flight sensor system 100, reflect from an object in the environment of the time-of-flight sensor system 100, and return to the time-of-flight sensor system 100. The receiver system 104 can include a sensor 108 that can collect the light received at the time-of-flight sensor system 100 and output a frame stream.

[0031] The computing system 106 includes a processor 110 and a memory 112; the memory 112 includes computer-executable instructions executed by the processor 110. According to various examples, the processor 110 can be or include a graphics processing unit (GPU), multiple GPUs, a central processing unit (CPU), multiple CPUs, an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a microcontroller, a programmable logic controller (PLC), a field-programmable gate array (FPGA), etc.

[0032] The computing system 106 can receive the frame stream output by the sensor 108 of the time-of-flight sensor system 100. The frame stream includes a series of frame sequences. The frame sequence includes a set of frames, where the frames in the set have different frame types. The frame type of a frame represents the sensor parameters of the time-of-flight sensor system 100 when the time-of-flight sensor system 100 captures the frame. Thus, different frame types represent different frame parameters.

[0033] The various sensor parameters of the time-of-flight sensor system 100 are intended to fall within the scope of the appended claims. For example, the sensor parameters for a frame can include the illumination state of the time-of-flight sensor system 100 for that frame. The illumination state can indicate whether the time-of-flight sensor system 100 (e.g., the transmitter system 102) is emitting light or is prohibited from emitting light for that frame (e.g., whether the frame is a passive frame or a frame in which the transmitter system 102 emits a modulated light signal). Additionally, the sensor parameters of the time-of-flight sensor system 100 for a frame can include the relative phase delay between the transmitter system 102 and the receiver system 104 of the time-of-flight sensor system 100 for that frame. Additionally, the sensor parameters of the time-of-flight sensor system 100 for a frame can include the integration time of the sensor 108 of the time-of-flight sensor system 100 for that frame. It is contemplated that the time-of-flight sensor system 100 can employ a combination of the above sensor parameters.

[0034] The depth data of an object in the environment of the time-of-flight sensor system 100 can be calculated based on the frames in the frame sequence. However, as described herein, the relative motion between the time-of-flight sensor system 100 and the object(s) in the environment may cause pixel misalignment between the frames in the frame sequence, thereby introducing errors in the depth estimation(s). Therefore, the computing system 106 can employ some techniques to mitigate such motion misalignment between the frames.

[0035] The memory 112 of the computing system 106 can include a motion analysis component 114, a misalignment correction component 116, and a depth detection component 118. The motion analysis component 114 can identify a pair of non-adjacent frames in the frame stream received from the sensor 108. The pair of non-adjacent frames can be identified based on a frame type prior. Additionally, the motion analysis component 114 can calculate the computed optical flow data based on the pair of non-adjacent frames in the frame stream. The motion analysis component 114 can also generate estimated optical flow data for at least one different frame in the frame stream other than the pair of non-adjacent frames based on the computed optical flow data. Further, the misalignment correction component 116 can realign the at least one different frame based on the estimated optical flow data. The depth detection component 118 can also calculate the object depth data based on the realigned frames in the frame sequence. The depth detection component 118 can also output a point cloud including the object depth data.

[0036] Now turning to Figure 2 , an exemplary frame stream 200 output by the sensor 108 of the time-of-flight sensor system 100 is illustrated. As described above, the frame stream 200 includes a series of frame sequences. Figure 2 A portion of the frame sequence X and the next frame sequence X + 1 in the frame stream 200 is depicted in (where X can be substantially any integer greater than 0). The frame sequence X includes a set of frames, and each frame in the frame sequence X has a different frame type. Thus, each frame in the frame sequence X is captured by the time-of-flight sensor system 100 using different sensor parameters. In Figure 2In the depicted example, the frame sequence X includes nine frames (e.g., nine different frame types): frame (X,0), frame (X,1), frame (X,2), frame (X,3), frame (X,4), frame (X,5), frame (X,6), frame (X,7), and frame (X,8). Additionally, other frame sequences in the series of frame sequences of the frame stream 200 can be substantially similar to the frame sequence X. For example, the frame sequence X+1 can similarly include nine frames (e.g., nine different frame types): frame (X+1,0), frame (X+1,1), frame (X+1,2), frame (X+1,3), frame (X+1,4), frame (X+1,5), frame (X+1,6), frame (X+1,7), and frame (X+1,8). Further, the sensor parameters employed by the time-of-flight sensor system 100 when capturing the first frame in a frame sequence can be the same between frame sequences (e.g., the sensor parameters of frame (X,0) and frame (X+1,0) are substantially similar and are both frame type 0), and the sensor parameters employed by the time-of-flight sensor system 100 when capturing the second frame in a frame sequence can be the same between frame sequences (e.g., the sensor parameters of frame (X,1) and frame (X+1,1) are substantially similar and are both frame type 1); and so on. Although the various examples set forth herein describe frame sequences in the frame stream 200 that include nine frames, it should be understood that a frame sequence can include two or more frames; thus, the claimed subject matter is not limited to frame sequences that include nine frames.

[0037] Examples of the sensor parameters of each frame in the frame sequence X (and other similar frame sequences in the frame stream 200) are described below. Again, it should be understood that the claimed subject matter is not limited thereto, as each frame sequence can include other numbers of frames or different sensor parameters can be employed.

[0038] The first frame (X,0) in the frame sequence X can be an unilluminated passive frame (e.g., a grayscale frame). Thus, the illumination state of this first frame (X,0) can indicate that the time-of-flight sensor system 100 (e.g., the emitter system 102) is prohibited from emitting light for this frame (X,0). The first frame (X,0) can also have a relatively long integration time of the sensor 108. The remaining eight frames in the frame sequence X can be illuminated frames; thus, the illumination states of the remaining frames (X,1)-(X,8) can indicate that the time-of-flight sensor system 100 (e.g., the emitter system 102) emits light for these frames. Additionally, the frames (X,1)-(X,8) can have different combinations of the relative phase delay between the emitter system 102 and the receiver system 104 of the time-of-flight sensor system 100 and the integration time of the sensor 108.

[0039] More specifically, the second frame (X, 1) in the frame sequence X can have a relative phase delay of 0° between the transmitter system 102 and the receiver system 104, and have a relatively long integration time of the sensor 108. In addition, the third frame (X, 2) in the frame sequence X can have a relative phase delay of 90° between the transmitter system 102 and the receiver system 104, and have a relatively long integration time of the sensor 108. Further, the fourth frame (X, 3) in the frame sequence X can have a relative phase delay of 180° between the transmitter system 102 and the receiver system 104, and have a relatively long integration time of the sensor 108. The fifth frame (X, 4) in the frame sequence X can have a relative phase delay of 270° between the transmitter system 102 and the receiver system 104, and have a relatively long integration time of the sensor 108.

[0040] The sixth frame (X, 5) in the frame sequence X can have a relative phase delay of 0° between the transmitter system 102 and the receiver system 104, and have a relatively short integration time of the sensor 108. The seventh frame (X, 6) in the frame sequence X can have a relative phase delay of 90° between the transmitter system 102 and the receiver system 104, and have a relatively short integration time of the sensor 108. In addition, the eighth frame (X, 7) in the frame sequence X can have a relative phase delay of 180° between the transmitter system 102 and the receiver system 104, and have a relatively short integration time of the sensor 108. Further, the ninth frame (X, 8) in the frame sequence X can have a relative phase delay of 270° between the transmitter system 102 and the receiver system 104, and have a relatively short integration time of the sensor 108.

[0041] As described above, each frame sequence in the frame stream 200 can be substantially similar to each other. Therefore, the order of the frame types within each frame sequence in the frame stream 200 can be repeated.

[0042] According to an example, the time-of-flight sensor system 100 can capture a set of frames in a frame sequence within a period of about several milliseconds or dozens of milliseconds (e.g., between 1 millisecond and 100 milliseconds, between 10 milliseconds and 100 milliseconds). The period of time for capturing the frames in the frame sequence and the relative movement between the time-of-flight sensor system 100 and the object(s) in the scene may cause misalignment between the pixels of the frames (or parts thereof).

[0043] In addition, due to the frame-by-frame change in the scene structure caused by changing the inter-frame active illumination (e.g., changing the sensor parameters of the time-of-flight sensor 100 for each frame in a frame sequence), conventional relative motion estimation techniques may not be applicable to the frames in the frame stream 200. A prerequisite for such conventional relative motion estimation techniques is that each frame in the stream is substantially similar to the previous frame. In contrast, as described herein, the computing system 106 can mitigate such motion misalignment. The computing system 106 can correct the time-based misalignment between frames of different frame types. The motion analysis component 114 identifies pairs of frames (e.g., non-adjacent frames in the frame stream 200) having a similar scene structure. The motion analysis component 114 further compares such frames in the pair to calculate the calculated optical flow data. In addition, the motion analysis component 114 also generates estimated optical flow data for intermediate or future frame(s) based on the calculated optical flow data (e.g., via interpolation or extrapolation).

[0044] Reference Figure 3 , illustrates an exemplary technique 300 that the motion analysis component 114 can employ to interpolate the estimated optical flow data for the frames in the frame stream 200. As described above, the motion analysis component 114 identifies a pair of non-adjacent frames in the frame stream. In addition, the motion analysis component 114 calculates the calculated optical flow data based on the pair of non-adjacent frames. In addition, the motion analysis component 114 estimates the optical flow data for at least one different frame in the frame stream other than the pair of non-adjacent frames based on the calculated optical flow data.

[0045] In Figure 3 's example, a pair of non-adjacent frames identified by the motion analysis component 114 includes frame (X,0) and frame (X + 1,0). Thus, in Figure 3 's example, the motion analysis component 114 identifies a pair of non-adjacent frames in the frame stream that belong to the same frame type. More specifically, in Figure 3 's example, the pair of non-adjacent frames in the frame stream identified by the motion analysis component 114 includes successive passive frames for which the time-of-flight sensor system 100 is prohibited from emitting light (e.g., frame (X,0) and frame (X + 1,0) are successive grayscale frames from a successive frame sequence in the frame stream).

[0046] In addition, the motion analysis component 114 can calculate the calculated optical flow data based on the pair of non-adjacent frames in the frame stream. Thus, in Figure 3 the depicted example, the motion analysis component 114 can calculate the calculated optical flow data based on frame (X,0) and frame (X + 1,0) (e.g., based on the comparison between frame (X,0) and frame (X + 1,0)). Thus, the motion analysis component 114 performs optical flow analysis not between adjacent frames in the frame stream, but between pairs of non-adjacent frames in the frame stream.

[0047] Figure 4 depicts examples of successive passive frames (X,0) (e.g., solid line) and (X+1,0) (e.g., dashed line). The motion analysis component 114 can calculate the vertical and horizontal optical flow values for each pixel in the frame. Thus, the calculated optical flow data can represent the relative motion of the shown car between successive passive frames with respect to the time-of-flight sensor system 100 (e.g., the horizontal shift between frames).

[0048] Referring again to Figure 3 . As described above, the motion analysis component 114 generates estimated optical flow data for at least one different frame in the frame stream other than the pair of non-adjacent frames based on the calculated optical flow data. In Figure 3 the example set forth in, the motion analysis component 114 interpolates the estimated optical flow data for the intermediate frames between the pair of non-adjacent frames based on the calculated optical flow data. Thus, the estimated optical flow data for frames (X,1)-(X,8) can be interpolated based on the calculated optical flow data (e.g., calculated based on frames (X,0) and (X+1,0)). In Figure 3 the example of, the motion analysis component 114 calculates the calculated optical flow data based on the successive passive frames for which the time-of-flight sensor system 100 is prohibited from emitting light, and the motion analysis component 114 interpolates the estimated optical flow data for each intermediate frame between the successive passive frames based on the calculated optical flow data. In Figure 3 the example of, the other frames in the stream other than the passive frames are not used to calculate the calculated optical flow data; instead, the estimated optical flow data for these frames other than the passive frames is interpolated.

[0049] The motion analysis component 114 can also generate the estimated optical flow data for the intermediate frames between the pair of non-adjacent frames based on the timestamp information of the intermediate frames. For example, based on the normalized frame period time T = 1 (e.g., the normalized time period between frame (X,0) and frame (X+1,0)), examples of the measured timing ratio factor F (e.g., determined based on the corresponding timestamp information) are provided below, which can be used by the motion analysis component 114 to represent the timing of frames (X,1)-(X+1,O) relative to frame (X,0): F = [65.2174e-003,152.1739e-003,217.3913e-003,282.6087e-003,369.5652e-003,456.5217e-003,521.7391e-003,608.6957e-003,l.OOOOe+OOO], However, it is contemplated that the claimed subject matter is not limited to the above examples, as the timestamp information for each frame can be recorded and used by the motion analysis component 114 to calculate the measured timing ratio factor F.

[0050] An example of an algorithm for estimated optical flow data that can be implemented by the motion analysis component 114 to interpolate intermediate frames is described below.

[0051] [IJ] = [I + Round(F * OVFv) J + Round(F * OVFh)]

[0052] Where

[0053] [IJ]: represents the row and column indices of the pixels of the frame;

[0054] F: measured timing ratio factor; and

[0055] OVFv and OVFh: vertical and horizontal optical flow of each pixel.

[0056] According to various examples, the motion between the pair of non-adjacent frames can be assumed to be linear (e.g., the motion analysis component 114 can linearly interpolate the estimated optical flow data of the intermediate frames). However, in other examples, it is envisioned that the motion can be modeled in a non-linear manner (e.g., acceleration can be modeled as part of the interpolation performed by the motion analysis component 114).

[0057] Refer to Figure 5 , another exemplary technique 500 is illustrated, which in various embodiments can be used by the motion analysis component 114 to interpolate the estimated optical flow data of the frames in the frame stream 200. Figure 5 Depicts a series of operations that the motion analysis component 114 can perform; this series of operations can be repeatedly executed throughout the frame stream.

[0058] At 502, a pair of non-adjacent frames identified by the motion analysis component 114 includes frame (X,0) and frame (X + 1,0). Similar to the Figure 3 example, the motion analysis component 114 can identify a pair of non-adjacent frames belonging to the same frame type in the frame stream. In addition, the motion analysis component 114 can calculate the calculated optical flow data based on the pair of non-adjacent frames (X,0) and (X + 1,0). In addition, at 502, the motion analysis component 114 interpolates the estimated optical flow data for the intermediate frames between the pair of non-adjacent frames based on the calculated optical flow data. More specifically, the motion analysis component 114 interpolates the estimated optical flow data of frame (X,l) based on the calculated optical flow data, which is calculated based on the pair of non-adjacent frames (X,0) and (X + 1,0).

[0059] In addition, at 504, the pair of non - adjacent frames identified by the motion analysis component 114 includes frame (X, l) and frame (X + 1, 1). Similarly, the pair of non - adjacent frames (X, l) and (X + 1, 1) belong to the same frame type (but are of a different frame type from the frame pairs used at 502). In addition, the motion analysis component 114 can calculate the calculated optical flow data based on the pair of non - adjacent frames (X, l) and (X + 1, 1). In addition, at 504, the motion analysis component 114 interpolates the estimated optical flow data for the intermediate frame (X, 2) between the pair of non - adjacent frames (X, l) and (X + 1, 1) based on the calculated optical flow data, and the calculated optical flow data is calculated based on the pair of non - adjacent frames (X, l) and (X + 1, 1).

[0060] In addition, at 506, the pair of non - adjacent frames identified by the motion analysis component 114 includes frame (X, 2) and frame (X + 1, 2). Similarly, the pair of non - adjacent frames (X, 2) and (X + 1, 2) belong to the same frame type (but are of a different frame type from the frame pairs used at 502 and 504). In addition, the motion analysis component 114 can calculate the calculated optical flow data based on the pair of non - adjacent frames (X, 2) and (X + 1, 2). In addition, at 506, the motion analysis component 114 interpolates the estimated optical flow data for the intermediate frame (X, 3) between the pair of non - adjacent frames (X, 2) and (X + 1, 2) based on the calculated optical flow data, and the calculated optical flow data is calculated based on the pair of non - adjacent frames (X, 2) and (X + 1, 2).

[0061] The above operations can be repeated throughout the frame stream. Thus, the motion analysis component 114 can calculate the calculated optical flow data for each pair of non - adjacent frames in a sequence of successive frames in the frame stream that belong to the same frame type. In addition, similar to the content above regarding Figure 3 the motion analysis component 114 can also generate the estimated optical flow data for the intermediate frames based on the timestamp information of the intermediate frames.

[0062] According to an example, it is envisioned that the calculated optical flow data generated at 502 can be used to interpolate the estimated optical flow data for frame (X, l) and then discarded. In this example, the calculated optical flow data generated at 504 can be used to interpolate the estimated optical flow data for frame (X, 2) without using the calculated optical flow data generated at 502. However, in another example, the calculated optical flow data generated at 502 can be combined with the calculated optical flow data generated at 504 and used to interpolate the estimated optical flow data for frame (X, 2).

[0063] Moving on to Figure 6 illustrates an exemplary technique 600, which, in various embodiments, can be used by the motion analysis component 114 to extrapolate the estimated optical flow data for each frame in the frame stream 200. InFigure 6 In the example of Figure 6 , a pair of non - adjacent frames identified by the motion analysis component 114 includes successive frames with a relative phase delay having a 180 - degree phase difference. Thus, the pair of non - adjacent frames identified by the motion analysis component 114 includes frame (X, l) and frame (X, 3). Although frame (X, l) and (X, 3) belong to different frame types, the relative phase delay with a 180 - degree phase difference causes these frames to be related to each other. The motion analysis component 114 can calculate the calculated optical flow data based on the pair of non - adjacent frames (X, l) and (X, 3). In addition, the motion analysis component 114 can extrapolate the estimated optical flow data of at least one successive frame after the pair of non - adjacent frames based on the calculated optical flow data. For example, the motion analysis component 114 can extrapolate the estimated optical flow data of successive frames (X, 4)-(X, 8) based on the calculated optical flow data calculated according to the frame pair (X, l) and (X, 3). According to another example, it is contemplated that the motion analysis component 114 can also extrapolate the estimated optical flow data of (one or more) frames of the next frame sequence X + l based on the calculated optical flow data calculated according to the frame pair (X, l) and (X, 3). According to another example, the motion analysis component 114 can utilize different frame pairs, where these different frame pairs include successive frames with a relative phase delay having a 180 - degree phase difference within the frame sequence (e.g., the frame pair (X, l) and (X, 3) can be used to extrapolate the estimated optical flow data of frame (X, 4), the frame pair (X, 2) and (X, 4) can be used to extrapolate the estimated optical flow data of frame (X, 5), and so on).

[0064] Now referring generally to Figures 1-6 . As described herein, the motion analysis component 114 calculates the calculated optical flow data based on a pair of non - adjacent frames in the frame stream. In addition, the motion analysis component 114 can utilize one or more of the techniques described herein to generate the estimated optical flow data of each frame in the frame stream. The misalignment correction system 116 can employ the calculated optical flow data and the estimated optical flow data to realign the frames in the frame stream. Thus, the optical flow data for each frame can be used to realign the frames to mitigate motion misalignment artifacts in the frame stream. In addition, the depth detection component 118 can calculate the object depth data based on the realigned frames. For example, the depth detection component 118 can calculate the object depth data based on the realigned frames in the frame sequence. In addition, the depth detection component 118 can output a point cloud including the object depth data.

[0065] The techniques described herein can achieve motion artifact reduction because the frames in the frame sequence can be realigned. Thus, the measurements at a single pixel between the realigned frames in the frame sequence are more likely to correspond to a common object at a relatively same depth in the scene. In addition, overlapping the realigned frames can improve the signal - to - noise ratio of a given pixel; thus, frame alignment improves the signal - to - noise ratio of the combined image. In addition, the depth accuracy of the realigned points can be enhanced.

[0066] Now turning to Figure 7 , an example of a time-of-flight sensor system 100 according to various embodiments is illustrated (e.g., a time-of-flight camera system). Figure 7 The time-of-flight sensor system 100 similarly includes a transmitter system 102, a receiver system 104, and a computing system 106; however, it should also be understood that the computing system 106 can be separate from the time-of-flight sensor system 100 but communicate therewith. The time-of-flight sensor system 100 may also include an oscillator 702. The transmitter system 102 may also include a phase shifter 704, a driver 706, a light source 708, and an optical device 710. Additionally, the receiver system 104 may include a sensor 108 (e.g., a time-of-flight sensor chip) and an optical device 712.

[0067] The oscillator 702 may generate a radio frequency (RF) oscillator clock signal that can be sent to the phase shifter 704 and the sensor 108. The phase shifter 704 may delay the RF oscillator clock signal received from the oscillator 702 (relative to the RF oscillator clock signal provided to the sensor 108), thereby providing a desired relative phase delay between the transmitter system 102 and the receiver system 104 for a given frame. The delayed signal may be input to the driver 706 to modulate the light source 708 (e.g., a light emitting diode (LED) or an LED array). The modulated light output by the light source 708 may be shaped by the optical device 710 and emitted into the environment of the time-of-flight sensor system 100. Thus, the modulated light emitted by the transmitter system 102 may include an RF signal (e.g., an amplitude modulated signal, an RF-wavefront).

[0068] The light emitted into the environment may impinge on one or more objects in the environment and may be backscattered. The returned light carries three-dimensional information via RF signals (e.g., RF-wavefronts) with different time-of-flight delays, and the returned light may be mapped by the optical device 712 onto the sensor 108 (e.g., a time-of-flight sensor). Additionally, the sensor 108 may communicate with the computing system 106. The computing system 106 may control the sensor 108 and / or may receive a frame stream from the sensor 108 (e.g., including digitized three-dimensional information). As described herein, the computing system 106 may perform various signal processing on the frame stream to generate an output (e.g., a point cloud).

[0069] Although not shown, in another example, it is contemplated that a phase shifter 704 can be included as part of the receiver system 102 instead of as part of the transmitter system 102. In accordance with this example, the oscillator 702 can send an RF oscillator clock signal to the driver 706 and the phase shifter 704 (which can be included between the oscillator 702 and the sensor 108). In accordance with another example, the time-of-flight sensor system 100 can include two phase shifters.

[0070] Now referring Figure 8 , another example of the time-of-flight sensor system 100 is illustrated. Again, the time-of-flight sensor system 100 includes a transmitter system 102 and a receiver system 104 (including the sensor 108). The time-of-flight sensor system 100 can also include a computing system 106. In Figure 8 's example, the memory 112 includes a motion analysis component 114, a misalignment correction component 116, a depth detection component 118, and a velocity estimation component 802.

[0071] Similar to the above, the computing system 106 can receive a frame stream output by the sensor 108 (e.g., frame stream 200). Additionally, the motion analysis component 114 can identify a pair of non-adjacent frames in the frame stream. The motion analysis component 114 can also calculate the calculated optical flow data based on the pair of non-adjacent frames in the frame stream.

[0072] The velocity estimation component 802 can generate lateral velocity estimation data of an object in the non-adjacent frames based on the calculated optical flow data. The lateral velocity estimation data can include vertical and horizontal velocity estimation data. The velocity estimation component 802 can also generate the lateral velocity estimation data of the object based on the area included in the field of view of the frame in the time-of-flight sensor 100 environment. Additionally, the velocity estimation component 802 can generate the lateral velocity estimation data of the object based on the object depth data of the object (e.g., generated by the depth detection component 118 based on the realigned frames in the frame sequence adjusted by the misalignment correction component 116 described herein). According to another example, it is contemplated that the velocity estimation component 802 can additionally or alternatively generate the velocity estimation data of the object based on the estimated optical flow data generated by the motion analysis component 114.

[0073] According to various examples, it is contemplated that the velocity estimation component 802 can additionally or alternatively generate radial velocity estimation data of an object. The velocity estimation component 802 can utilize two depth maps and two optical flow maps to evaluate the pixel-level correspondence between two consecutive depth map estimates. Based on this pixel-level correspondence, the velocity estimation component 802 can output the radial velocity estimation data.

[0074] Turning to Figure 9, an autonomous vehicle 900 is illustrated. The autonomous vehicle 900 can travel on a road without human control based on sensor signals output by the sensor system of the autonomous vehicle 900. The autonomous vehicle 900 includes multiple sensor systems. More specifically, the autonomous vehicle 900 includes the time-of-flight sensor system 100 described herein. The autonomous vehicle 900 may also include one or more heterogeneous sensor systems 902. The heterogeneous sensor systems 902 may include (one or more) GPS sensor systems, (one or more) ultrasonic sensor systems, (one or more) infrared sensor systems, (one or more) camera systems, (one or more) lidar sensor systems, (one or more) radio radar sensor systems, etc. The sensor systems 100 and 902 may be arranged around the autonomous vehicle 900.

[0075] The autonomous vehicle 900 also includes multiple mechanical systems for enabling the proper movement of the autonomous vehicle 900. For example, the mechanical systems may include, but are not limited to, a vehicle propulsion system 904, a braking system 906, and a steering system 908. The vehicle propulsion system 904 may be an electric engine or an internal combustion engine. The braking system 906 may include engine brakes, brake pads, actuators, and / or any other suitable components configured to assist the autonomous vehicle 900 in decelerating. The steering system 908 includes suitable components configured to control the moving direction of the autonomous vehicle 900.

[0076] The autonomous vehicle 900 also includes a computing system 910 that communicates with the sensor systems 100 and 902, the vehicle propulsion system 904, the braking system 906, and the steering system 908. The computing system 910 includes a processor 912 and a memory 914; the memory 914 includes computer-executable instructions executed by the processor 912. According to various examples, the processor 912 may be or include a graphics processing unit (GPU), multiple GPUs, a central processing unit (CPU), multiple CPUs, an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a microcontroller, a programmable logic controller (PLC), a field-programmable gate array (FPGA), etc.

[0077] According to an example, the computing system 910 may include the computing system 106. In another example, the time-of-flight sensor system 100 may include the computing system 106, and the computing system 910 may communicate with the computing system 106 of the time-of-flight sensor system 100.

[0078] The memory 914 of the computing system 910 may include a positioning system 916, a perception system 918, a planning system 920, and a control system 922. The positioning system 916 may be configured to determine the local position of the autonomous vehicle 900. The perception system 918 may be configured to perceive objects near the autonomous vehicle 900 (e.g., based on the outputs from the sensor systems 100 and 902). For example, the perception system 918 may detect, classify, and predict the behavior of objects near the autonomous vehicle 900. The perception system 918 (and / or one or more different systems included in the memory 914) may track objects near the autonomous vehicle 900 and / or make predictions regarding the operating environment of the autonomous vehicle 900 (e.g., predict the behavior of objects near the autonomous vehicle 900). Additionally, the planning system 922 may plan the movement of the autonomous vehicle 900. Additionally, the control system 922 may be configured to control at least one mechanical system of the autonomous vehicle 900 (e.g., at least one of the vehicle propulsion system 904, the braking system 906, and / or the steering system 908).

[0079] The computing system 910 may control the operation of the autonomous vehicle 900 based at least in part on the output data generated by the time-of-flight sensor system 100. Although Figure 9 the time-of-flight sensor system 100 described as included is part of the autonomous vehicle 900, it is contemplated that the time-of-flight sensor system 100 may be used in other types of scenarios (e.g., included in other types of systems, etc.).

[0080] Figures 10-11 Exemplary methods related to the operation of the time-of-flight sensor system are illustrated. Although these methods are shown and described as a series of actions performed in sequence, it should be understood and recognized that these methods are not limited by the sequence order. For example, some actions may occur in a different order than described herein. Additionally, one action may occur concurrently with another action. Additionally, in some cases, not all actions are necessary to implement the methods described herein.

[0081] Furthermore, the actions described herein may be computer-executable instructions that may be implemented by one or more processors and / or stored on one or more computer-readable media. The computer-executable instructions may include routines, subroutines, programs, execution threads, etc. Additionally, the results of the method actions may be stored in a computer-readable medium, displayed on a display device, etc.

[0082] Figure 10Method 1000 for reducing motion misalignment of a time-of-flight sensor system is illustrated. At 1002, a frame stream output by a sensor of the time-of-flight sensor system can be received. The frame stream includes a series of frame sequences. The frame sequence includes a set of frames, where the frames in the set have different frame types. The frame type of a frame represents the sensor parameters of the time-of-flight sensor system when the time-of-flight sensor system captures the frame. Thus, different frame types represent different sensor parameters. At 1004, a pair of non-adjacent frames in the frame stream can be identified. At 1006, optical flow data calculated based on the pair of non-adjacent frames in the frame stream can be obtained. At 1008, based on the calculated optical flow data, estimated optical flow data for at least one different frame other than the pair of non-adjacent frames in the frame stream can be generated. At 1010, based on the estimated optical flow data, at least one different frame is realigned.

[0083] In various examples, object depth data can also be calculated based on the realigned frames in the frame sequence. Additionally, a point cloud including the object depth data can be output.

[0084] Go to Figure 11 Method 1100 performed by a time-of-flight sensor system is illustrated. At 1102, a frame stream output by a sensor of the time-of-flight sensor system can be received. The frame stream includes a series of frame sequences. The frame sequence includes a set of frames, where the frames in the set have different frame types. The frame type of a frame represents the sensor parameters of the time-of-flight sensor system when the time-of-flight sensor system captures the frame. Thus, different frame types represent different sensor parameters. At 1104, a pair of non-adjacent frames in the frame stream can be identified. At 1106, optical flow data calculated based on the pair of non-adjacent frames in the frame stream can be obtained. At 1108, estimated lateral velocity data of an object in the non-adjacent frames can be generated based on the calculated optical flow data.

[0085] According to various examples, estimated optical flow data for at least one different frame other than the pair of non-adjacent frames in the frame stream can be generated based on the calculated optical flow data. Additionally, the at least one different frame can be realigned based on the estimated optical flow data. Additionally, object depth data of the object can be calculated based on the realigned frames in the frame sequence. Thus, in addition to the optical flow data, estimated lateral velocity data of the object can be generated based on the object depth data of the object.

[0086] Now refer to Figure 12, which illustrates a high - level diagram of an exemplary computing device 1200 that can be used according to the systems and methods disclosed herein. For example, the computing device 1200 can be or include a computing system 910. According to another example, the computing device 1200 can be or include a computing system 106. The computing device 1200 includes at least one processor 1202 that executes instructions stored in a memory 1204. These instructions can be, for example, instructions for implementing the functions described as being implemented by one or more of the above - mentioned systems or instructions for implementing one or more of the above - mentioned methods. The processor 1202 can be a GPU, multiple GPUs, a CPU, multiple CPUs, a multi - core processor, etc. The processor 1202 can access the memory 1204 via a system bus 1206. In addition to storing executable instructions, the memory 1204 can also store frames, timestamps, computed optical flow data, estimated optical flow data, object depth data, point clouds, etc.

[0087] The computing device 1200 also includes a data repository 1208 that the processor 1202 can access via the system bus 1206. The data repository 1208 can include executable instructions, frames, timestamps, computed optical flow data, estimated optical flow data, object depth data, point clouds, etc. The computing device 1200 also includes an input interface 1210 that allows external devices to communicate with the computing device 1200. For example, the input interface 1210 can be used to receive instructions from an external computer device, etc. The computing device 1200 also includes an output interface 1212 that interfaces the computing device 1200 with one or more external devices. For example, the computing device 1200 can transmit control signals to a vehicle propulsion system 904, a braking system 906, and / or a steering system 908 via the output interface 912.

[0088] In addition, although the computing device 1200 is illustrated as a single system, it should be understood that it can be a distributed system. Thus, for example, multiple devices can communicate via a network connection and can jointly perform the tasks described as being performed by the computing device 1200.

[0089] The various functions described herein can be implemented by hardware, software, or any combination thereof. If implemented in software, the functions can be stored on or transmitted via a computer-readable medium as one or more instructions or code. A computer-readable medium includes a computer-readable storage medium. A computer-readable storage medium can be any available storage medium accessible by a computer. By way of example, and not limitation, such computer-readable storage medium can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired program code in the form of instructions or data structures and that is accessible by a computer. As used herein, "disk" or "optical disk" includes compact disk (CD), laser disk, optical disk, digital versatile disk (DVD), floppy disk, and Blu-ray disk (BD), where disks typically reproduce data magnetically, while optical disks typically reproduce data optically with lasers. Additionally, propagated signals are not included within the scope of computer-readable storage media. A computer-readable medium also includes a communication medium, which includes any medium that facilitates transfer of a computer program from one place to another. For example, a connection can be a communication medium. For instance, if software is transferred from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of communication medium. Combinations of the above should also be included within the scope of computer-readable media.

[0090] Alternatively or additionally, the functions described herein can be performed, at least in part, by one or more hardware logic components. By way of example, and not limitation, illustrative types of hardware logic components that can be used include field-programmable gate arrays (FPGA), application-specific integrated circuits (ASIC), application-specific standard products (ASSP), system-on-a-chip (SOC), complex programmable logic devices (CPLD), and the like.

[0091] The systems and methods have been described herein with reference to at least the following examples.

[0092] (Al) In one aspect, the present disclosure describes a computing system that includes a processor and a memory. The memory stores computer-executable instructions that, when executed by the processor, cause the processor to perform the actions described herein. These actions include receiving a frame stream output by sensors of a time-of-flight sensor system, where the frame stream includes a sequence of frames. The sequence of frames includes a set of frames, where the frames in the set have different frame types, and the frame type of a frame represents the sensor parameters of the time-of-flight sensor system when the frame is captured by the time-of-flight sensor system. Thus, the different frame types represent different sensor parameters. The actions also include identifying a pair of non-adjacent frames in the frame stream. Additionally, the actions include calculating calculated optical flow data based on the pair of non-adjacent frames in the frame stream. Additionally, the actions include generating estimated optical flow data for at least one different frame in the frame stream other than the pair of non-adjacent frames based on the calculated optical flow data. The actions also include realigning the at least one different frame based on the estimated optical flow data.

[0093] (A2) In some embodiments of the computing system of (Al), the actions further include: calculating object depth data based on the realigned frames in the sequence of frames, and outputting a point cloud that includes the object depth data.

[0094] (A3) In some embodiments of the computing system of (A2), the set of frames in the sequence of frames is captured by the time-of-flight sensor system within a time period between 1 millisecond and 100 milliseconds.

[0095] (A4) In some embodiments of at least one of the computing systems of (A1) to (A3), the sensor parameters of the time-of-flight sensor system when capturing a frame include at least one of the following: the illumination state of the time-of-flight sensor system, such that the time-of-flight sensor system emits light or is prohibited from emitting light for the frame; the relative phase delay between the transmitter system and the receiver system of the time-of-flight sensor system for the frame; or the integration time of the sensor of the time-of-flight sensor system for the frame.

[0096] (A5) In some embodiments of at least one of the computing systems of (A1) to (A4), generating the estimated optical flow data for the at least one different frame in the stream other than the pair of non-adjacent frames includes interpolating the estimated optical flow data for at least one intermediate frame between the pair of non-adjacent frames based on the calculated optical flow data.

[0097] (A6) In some embodiments of at least one of the computing systems of (A1) to (A4), generating the estimated optical flow data for the at least one different frame in the stream other than the pair of non-adjacent frames includes interpolating the estimated optical flow data for the intermediate frames between the pair of non-adjacent frames based on the calculated optical flow data.

[0098] (A7)In some embodiments of at least one computing system among (A1)-(A4), generating the estimated optical flow data for at least one different frame in the stream other than the pair of non-adjacent frames includes: extrapolating the estimated optical flow data for at least one successive frame after the pair of non-adjacent frames based on the calculated optical flow data.

[0099] (A8)In some embodiments of at least one computing system among (A1)-(A7), the estimated optical flow data for the at least one different frame is further generated based on the timestamp information of the at least one different frame.

[0100] (A9)In some embodiments of at least one computing system from (A1) to (A8), the pair of non-adjacent frames in the stream includes successive frames of the same frame type.

[0101] (A10)In some embodiments of at least one computing system from (A1) to (A8), the calculated optical flow data is calculated for each pair of non-adjacent frames of the same frame type in a sequence of successive frames in the frame stream.

[0102] (A11)In some embodiments of at least one computing system from (A1) to (A8), the pair of non-adjacent frames in the stream for calculating the calculated optical flow data includes successive passive frames for which the time-of-flight sensor system is prohibited from emitting light.

[0103] (A12)In some embodiments of at least one computing system from (A1) to (A8), the pair of non-adjacent frames in the stream for calculating the calculated optical flow data includes successive frames with a relative phase delay of 180-degree phase difference.

[0104] (A13)In some embodiments of at least one computing system from (A1) to (A12), the time-of-flight sensor system includes the computing system.

[0105] (A14)In some embodiments of at least one computing system from (A1) to (A13), an autonomous vehicle includes the time-of-flight sensor system and the computing system.

[0106] (B1)On the other hand, a method for reducing motion misalignment of a time-of-flight sensor system is disclosed herein. The method includes receiving a frame stream output by a sensor of the time-of-flight sensor system, the frame stream including a series of frame sequences. The frame sequence includes a set of frames, wherein the frames in the set have different frame types, and the frame type of a frame represents the sensor parameters of the time-of-flight sensor system when the time-of-flight sensor system captures the frame, so different frame types represent different sensor parameters. The method further includes identifying a pair of non-adjacent frames in the frame stream. In addition, the method includes calculating calculated optical flow data based on the pair of non-adjacent frames in the frame stream. In addition, the method further includes generating estimated optical flow data for at least one different frame in the frame stream other than the pair of non-adjacent frames based on the calculated optical flow data. The method further includes re-aligning the at least one different frame based on the estimated optical flow data.

[0107] (B2)In some embodiments of the method of (B1), the method further includes: calculating object depth data based on the re-aligned frames in the frame sequence, and outputting a point cloud including the object depth data.

[0108] (B3)In some embodiments of at least one of the methods of (B1) to (B2), generating estimated optical flow data for at least one different frame in the stream other than the pair of non-adjacent frames includes interpolating estimated optical flow data for at least one intermediate frame between the pair of non-adjacent frames based on the calculated optical flow data.

[0109] (B4)In some embodiments of at least one of the methods of (B1) to (B2), generating estimated optical flow data for the at least one different frame in the stream other than the pair of non-adjacent frames includes interpolating estimated optical flow data for an intermediate frame between the pair of non-adjacent frames based on the calculated optical flow data.

[0110] (B5)In some embodiments of at least one of the methods of (B1) to (B2), generating estimated optical flow data for the at least one different frame in the stream other than the pair of non-adjacent frames includes: extrapolating estimated optical flow data for at least one successive frame after the pair of non-adjacent frames based on the calculated optical flow data.

[0111] (C1)In another aspect, a time-of-flight sensor system is disclosed herein, wherein the time-of-flight sensor system includes: a receiver system including a sensor; and a computing system in communication with the receiver system. The computing system includes a processor and a memory that stores computer-executable instructions that, when executed by the processor, cause the processor to perform actions. The actions include: receiving a frame stream output by the sensor of the receiver system of the time-of-flight sensor system, the frame stream including a series of frame sequences. The frame sequence includes a set of frames, wherein the frames in the set have different frame types, and the frame type of a frame represents the sensor parameters of the time-of-flight sensor system when the frame is captured by the time-of-flight sensor system, so different frame types represent different sensor parameters. The actions further include identifying a pair of non-adjacent frames in the frame stream. Additionally, the actions include calculating calculated optical flow data based on the pair of non-adjacent frames in the frame stream. Additionally, the actions include: generating estimated optical flow data for at least one different frame in the frame stream other than the pair of non-adjacent frames based on the calculated optical flow data. The actions further include: re-aligning the at least one different frame based on the estimated optical flow data; calculating object depth data based on the re-aligned frames in the frame sequence; and outputting a point cloud including the object depth data.

[0112] (D1)In another aspect, a computing system is disclosed herein, wherein the computing system includes a processor and a memory that stores computer-executable instructions that, when executed by the processor, cause the processor to perform actions. The actions include: receiving a frame stream output by the sensor of a time-of-flight sensor system, the frame stream including a series of frame sequences. The frame sequence includes a set of frames, wherein the frames in the set have different frame types, and the frame type of a frame represents the sensor parameters of the time-of-flight sensor system when the frame is captured by the time-of-flight sensor system, so different frame types represent different sensor parameters. The actions further include identifying a pair of non-adjacent frames in the frame stream. Additionally, the actions include calculating calculated optical flow data based on the pair of non-adjacent frames in the frame stream. The actions further include: generating estimated lateral velocity data for an object in the non-adjacent frames based on the calculated optical flow data.

[0113] (D2)In some embodiments of the computing system of (D1), the estimated lateral velocity data for the object is further generated based on a region in the environment of the time-of-flight sensor system included in the field of view of the frame.

[0114] (D3)In some embodiments of at least one computing system of (D1) to (D2), the actions further include generating estimated optical flow data for at least one different frame in the frame stream other than the pair of non - adjacent frames based on the computed optical flow data; realigning the at least one different frame based on the estimated optical flow data; and computing object depth data of the object based on the realigned frames in the frame sequence, wherein the lateral velocity estimation data of the object is further generated based on the object depth data of the object.

[0115] (D4)In some embodiments of the computing system of (D3), the set of frames in the frame sequence is captured by a time - of - flight sensor system within a time period between 1 millisecond and 100 milliseconds.

[0116] (D5)In some embodiments of at least one computing system of (D1) to (D4), the sensor parameters of the time - of - flight sensor system when capturing the frames include at least one of the following: the illumination state of the time - of - flight sensor system such that the time - of - flight sensor system emits light or is prohibited from emitting light for the frame; the relative phase delay between the transmitter system and the receiver system of the time - of - flight sensor system for the frame; or the integration time of the sensor of the time - of - flight sensor system for the frame.

[0117] (D6)In some embodiments of at least one computing system of (D1) to (D5), the pair of non - adjacent frames in the stream includes successive frames of the same frame type.

[0118] (D7)In some embodiments of at least one computing system of (D1) to (D5), the computed optical flow data is computed for each pair of non - adjacent frames of the same frame type in a successive frame sequence in the frame stream.

[0119] (D8)In some embodiments of at least one computing system of (D1) to (D5), the pair of non - adjacent frames in the stream for computing the computed optical flow data includes successive passive frames for which the time - of - flight sensor system is prohibited from emitting light.

[0120] (D9)In some embodiments of at least one computing system of (D1) to (D5), the pair of non - adjacent frames in the stream for computing the computed optical flow data includes successive frames with a relative phase delay of 180 - degree phase difference.

[0121] (D10)In some embodiments of at least one computing system of (D1) to (D9), the time - of - flight sensor system includes the computing system.

[0122] In some embodiments of at least one computing system among (D1) to (D10), the autonomous vehicle includes the time-of-flight sensor system and the computing system.

[0123] (E1) On the other hand, a method performed by a time-of-flight sensor system is disclosed herein. The method includes receiving a frame stream output by a sensor of the time-of-flight sensor system, the frame stream including a sequence of frames. The sequence of frames includes a set of frames, where the frames in the set have different frame types, and the frame type of a frame represents the sensor parameters of the time-of-flight sensor system when the time-of-flight sensor system captures the frame. Thus, different frame types represent different sensor parameters. Further, the method includes identifying a pair of non-adjacent frames in the frame stream. Additionally, the method includes calculating calculated optical flow data based on the pair of non-adjacent frames in the frame stream. The method also includes generating lateral velocity estimation data of an object in the non-adjacent frames based on the calculated optical flow data.

[0124] (E2) In some embodiments of the method of (E1), the lateral velocity estimation data of the object is also generated based on a region in the environment of the time-of-flight sensor system included in the field of view of the frame.

[0125] (E3) In some embodiments of at least one method among (E1) to (E2), the method further includes: generating estimated optical flow data of at least one different frame in the frame stream other than the pair of non-adjacent frames based on the calculated optical flow data; realigning the at least one different frame based on the estimated optical flow data; and calculating object depth data of the object based on the realigned frames in the sequence of frames, wherein the lateral velocity estimation data of the object is also generated based on the object depth data of the object.

[0126] (E4) In some embodiments of at least one method among (E1) to (E3), the sensor parameters of the time-of-flight sensor system when capturing the frame include: the illumination state of the time-of-flight sensor system, such that the time-of-flight sensor system emits light or is prohibited from emitting light for the frame; the relative phase delay between the transmitter system and the receiver system of the time-of-flight sensor system for the frame; and the integration time of the sensor of the time-of-flight sensor system for the frame.

[0127] (E5) In some embodiments of at least one method among (E1) to (E4), the pair of non-adjacent frames in the stream includes successive frames of the same frame type.

[0128] (E6) In some embodiments of at least one of the methods (E1) to (E4), the calculated optical flow data is calculated for each pair of non - adjacent frames of the same frame type in a sequence of successive frames in the frame stream.

[0129] (E7) In some embodiments of at least one of the methods (E1) to (E4), the pair of non - adjacent frames in the stream for calculating the calculated optical flow data includes successive passive frames for which the time - of - flight sensor system is prohibited from emitting light.

[0130] (E8) In some embodiments of at least one of the methods (E1) to (E4), the pair of non - adjacent frames in the stream for calculating the calculated optical flow data includes successive frames with relative phase delays having a 180 - degree phase difference.

[0131] (F1) In another aspect, a time - of - flight sensor system is disclosed herein. The time - of - flight sensor system includes: a receiver system including a sensor; and a computing system in communication with the receiver system. The computing system includes a processor and a memory, and the memory stores computer - executable instructions that, when executed by the processor, cause the processor to perform actions. The actions include: receiving a frame stream output by the receiver system of the time - of - flight sensor system, the frame stream including a sequence of frames. The frame sequence includes a set of frames, where the frames in the set have different frame types, and the frame type of a frame represents the sensor parameters of the time - of - flight sensor system when the frame is captured by the time - of - flight sensor system, so different frame types represent different sensor parameters. The actions further include identifying a pair of non - adjacent frames in the frame stream. Additionally, the actions include calculating calculated optical flow data based on the pair of non - adjacent frames in the frame stream. The actions also include: generating estimated optical flow data for at least one different frame in the frame stream other than the pair of non - adjacent frames based on the calculated optical flow data. The actions also include: realigning the at least one different frame based on the estimated optical flow data. The actions also include: calculating object depth data of an object based on the realigned frames in the frame sequence. Additionally, the actions include: generating lateral velocity estimation data of the object based on the calculated optical flow data and the object depth data of the object.

[0132] The foregoing includes examples of one or more embodiments. Of course, in order to describe the various aspects above, it is not possible to describe every conceivable modification and variation of the above-described apparatus or method, but one of ordinary skill in the art will recognize that many further modifications and permutations are possible for the various aspects. Accordingly, the various aspects are intended to cover all such alterations, modifications, and variations that fall within the scope of the appended claims. In addition, when the term "comprising" is used in the detailed description or claims, such term is intended to be inclusive in a manner similar to the term "including" as "including" is interpreted to be inclusive when used as a transitional word in a claim.

Claims

1. A computing system, comprising: a processor; and a memory storing computer-executable instructions that, when executed by the processor, cause the processor to perform actions including: receiving a frame stream output by a sensor of a time-of-flight sensor system, the frame stream including a sequence of frames, where the sequence of frames includes a set of frames, where the frames in the set have different frame types, and where the frame type of a frame represents the sensor parameters of the time-of-flight sensor system when the frame is captured by the time-of-flight sensor system, such that the different frame types represent different sensor parameters; identifying a pair of non-adjacent frames in the frame stream; calculating calculated optical flow data based on the pair of non-adjacent frames in the frame stream; generating estimated optical flow data for at least one different frame in the frame stream other than the pair of non-adjacent frames based on the calculated optical flow data; and re-aligning the at least one different frame based on the estimated optical flow data.

2. The computing system according to claim 1, wherein the actions further include: calculating object depth data based on the re-aligned frames in the sequence of frames; and outputting a point cloud including the object depth data.

3. The computing system according to claim 2, wherein, The set of frames in the sequence of frames is captured by the time-of-flight sensor system within a time period between 1 millisecond and 100 milliseconds.

4. The computing system according to at least one of claims 1-3, wherein, The sensor parameters of the time-of-flight sensor system when capturing the frames include at least one of the following: the illumination state of the time-of-flight sensor system such that the time-of-flight sensor system emits light or is prohibited from emitting light for the frame; the relative phase delay between the transmitter system and the receiver system of the time-of-flight sensor system for the frame; or the integration time of the sensor of the time-of-flight sensor system for the frame.

5. The computing system according to at least one of claims 1-4, wherein Generating the estimated optical flow data for the at least one different frame in the stream other than the pair of non-adjacent frames includes interpolating the estimated optical flow data for at least one intermediate frame between the pair of non-adjacent frames based on the calculated optical flow data.

6. The computing system according to at least one of claims 1-4, wherein, Generating the estimated optical flow data for the at least one different frame in the stream other than the pair of non-adjacent frames includes interpolating the estimated optical flow data for an intermediate frame between the pair of non-adjacent frames based on the calculated optical flow data.

7. The computing system according to at least one of claims 1-4, wherein Generating the estimated optical flow data for the at least one different frame in the stream other than the pair of non-adjacent frames includes: extrapolating the estimated optical flow data for at least one successive frame after the pair of non-adjacent frames based on the calculated optical flow data.

8. The computing system according to at least one of claims 1 to 7, wherein, The estimated optical flow data for the at least one different frame is also generated based on the timestamp information of the at least one different frame.

9. The computing system according to at least one of claims 1 to 8, wherein, The pair of non-adjacent frames in the stream includes successive frames of the same frame type.

10. The computing system according to at least one of claims 1 to 8, wherein, The calculated optical flow data is calculated for each pair of non-adjacent frames of the same frame type in a successive frame sequence in the frame stream.

11. The computing system according to at least one of claims 1 to 8, wherein, The pair of non-adjacent frames in the stream for calculating the calculated optical flow data includes successive passive frames for which the time-of-flight sensor system is prohibited from emitting light.

12. The computing system according to at least one of claims 1 to 8, wherein, The pair of non - adjacent frames in the flow for calculating the calculated optical flow data includes successive frames having a relative phase delay with a 180 - degree phase difference.

13. A method for reducing motion misalignment of a time - of - flight sensor system, comprising: Receiving a frame stream output by a sensor of the time - of - flight sensor system, the frame stream including a sequence of frames, where the frame sequence includes a set of frames, where the frames in the set have different frame types, and where the frame type of a frame represents the sensor parameters of the time - of - flight sensor system when the frame is captured by the time - of - flight sensor system, so different frame types represent different sensor parameters; Identifying a pair of non - adjacent frames in the frame stream; Calculating calculated optical flow data based on the pair of non - adjacent frames in the frame stream; Generating estimated optical flow data for at least one different frame in the frame stream other than the pair of non - adjacent frames based on the calculated optical flow data; And Re - aligning the at least one different frame based on the estimated optical flow data.

14. The method according to claim 13, further comprising: Calculating object depth data based on the re - aligned frames in the frame sequence; And Outputting a point cloud including the object depth data.

15. The method according to at least one of claims 13 - 14, wherein, Generating estimated optical flow data for at least one different frame in the frame stream other than the pair of non - adjacent frames includes: interpolating estimated optical flow data for at least one intermediate frame between the pair of non - adjacent frames based on the calculated optical flow data.

Images