SYSTEM AND METHOD FOR MOTION PREDICTION IN AUTONOMOUS DRIVING

Abstract

The present disclosure provides a system and method for motion prediction in autonomous driving. The system disclosed herein provides an efficient deep neural network based system for jointly performing perception and motion prediction from 3D point clouds. The system can take as input a pair of LiDAR sweeps and output, for each point in the second sweep, a classification of the point into one of a set of semantic classes and a motion vector indicating the movement of the point in a world coordinate system. The system comprises a spatio-temporal pyramid network that hierarchically extracts deep spatial and deep temporal features. Training of the system is regularized using spatial and temporal consistency losses. Thus, an improved motion planner for autonomous driving applications is provided.

Description

[Technical field]

[0001] The present disclosure relates generally to automated driving systems, and more particularly to systems and methods for neural network-based motion prediction. [Background technology]

[0002] Autonomous driving utilizes sensing technologies for robust and dynamic object perception and sequentially uses various perception sensors for reliable and safe vehicle decision-making. Among the various perception sensors, cameras and lidars are the two main sensors used for surrounding object recognition. Cameras provide semantically rich visual features of the traffic scenario, while lidars provide high-resolution point clouds that allow the identification of object distances.

[0003] Those familiar with the art will know that determining the state of the environment is important for deploying AVs (Automated Vehicles). Accurate environment state information for AVs is useful in facilitating motion planning and providing a smooth user experience. Estimating the environment state for AVs typically involves two tasks: (1) perception, which identifies foreground objects from the background, and (2) motion prediction, which predicts the future trajectory of objects such as foreground objects. So far, various methods have been developed to handle these two tasks separately or together with the help of deep learning.

[0004] Previous approaches to environment perception mainly rely on bounding box detection techniques implemented by 2D object detection based on camera data, 3D object detection based on LiDAR data, or fusion-based detection. The detected bounding boxes are then fed into an object tracker and subsequently into a motion predictor. Some recent work implements all these modules into an end-to-end framework that directly generates bounding boxes along the future trajectory.

[0005] However, these state estimation strategies tend not to work in open-set real-world traffic scenarios because they rely on object detection: in particular, it is difficult for the object detector to generalize to classes (in deep learning) that have never appeared in the training set, leading to failures in downstream estimation modules.

[0006] Some alternative solutions to environmental state detection and motion prediction include depicting 3D environmental information by using OGM (Occupancy Grid Map). OGM discretizes the 3D point cloud of the environment into 2D grid cells that are equal to each other. Each of these cells is constructed with the idea that the corresponding space is occupied by at least one point. With this design, OGM can be used to specify the future drivable space, and therefore can accommodate motion planning. However, one major weakness of OGM is the difficulty in finding the correspondence between 2D grid cells over time. This makes it difficult to explicitly model the dynamics of objects that are highly time-dependent. In addition to this, object category information is usually discarded in OGM. Thus, it is impossible to consider category-specific constraints on traffic actor movements to understand the relationships.

[0007] Another extension of OGM is to represent the world state with a Bird's Eye View (BEV) map. In a BEV map, each OGM cell additionally contains information about the cell's motion and the cell's category, but does not contain information about specific object motions or categories. While BEV is an improvement over OGM, utilizing a fixed grid of cells leads to discretization errors that do not accurately capture the continuous nature of the environment. More specifically, a BEV cell may contain points from multiple distinct objects, each with different categories and / or motions. By assigning the same value category and motion to all points in a cell, such a system fails to correctly represent the true state of the world.

[0008] Thus, there is a recognized need for improved systems and methods for more accurate motion prediction and state estimation of moving objects, such as AVs. Summary of the Invention

[0009] Therefore, it is an object of some embodiments to address the above-mentioned challenges in the context of autonomous driving.

[0010] Some embodiments are based on the recognition that there is a need for a point cloud motion sensor to estimate motion information of at least some points of an environment without approximating the point cloud, because approximating these point clouds by projecting them onto a 2D grid of cells or BEVs is disadvantageous for various traffic assistance and control applications such as controlling automated devices such as AVs or robots, traffic estimation, anomaly detection, generating crosswalk warning messages, and traffic light schedule adjustment. For example, a bicycle that is far away from the control vehicle can be represented by only a few points, and approximating these few points in the grid or BEV map may remove important information.

[0011] Therefore, it is an objective of some embodiments to provide motion information for each point of an input 3D point cloud, such as a classification of the point as static or dynamic, a displacement vector indicating the displacement of the point from the previous point cloud to the current point cloud, and a prediction vector indicating the prediction of the motion of the point from the current point cloud to the next point cloud.

[0012] Some embodiments are based on the realization that at least two obstacles stand in the way of achieving these goals. First, there are so many points in a 3D point cloud that processing all the points and their temporal relationships is computationally expensive, even too costly. Second, it is difficult to provide a temporal correlation between corresponding points in different 3D point clouds that indicates their motion. For example, it is possible to find correspondences between points using techniques borrowed from optical and fluid analysis, but these techniques are also computationally expensive.

[0013] It is therefore an objective of some embodiments to provide a system and method suitable for finding point correspondences over time by utilizing time-invariant spatial neighborhoods to extract spatio-temporal features in a computationally cost-efficient manner, without the need to build point correspondences in a subsequent 3D point cloud.

[0014] Some embodiments are based on the realization that point-to-point correspondences and comparisons of different corresponding points can be replaced by correspondences between neighboring locations of a point of interest at different times. Thus, by comparing features of neighboring points, features that are close in space and time can be compared without the need to construct corresponding points. This is because, in the case of point clouds, corresponding points are likely to be located within a neighborhood, and comparisons can be made to these neighboring features rather than to the features of the individual points. This reduces the amount of computation, which is advantageous in time-sensitive applications such as real-time autonomous driving applications.

[0015] Therefore, some embodiments disclose a motion sensor comprising an encoder for encoding spatio-temporal features of each point of a current 3D point cloud, and a decoder for decoding the spatio-temporal encoding of each point to generate motion information. The encoding is performed in two steps. In a first step, only spatial features of individual 3D point clouds that are close to a location corresponding to a location of a point of interest in the current 3D point cloud are considered. Some embodiments are based on the realization that these features are close and are determined separately for each point cloud, so that the computational cost can be consolidated. Also, since features of the neighborhood of the point locations are obtained instead of features of corresponding points, there is no need to establish such point-to-point correspondences. Thus, in a first step, spatial encodings of features of the neighborhood of the current point in the current 3D frame at different time instants are calculated.

[0016] Then, during a second step, the spatial codings of the features at different times are combined to generate a spatio-temporal coding of the points in the current 3D point cloud. Thus, the temporal correlation does not depend on an estimation of the corresponding pixels, i.e., the temporal correlation is a position-based correlation rather than a pixel-based correlation. Some embodiments are based on the realization that by doing so, the computational requirements for determining such a spatio-temporal coding of all points in the current 3D point cloud can be reduced.

[0017] Various embodiments are based on the recognition that the ability to reliably approximate the state of the environment, in particular the presence of objects and their motion behavior, is crucial for autonomous driving. Accordingly, some embodiments provide an efficient deep neural network-based system for performing joint perception and motion prediction from 3D point clouds. The system can take as input a pair of LiDAR sweeps and output, for each point in the second sweep, a classification of the point into one of a set of semantic classes and a motion vector indicating the movement of the point in a world coordinate system.

[0018] Some embodiments provide a spatio-temporal pyramid network as part of a motion estimator that hierarchically extracts deep spatial and deep temporal features of the input 3D point cloud. To ensure smoothness of predictions across both space and time, the training of the system is further regularized with novel spatial and temporal consistency losses. The proposed system can act as a backup to systems based on bounding boxes, which are well known in the art, and provide supplementary information to the motion planner in autonomous driving applications.

[0019] According to an embodiment, a point cloud motion sensor for estimating motion information of at least some points of an environment is provided. The motion sensor includes a depth sensor, such as a Lidar, configured to sense a dynamic environment and collect time-series 3D (three-dimensional) point clouds of the environment, including a current 3D point cloud and a previous 3D point cloud. The motion sensor further includes a motion estimator operatively connected to the depth sensor via one or a combination of a wired or wireless communication channel. The motion estimator is configured to iteratively process the sequence of 3D point clouds with a neural network. The neural network includes an encoder to provide a spatio-temporal encoding of each point in each of the 3D clouds, and a decoder to decode the spatio-temporal encoding to generate motion information for each point in each of the 3D clouds. To encode the current point in the current 3D point cloud, the encoder is configured to extract features of adjacent points in the current 3D point cloud that are located in a vicinity of a position of the current point, and generate a current spatial encoding of the current point in the current frame. The encoder is further configured to extract features of adjacent points in the previous 3D point cloud that are located near a location in the previous 3D point cloud that corresponds to the location of the current point, to generate a previous spatial encoding of the current point in the previous frame, and to combine the current spatial encoding and the previous spatial encoding to generate a space-time encoding of the current point.

[0020] According to another embodiment, a computer-implemented method for estimating motion information of at least some points of an environment is provided. The method includes detecting a dynamic environment and collecting a time series of 3D (three dimensional) point clouds of the environment, including a current 3D point cloud and a previous 3D point cloud. The method further includes iteratively estimating motion information of the series of 3D point clouds with a neural network. The neural network includes an encoder for providing a spatio-temporal coding of each point in each of the 3D clouds, and a decoder for decoding the spatio-temporal coding to generate motion information for each point in each of the 3D clouds. The encoder is configured to extract features of adjacent points in the current 3D point cloud that are located near a position of the current point to generate a current spatial coding of the current point in the current frame. The encoder is further configured to extract features of adjacent points in the previous 3D point cloud that are located near a position in the previous 3D point cloud that corresponds to a position of the current point to generate a previous spatial coding of the current point in the previous frame. The encoder is further configured to combine the current spatial coding and the previous spatial coding to generate a spatio-temporal coding of the current point. The method then includes outputting the estimated motion information. [Brief description of the drawings]

[0021] The presently disclosed embodiments will be further described with reference to the accompanying drawings, in which the drawings are not necessarily to scale, with emphasis instead generally being placed upon illustrating the principles of the presently disclosed embodiments.

[0022] [Figure 1] FIG. 1 is a block diagram of a motion estimation system for estimating motion information of a 3D point cloud according to an embodiment of the present disclosure. [Diagram 2] 2 is a block diagram of the motion estimation system of FIG. 1 including a neural network according to an embodiment of the present disclosure. [Diagram 3] FIG. 3 is a block diagram of an example practical use case of the neural network of FIG. 2 in accordance with an embodiment of the present disclosure. [Figure 4]FIG. 3 is another block diagram of the internal structure of the neural network of FIG. 2 according to an embodiment of the present disclosure. [Diagram 5] FIG. 3 is a block diagram of intermediate operational steps of the neural network of FIG. 2 based on proxy point calculation, according to an embodiment of the present disclosure. [Figure 6] FIG. 1 is a schematic diagram illustrating a computing device that can be used to implement the systems and methods of the present disclosure. [Figure 7] FIG. 1 is a schematic diagram illustrating a use case for implementing the systems and methods of the present disclosure. [Figure 8] FIG. 1 is a schematic diagram to illustrate another use case for implementing the systems and methods of the present disclosure. [Figure 9] FIG. 13 is a schematic diagram illustrating yet another use case for implementing the systems and methods of the present disclosure. [Figure 10] 1 is a block diagram of a method for motion information estimation according to an embodiment of the present disclosure. [Figure 11] FIG. 1 is a schematic diagram illustrating a system that can be used to implement the systems and methods of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that the present disclosure may be practiced without these specific details. In other instances, devices and methods are shown only as block diagrams in order to avoid obscuring the present disclosure.

[0024] As used herein and in the appended claims, the terms "for example," "for instance," and "such as," as well as the verbs "comprising," "having," "including," and other verb forms thereof, when used in conjunction with a list of one or more components or other items, should be construed as open-ended, meaning that the list should not be considered to exclude other additional components or other items. The term "based on" means based, at least in part, on. Moreover, it is to be understood that the phraseology and terminology used herein are for purposes of description and should not be regarded as limiting. The headings used herein are for convenience only and do not have any legal or limiting effect.

[0025] FIG. 1 illustrates a block diagram of a point cloud motion sensor 100 including a motion estimation system 120 (synonymously referred to as a motion estimator) for estimating motion information of a 3D point cloud, according to an embodiment of the present disclosure. The motion estimation system 120 is configured to receive as input a time-series 3D point cloud 110 associated with an environment of the motion estimation system 120, which in one example includes an autonomous driving environment. The time-series 3D point cloud 110 is sensed by a sensor, such as a depth sensor. The sensor can sense a dynamic environment and collect the time-series 3D point cloud 110. The time-series 3D point cloud 110 includes a current 3D point cloud and a previous 3D point cloud.

[0026] The motion estimation system 120 is operatively connected to the depth sensor via one or a combination of wired and wireless communication channels. In some embodiments, the motion estimation system 120 may be implemented as a system having a processor and a memory. The memory may store computer-executable instructions that are executed by the processor to perform the functions of the motion estimation system 120. For example, the motion estimation system is configured to iteratively process the time-series 3D point clouds using a neural network. The detailed structure of such a neural network is detailed in FIG. 2.

[0027] Processing the time series 3D point cloud 110 by the neural network of the motion estimation system 120 results in the generation of a multi-head motion estimation output. As such, the neural network of the motion estimation system 120 is a multi-head neural network that generates different outputs 130 as different heads of the neural network. These different outputs 130 can be motion information head 1 130a and motion information head 2 130b, etc. Although the motion estimation system 120 of FIG. 1 shows two outputs, this is for illustrative purposes only. In practice, there can be any number of heads in the output of the motion estimation system 120 without departing from the scope of this disclosure.

[0028] In one embodiment, the multi-head motion estimation output 130 includes a motion vector head and a motion state head, such that the neural network included in the motion estimation system 120 is a multi-head network that processes the set of 3D point clouds 110 and generates predicted motion vectors and estimated motion states for every 3D point in the current point cloud.

[0029] In one embodiment, the multi-head motion estimation output 130 includes a segmentation head and a motion state head. Thus, the motion estimation system 120 includes a neural network that processes the set of 3D point clouds 110 and generates semantic labels and estimated motion states for all 3D points in the current point cloud. Thus, the neural network may further include a segmentation neural network trained to extract semantic features from each 3D point cloud in the input set of 3D point clouds 110. The neural network may further include a segmentation head trained to estimate semantic labels for all 3D points in the current point cloud. The neural network further includes a point motion neural network trained to extract motion features based on semantic features across multiple timestamps and a motion state head trained to estimate motion states.

[0030] The detailed structure of the neural network included in the motion estimation system 120 will be described below with reference to FIGS. 2, 3, 4, and 5. FIG.

[0031] FIG. 2 illustrates a block diagram of the internal structure of neural network 140 included in motion estimation system 120, according to an embodiment of the present disclosure.

[0032] The neural network 140 comprises an encoder 150 for providing a spatio-temporal encoding of each point in each of the 3D point clouds 110. To this end, in order to encode a current point in the current 3D point cloud, the encoder 150 is configured to extract features of adjacent points in the current 3D point cloud located in the vicinity of a position of the current point, to generate a current spatial encoding of the current point in the current frame. Furthermore, the encoder 150 is configured to extract features of adjacent points in the previous 3D point cloud located in the vicinity of a position in the previous 3D point cloud corresponding to the position of the current point, to generate a previous spatial encoding of the current point in the previous frame. Furthermore, the encoder 150 is configured to combine the current spatial encoding and the previous spatial encoding to generate a spatio-temporal encoding of the current point.

[0033] The neural network 140 further comprises a decoder 160 which decodes the spatio-temporal coding produced by the encoder 150 and generates motion information for each point in each of the 3D point clouds 110. The neural network 140 is thus a multi-head (neural) network 140 which processes the set of 3D point clouds 110 and generates, for every 3D point in the input set of 3D point clouds 110, predicted motion vectors in the form of a motion vector head 130a and estimated motion states in the form of a motion state head 130b.

[0034] The multi-head neural network 140 generates multiple outputs by adding multiple heads (head 1 130a, head 2 130b, etc.) to the end of the multi-head neural network 140 depending on the desired function. For example, for motion classification, the two heads include a motion vector head 130a that predicts a motion vector for each 3D point in the current frame, and a motion state head 130b that estimates a binary motion state for each 3D point in the current frame. Thus, in one exemplary embodiment, the motion vector head 130a is implemented with three fully connected layers, and the motion state head 130a is implemented with one fully connected layer followed by a Leaky Rectified Linear Unit (ReLU).

[0035] In an embodiment, the neural network 140 uses the Smooth L1 Loss L vector to supervise the motion vector prediction in the motion vector head 130a and to supervise the motion state head 130b with the cross entropy error L state The total loss L = L vector +0.1L state The motion state head 130b for motion state estimation and the motion vector head 130b for motion vector prediction are jointly trained using the motion state estimation head 130b. In an exemplary embodiment, a binary estimation of the motion state is used to refine the predicted motion vector during the testing phase. If the estimated motion state is a static state, the corresponding motion vector is set to 0.

[0036] In another example, a multi-head neural network produces the various outputs shown in FIG.

[0037] FIG. 3 illustrates another block diagram of the neural network 140 of FIG. 2 for generating the multi-head output 130 according to an embodiment of the present disclosure.

[0038] The neural network 140 is a multi-head network that processes a series of 3D point clouds 110, generates semantic or class labels in the segmentation head 130a1, and generates estimated motion states as the motion state head 130b for all 3D points in the current point cloud.

[0039] In some embodiments, the neural network 140 includes a segmentation neural network trained to extract semantic features from each 3D point cloud, a segmentation head 130a1 trained to estimate semantic labels, a point motion neural network trained to extract motion features based on semantic features across multiple timestamps, and a motion state head 130b trained to estimate motion states.

[0040] To generate semantic labels in the segmentation head 130a1, a multi-sweep semantic segmentation is performed on the given sequence of 3D point clouds 110 by the multi-head neural network 140.

[0041] In an embodiment, a multi-head neural network 140 is configured to take multiple sweeps as input 110 and directly estimate all semantic and motion labels at output 130 .

[0042] Semantic labels include, for example, cars, roads, buildings, sidewalks, plants, etc. However, in many implementations of neural network 140 for motion estimation in dynamic environments, etc., the foreground categories or semantic labels, such as vehicles, pedestrians, and cyclists, contain mostly independent semantic and motion information. Therefore, it is not advantageous to jointly train neural network 140 to generate semantic and motion information. Also, the 3D point clouds in the previous sweep may not be of much benefit to the semantic labels for the points in the current sweep, because there is no clear point-to-point correspondence for points across multiple sweeps. Therefore, in an embodiment, the multi-sweep semantic segmentation performed by neural network 140 is divided into two separate tasks: semantic segmentation based on one sweep and motion state estimation based on multiple sweeps.

[0043]

number

[0044] The segmentation head 130a1 takes the semantic features of all points in the current sweep and generates a semantic label for each point. It is implemented by a fully connected layer followed by a softmax layer. The third component, a neural network 140, learns motion features based on the semantic features from the segmentation backbone across all sweeps. The motion state head 130b takes the motion features and estimates the motion state for each point in the current sweep. It is implemented by a fully connected layer followed by a sigmoid layer.

[0045] In an embodiment, cross-entropy loss is employed for both the multi-category semantic segmentation performed by segmentation head 130a1 and the binary estimation of motion states performed by motion state head 130b. Furthermore, a foreground mask is applied to motion state estimation head 130b based on the ground truth of semantic categories so that only the loss of foreground points is counted. The overall loss function is a combination of the semantic segmentation loss and the motion state loss. In one example, a pre-trained segmentation backbone is used in the training phase to focus on motion training and reduce training time. In the inference phase, for an estimated background category, the neural network 140 then sets the corresponding estimated motion state to a static state.

[0046] At the heart of the multi-sweep semantic segmentation system is an underlying neural network 140, which is described below in conjunction with FIG.

[0047] FIG. 4 is a diagram illustrating the detailed internal structure of neural network 140 of FIG. 2, showing contraction branches 170 and expansion branches 180, in accordance with an embodiment of the present disclosure.

[0048] Some embodiments are based on the realization that location-based extraction of features from a 3D point cloud, followed by a subsequent perceptron combination, results in a spatio-temporal convolution (STC) of the 3D point cloud that is advantageous for extracting motion information. However, the effectiveness of spatio-temporal convolution can be further enhanced by extending the principles described above to a deep neural network 140 structure as shown in FIG.

[0049] Thus, in some embodiments, the neural network 140 includes a pyramidal and / or U-shaped architecture formed by a reduction branch 170 that successively downsamples the input and an expansion branch 180 that successively upsamples the input. The reduction branch 170 includes one or more pairs of encoders as encoder layers, and the expansion branch 180 includes multiple corresponding decoder layers that enable spatio-temporal feature representation learning. In addition, the reduction branch 170 includes one or more downsampling layers, and the expansion branch 180 includes one or more pairs of decoders and corresponding upsampling layers that enable multi-scale feature representation learning. Some embodiments also include skip connections that enable the aggregation layer to fuse features from both the encoder 150 and the decoder 160.

[0050] Some embodiments are based on the realization that the encoder generates an encoding of a current point cloud. When the encoder is first run, the current point cloud 210 is encoded into an encoding 220 based on other previous point clouds 200 and 190. However, in order to further encode encoding 220 into a secondary encoding 230, encodings 240 and 250 need to correspond to encoding 220.

[0051] Thus, if point cloud 200 or 190 were the current frame, one embodiment uses the encodings 240 and 250 determined in the previous iteration. This embodiment is computationally more cost-efficient, but requires that the encodings 210, 200, and 190 be stored in memory for subsequent processing.

[0052] Thus, the reduction branch 170 comprises multiple encoders arranged to process a downsampled output of a first encoder indicative of a current encoding of spatio-temporal features of the current point cloud, including a first encoder corresponding to encoding 210 and a second encoder corresponding to encoding 220, the second encoder processing the current encoding together with a corresponding encoding of a previous point cloud determined during a previous iteration.

[0053] To this end, the neural network 140 receives a sequence of 3D point clouds, where each of the original point cloud frames is described by its local coordinate system. Moreover, all past frames are synchronized with the current frame, i.e., the coordinate transformation represents all of the point cloud in the ego-vehicle's current coordinate system. This is important to counteract the AV's ego-motion and avoid spurious motion estimation. In addition to this, it aggregates more points of the static background while giving hints about the motion of the moving object.

[0054] The neural network 140 further classifies each point in the point cloud at the current time and estimates its future location. The spatio-temporal pyramid network shown in Figure 4 extracts features hierarchically along both spatial and temporal dimensions. The left part acts like an encoder 150 that extracts spatio-temporal features at multiple scales, and the right part acts like a decoder 160 that focuses on aggregating only the features in the current frame from both the previous layers and the encoder.

[0055] The basic building block of the neural network 140 is the Point-Space-Time Convolution (p-STC) block. As the main component to extract spatio-temporal features from points, the Point-STC layer implements 4D point spatio-temporal convolution. The first layer takes raw points from multiple frames as input, where the point features are their 3D coordinates. For deeper layers, the point features are features not seen by previous layers. For each point, the spatio-temporal neighborhood is implemented by collecting all 3D points that are within a predefined spatial distance to this query point at each timestamp. This is illustrated with the help of the following example equations:

[0056]

number

[0057] Therefore, instead of finding point-to-point correspondences, implementing the neural network 140 to perform Point-STC as described above allows us to exploit correspondences over time in a time-invariant neighboring space. Since this neighboring space for each point is fixed taking into account the radius, spatial features within this space are evaluated over time to extract motion features.

[0058]

number

[0059] Moreover, the point-STC calculation described above directly processes raw 4D points without discretization. The calculation only takes points that are adjacent to each other, so there is no need to process the entire point cloud at the same time. Instead, the large-scale point cloud is divided into multiple patches called small-scale point clouds, and each patch is processed locally. This significantly reduces the calculations, making it easier to process large-scale 3D point clouds on a computing system.

[0060] In an embodiment, 3D point clouds are downsampled to reduce the computational cost. Therefore, in large scenes, point clouds are usually sampled irregularly, with nearby points more densely packed and distant points more scattered. Unfortunately, downsampling makes distant points too scattered. To solve this problem, downsampling is implemented based on 3D voxels. First, the 3D scene is divided into contiguous non-overlapping 3D voxels. In each non-empty voxel, one 3D point is sampled. If there are multiple 3D points in one voxel, their centroids are taken as a proxy in this voxel. At each timestamp, the sampled points are collected from all voxels into one downsampled 3D point cloud. In this way, the more nearby points are downsampled and the less distant points are downsampled, balancing the spatial distribution of point density. The size of a voxel is d L+1 =2d L where L is the index of the network layer. In some embodiments, the downsampled 3D points are still irregularly scattered in the 3D space, which is clearly different from a regular voxelization.

[0061] The neural network 140 further includes an upsampling layer whose purpose is to populate point features from the downsampled point cloud into the original point cloud (the layer before the downsampling layer): for each 3D point in the original point cloud, we find its closest point in the downsampled point cloud and take its associated point feature.

[0062] The neural network 140 further includes an aggregation layer configured to aggregate point features at the same scale from both the encoder 150 and the decoder 160. In the pyramidal structure of Fig. 4, a skip connection 240 is shown as a line connecting the left encoder 150 branch to the right decoder 160 branch to concatenate the point feature at the current timestamp from the encoder side with the corresponding point feature from the upsampled point cloud. A multi-layer perceptron is then used to aggregate the concatenated features.

[0063] Additionally or alternatively, one embodiment reorders the points, changing the notion of a current point cloud and a previous point cloud. For example, after the encoder generates encoding 220 by processing current point cloud 210 and previous point cloud 200, an embodiment reorders the points and sends the reordered points as current point cloud 200 and previous point cloud 210 to generate encoding 240.

[0064] To this end, the encoder 150 is configured to distinguish between the current point cloud 210 and the previous point cloud 200 based on the order of the points in the sequence of point clouds. To this end, a first encoder is run to process the current point cloud 210 and the previous point cloud 200 to generate a current encoding 220. The order of the current point cloud and the previous point cloud is then changed. The first encoder is then run again to process the reordering of the current point cloud 200 and the previous point cloud 210 to generate a previous encoding 240. Furthermore, a second encoder is run to process the current encoding 220 and the previous encoding 240.

[0065]

number

[0066] Furthermore, the current formulation of using proxy points avoids the need to determine point-to-point correspondences between different sweeps, and instead leads to finding relationships between points that are in close proximity.

[0067] According to some embodiments, the motion estimation system 120 is configured to generate proxy points, where adjacent points in the current 3D point cloud and adjacent points in the previous 3D point cloud are constrained to a sphere centered on the position of the current point in the current 3D point cloud.

[0068] Some embodiments are based on the recognition that a location-to-location comparison may take into account features of a different number of points at different locations to evaluate the motion information of the current point. This comparison is more suitable for 3D point clouds than 2D images, but this difference may make the training of the neural network 140 unstable. To address this issue, some embodiments utilize a weighted kernel function that extracts features of neighboring points for multiple artificially introduced proxy points in each neighborhood. The weights of the weighted kernel function are determined by training the neural network 140, but the structure of the kernel function is preset to extract features of neighboring points for the current point. However, by introducing the proxy points into the kernel function, the difference between extracting features in a sphere centered on an existing point, such as the current 3D point cloud, and extracting features in a possibly empty space, such as the previous point cloud, or in a sphere centered on another point, is reduced.

[0069] Furthermore, the features of adjacent points in the current 3D point cloud and the features of adjacent points in the previous 3D point cloud are combined by processing the combined extracted features of the current 3D point cloud and the extracted features of the previous 3D point cloud with a multi-layer perceptron, so that the weighted kernel function includes weights that are trainable parameters of the neural network 140 shared by all points of the current point cloud.

[0070]

number

[0071] Thus, the neural network 140 based motion estimation system 120 described in the previous embodiments may be used for multiple purposes in real-time automated, semi-automated, or manual driving applications utilizing any suitable computing device.

[0072] FIG. 6 is a schematic diagram illustrating a computing device 600 that can be used to implement the systems and methods of the present disclosure.

[0073] The computing device 600 comprises a power supply 601, a processor 603, a memory 605, and a storage device 607, all connected to the bus 609. Additionally, a high-speed interface 611, a low-speed interface 613, a high-speed expansion port 615, and a low-speed expansion port 617 may be connected to the bus 609. In addition, a low-speed connection port 619 is connected to the bus 609. Additionally, an input interface 621 may be connected to an external receiver 623 and an output interface 625 via the bus 609. A receiver 627 may be connected to an external transmitter 629 and a transmitter 631 via the bus 609. An external memory 633, an external sensor 635, a machine(s) 637, and an environment 639 may also be connected to the bus 609. Additionally, one or more external input / output devices 641 may be connected to the bus 609. A NIC (Network Interface Controller) 643 may be adapted to connect to a network 645 through the bus 609. Here, image data or other data may be rendered, among other things, on a third party display device, a third party imaging device, and / or a third party printing device external to the computing device 600 .

[0074] The memory 605 can store instructions executable by the computing device 600 and any data that can be utilized by the methods and systems of the present disclosure. The memory 605 can include a random access memory (RAM), a read only memory (ROM), flash memory, or other suitable memory system. The memory 605 can be one or more volatile storage devices and / or one or more non-volatile storage devices. The memory 605 can also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0075] The storage device 607 may be adapted to store auxiliary data and / or software modules utilized by the computing device 600. The storage device 607 may include a hard drive, an optical drive, a thumb drive, an array of drives, or any combination thereof. Additionally, the storage device 607 may include a computer-readable medium such as a floppy disk drive, a hard disk drive, an optical disk drive, or a tape drive, a flash memory or other similar solid-state memory device, or an array of devices including devices included in a storage area network or other configuration. The information carrier may store instructions that, when executed by one or more processing devices (e.g., the processor 603), perform one or more methods, such as those described above.

[0076] Computing device 600 can optionally be linked through bus 609 to a display interface or user interface (HMI) 647 adapted to connect computing device 600 to a display device 649 and a keyboard 651, where display device 649 may include, among other devices, a computer monitor, a camera, a television, a projector, or a mobile device. In some implementations, computing device 600 can include a printer interface for connecting to a printing device, where printing device may include, among other devices, a liquid inkjet printer, a solid ink printer, a large scale commercial printer, a thermal printer, a UV printer, or a dye-sublimation printer.

[0077] The high-speed interface 611 manages operations requiring more bandwidth for the computing device 600, and the low-speed interface 613 manages operations requiring more lower bandwidth. Such allocation of functions is merely exemplary. In some implementations, the high-speed interface 611 can be coupled to memory 605, to a user interface (HMI) 647, to a keyboard 651 and a display 649 (e.g., through a graphics processor or accelerator), and to a high-speed expansion port 615. The high-speed expansion port 615 can accept various expansion cards via a bus 609. In some implementations, the low-speed interface 613 is coupled to a storage device 607 and a low-speed expansion port 617 via a bus 609. The low-speed expansion port 617, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input / output devices 641. The computing device 600 may be connected to a server 653 and a rack server 655. The computing device 600 may be implemented in several different forms. For example, the computing device 600 may be implemented as part of a rack server 655 .

[0078] 7 is a schematic diagram illustrating a use case for implementing the systems and methods of the present disclosure, showing a controller 702 configured to control an autonomous device 704 moving in a dynamic environment based on at least some point movement information determined by the motion sensor 706 of FIG.

[0079] The autonomous device 704 may be an autonomous or semi-autonomous vehicle for which the control quantities are generated using some embodiments. The autonomous vehicle may be equipped with a motion estimation system 120 as part of a motion sensor 706. In some embodiments, the motion sensor 706 is used to estimate the motion state of the autonomous device, and subsequently the control quantities are generated specifying values ​​for one or a combination of the steering angle of the vehicle's wheels and the rotational speed of the wheels, and the rotational speed of the vehicle and the acceleration of the vehicle.

[0080] The generated maneuvers are aimed at keeping the automated device 704 within certain boundaries of the road and maintaining safe movement.

[0081] 8 is a schematic diagram illustrating another use case for implementing the systems and methods of the present disclosure. The use case illustrates a traffic assistance system 804 operatively connected with the motion sensor 802 of FIG. 1. The traffic assistance system 804 includes a controller 806 configured to perform a control action based on at least some of the point motion information, the control action including one or a combination of traffic estimation, anomaly detection, generation of a pedestrian crossing warning message, and traffic light schedule adjustment.

[0082] 9 is a schematic diagram illustrating yet another use case for implementing the system and method of the present disclosure. The use case shows an automatic device 900 moving in a dynamic environment. The automatic device 900 includes the motion sensor 100 of FIG. 1 and an actuator controller 902 (hereinafter also referred to as actuator) configured to control the automatic device 900 based on the motion information of at least some points.

[0083] The actuator 902 may control a steering controller 904 and a brake / throttle controller 906 that control the rotation and acceleration of the automatic device 900. In such a case, the actuator 902 outputs control commands to the controllers 904 and 906 based on the amount of operation to control the motion state of the automatic device 900 (which may be a vehicle). In some embodiments, the controller also includes a higher-level controller, e.g., a lane-keep assist controller that further processes the control command of the actuator 902. In any case, the controller utilizes the output of the actuator controller 902, i.e., the control command for controlling at least one actuator of the vehicle 900, such as the steering wheel and / or brakes of the vehicle 900, to control the movement of the vehicle 900 according to the method and system implemented by the motion sensor 100.

[0084] FIG. 10 illustrates a method 1000 performed by the motion sensor 100 for estimating a motion state of an object, according to an embodiment of the disclosure.

[0085] A method 1000 is used to estimate motion information of at least some points of a dynamic environment. In step 1002, the method 1000 includes sensing the dynamic environment and collecting a time series of 3D (three dimensional) point clouds of the environment including a current 3D point cloud and a previous 3D point cloud. The sensing may be performed by a motion estimation system 120 shown in FIG. 1 by sensing the time series of 3D point cloud inputs 110 using a depth sensor.

[0086] Thereafter, in step 1004, the method 1000 includes iteratively estimating motion information for the series of 3D point clouds with a neural network, such as neural network 140, which has been described in detail in the previous embodiment and comprises an encoder, such as encoder 150, for providing a space-time coding for each point in each of the 3D clouds, and a decoder, such as decoder 160, for decoding the space-time coding to generate motion information for each point in each of the 3D point clouds in the input 110. Further, to encode a current point of the current 3D point cloud, the encoder 150 is configured to extract features of adjacent points in the current 3D point cloud located in the vicinity of a position of the current point to generate a current spatial coding of the current point in the current frame. Further, the encoder 150 is configured to extract features of adjacent points in the previous 3D point cloud located in the vicinity of a position in the previous 3D point cloud corresponding to the position of the current point to generate a previous spatial coding of the current point in the previous frame. In addition, the encoder 150 is configured to combine the current spatial coding and the previous spatial coding to generate a space-time coding of the current point. The neural network 140 also includes a decoder 160 that decodes the spatio-temporal coding to generate motion information for each point in the 3D cloud.

[0087] Thereafter, the method 1000 includes, at step 1006, outputting the estimated motion information. For example, this may be done by generating control signals for a controller associated with the motion estimation system to control objects in a dynamic environment. Figures 7, 8 and 9 show diagrams illustrating such objects and associated controllers that use a motion sensor 100 to implement the method 1000.

[0088] 11 is a schematic diagram illustrating a system 1100 that can be used to implement the systems and methods of the present disclosure. The system 1100 includes one or a combination of a transceiver 1101, an IMU (inertial measurement unit) 1103, a display 1105, a sensor(s) 1107, a memory 1109, and a processor 1111, which are operatively coupled together through a connection 1113. The connection 1113 may include a bus, a wire, a fiber, a link, or a combination thereof.

[0089] The transceiver 1101 may include, for example, a transmitter capable of transmitting one or more signals over one or more types of wireless communication networks and a receiver for receiving one or more signals transmitted over the one or more types of wireless communication networks. The transceiver 1101 may permit communication with wireless networks based on various technologies, such as, but not limited to, femtocells, which may be based on the IEEE 802.11 family of standards, Wi-Fi networks or Wireless Local Area Networks (WLANs), Bluetooth, Near Field Communication (NFC), Wireless Personal Area Networks (WPANs), such as networks based on the IEEE 802.15x family of standards, and / or Wireless Wide Area Networks (WWANs), such as LTE, WiMAX, etc. The system 1100 may also include one or more ports for communicating over wired networks.

[0090] In some embodiments, the system 1100 may include a depth sensor or image sensor, such as a CCD or CMOS sensor, a laser, and / or a camera, hereinafter referred to as "sensor 1107." For example, the sensor 1107 may convert an optical image into an electronic or digital image and send the resulting image to the processor 1111. Additionally or alternatively, the sensor 1107 may detect light reflected from objects of interest in the scene and send the intensity of the captured light to the processor 1111.

[0091] For example, the sensor 1107 may include a color or grayscale camera that provides "color information." The term "color information," as used herein, refers to color and / or grayscale information. Generally, as used herein, a color image or color information is viewed as including one to N channels, where N is an integer depending on the color space used to store the image. For example, an RGB image includes three channels, one for red, blue, and green information.

[0092] For example, the sensor 1107 may include a depth sensor to provide “depth information.” Depth information may be obtained in various ways using the depth sensor. The term “depth sensor” is used to refer to a functional unit that may be used to obtain depth information separately and / or in conjunction with other cameras. For example, in some embodiments, a depth sensor and an optical camera may be part of the sensor 1107. For example, in some embodiments, the sensor 1107 includes an RGBD camera that may capture per-pixel depth (D) information in addition to a color (RGB) image when the depth sensor is enabled.

[0093] As another example, in some embodiments, sensor 1107 may include a 3D Time Of Flight (3DTOF) camera. In embodiments having a 3DTOF camera, the depth sensor may take the form of a strobe light coupled to the 3DTOF camera that illuminates objects in the scene and the reflected light may be captured by a CCD / CMOS sensor in sensor 1107. Depth information may be obtained by measuring the time it takes for a pulse of light to travel to the object and return to the sensor.

[0094] As another example, the depth sensor can take the form of a light source coupled to the sensor 1107. In one embodiment, the light source projects a structured or organized light pattern, which may include one or more narrow bands of light, onto an object in the scene. Depth information is obtained by exploiting geometric distortions of the projected pattern caused by the surface shape of the object. One embodiment determines depth information from a stereo sensor, such as a combination of an infrared camera and an infrared structured light projector registered to an RGB camera.

[0095] In some embodiments, the sensor 1107 includes a stereoscopic camera. For example, the depth sensor may form part of a passive stereo vision sensor that may use two or more cameras to obtain depth information of a scene. Pixel coordinates of points in the captured scene that are common to both cameras may be used together with camera pose information and / or triangulation techniques to obtain per-pixel depth information.

[0096] In some embodiments, the system 1100 may be operatively connected to multiple sensors 1107, such as dual front cameras and / or front and rear cameras that may incorporate a variety of sensors. In some embodiments, the sensor 1107 may capture both still and moving images. In some embodiments, the sensor 1107 may include an RGBD or stereoscopic video camera capable of capturing images at, for example, 30 frames per second (fps). In one embodiment, the images captured by the sensor 1107 may be in a raw uncompressed format and may be compressed before being processed and / or stored in the memory 1109. In some embodiments, image compression may be performed by the processor 1111 using lossless or lossy compression techniques.

[0097] In some embodiments, the processor 1111 can also receive input from the IMU 1103. In other embodiments, the IMU 1103 can include a 3-axis accelerometer(s), a 3-axis gyroscope(s), and / or a magnetometer(s). The IMU 1103 can provide velocity, orientation, and / or other position related information to the processor 1111. In some embodiments, the IMU 1103 can output measurement information synchronized with the capture of each image frame by the sensor 1107. In some embodiments, a portion of the output of the IMU 1103 is used by the processor 1111 to fuse the sensor measurements and / or further process the fused measurements.

[0098] The system 1100 may also include a screen or display 1105 that depicts images, such as color and / or depth images. In some embodiments, the display 1105 may be used to display live images captured by the sensor 1107, fused images, AR (Augmented Reality) images, graphical user interfaces (GUIs), and other program output. In some embodiments, the display 1105 may include and / or be housed with a touch screen that allows a user to input data through a combination of virtual keyboards, icons, menus, or other GUIs, user gestures, and / or input devices such as styluses and other writing implements. In some embodiments, the display 1105 may be implemented using an LCD (Liquid Crystal Display) display or an LED (Light Emitting Diode) display, such as an OLED (Organic LED) display. In other embodiments, the display 1105 may be a wearable display. In some embodiments, the fusion results may be rendered on the display 1105 or sent to another application, which may be internal or external to the system 1100.

[0099] Additionally, the exemplary system 1100 may be modified in various ways consistent with this disclosure, such as by adding, combining, or removing one or more of the illustrated functional blocks. For example, in some configurations, the system 1100 does not include an IMU 1103 or a transceiver 1101. Furthermore, in certain exemplary implementations, the system 1100 includes various other sensors (not shown), such as an ambient light sensor, a microphone, an acoustic sensor, an ultrasonic sensor, a laser range finder, etc. In some embodiments, portions of the system 1100 take the form of one or more chipsets, or the like.

[0100] The processor 1111 may be implemented using a combination of hardware, firmware, and software. The processor 1111 may represent one or more circuits configurable to execute at least a portion of a computational procedure or process related to sensor fusion and / or a method for further processing the fused measurements. The processor 1111 reads instructions and / or data from the memory 1109. The processor 1111 may be implemented using one or more Application Specific Integrated Circuits (ASICs), CPUs and / or Central and / or Graphical Processing Units (GPUs), Digital Signal Processing Devices (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, embedded processor cores, electronics, other electronic units designed to perform the functions described herein, or combinations thereof.

[0101] Memory 1109 can be implemented within the processor 1111 and / or external to the processor 1111. As used herein, the term "memory" refers to any type of long-term, short-term, volatile, non-volatile, or other memory, and is not limited to a particular type or number of memories, or to a particular type of physical medium on which the memory is stored. In some embodiments, memory 1109 holds program code that facilitates automated parking.

[0102] For example, memory 1109 can store sensor measurements, such as still images, depth information, video frames, program results, and data provided by IMU 1103 and other sensors. Memory 1109 can store memory that stores vehicle geometry, maps of parking areas, kinematic models of the autonomous vehicle, and dynamic models of the autonomous vehicle. In general, memory 1109 can represent any data storage mechanism. Memory 1109 can include, for example, primary memory and / or secondary memory. Primary memory can include, for example, random access memory, read-only memory, and the like.

[0103] The secondary memory may include, for example, the same or similar type of memory as the primary memory and / or may include, for example, one or more data storage devices or one or more systems, such as, for example, a flash / USB memory drive, a memory card drive, a disk drive, an optical disk drive, a tape drive, a solid state drive, a hybrid drive, etc. In certain implementations, the secondary memory is operative to receive or is otherwise configurable to a non-transitory computer-readable medium in a removable media drive (not shown). In some embodiments, this non-transitory computer-readable medium forms part of the memory 1109 and / or the processor 1111.

[0104] The above description provides only exemplary embodiments and does not limit the scope, applicability, or configuration of the present disclosure. Rather, the description of the exemplary embodiments above provides one of ordinary skill in the art with an enabling description for implementing one or more exemplary embodiments. It is contemplated that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the subject matter as set forth in the appended claims.

[0105] Specific details have been set forth in the above description to provide a thorough understanding of the embodiments. However, those skilled in the art will appreciate that the embodiments may be practiced without these specific details. For example, systems, processes, and other elements of the disclosed subject matter may be illustrated in block diagrams as components so as not to obscure the embodiments in unnecessary detail. In other instances, well-known processes, structures, and techniques may be illustrated without unnecessary detail so as not to obscure the embodiments. Furthermore, like reference numbers and like symbols in the various drawings refer to like elements.

[0106] Also, each embodiment may be described as a process and illustrated as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may show operations as a sequential process, many of the operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process may terminate when its operations are completed, but may have additional steps not included in the drawings. Furthermore, not all operations included in the specifically described process may be performed in all embodiments. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, and the like. When a process corresponds to a function, the end of the function may correspond to the function returning to a calling function or a main function.

[0107] Moreover, at least some of the embodiments of the disclosed subject matter may be implemented manually or automatically. The manual or automatic implementation may be performed or at least assisted by the use of machine, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored on a machine-readable medium. A processor(s) may perform the necessary tasks.

[0108] The various methods or steps outlined herein may be coded as software executable on one or more processors employing any one of a variety of operating systems or platforms. Additionally, such software may be written using any number of suitable programming languages ​​and / or programming or scripting tools, or may be compiled as executable machine language code or intermediate code that runs on a framework or virtual machine. Typically the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0109] The embodiments of the present disclosure may be implemented as a method, examples of which have been provided. The operations performed as part of the method may be ordered as appropriate. Thus, embodiments may be constructed in which operations are performed in an order different from the illustrated order, and may include performing some operations simultaneously, even if shown as sequential operations in the illustrated embodiment.

[0110] Although the disclosure has been described by way of examples of certain preferred embodiments, it will be understood that various other adaptations and modifications may be made without departing from the spirit and scope of the disclosure, and it is therefore the object of the appended claims to cover all such variations and modifications that come within the spirit and scope of the disclosure.

Claims

1. A point cloud motion sensor for estimating motion information of at least a portion of points of an environment, comprising: a depth sensor configured to sense a dynamic environment and collect a time series of three-dimensional (3D) point clouds of the environment, the three-dimensional (3D) point cloud including a current 3D point cloud and a previous 3D point cloud; a motion estimator operatively connected to the depth sensor via one or a combination of a wired or wireless communication channel, the motion estimator comprising a processor and a memory storing instructions that, when executed by the processor, cause the motion estimator to iteratively process the sequence of 3D point clouds with a neural network, the neural network an encoder providing a space-time encoding of each point in each of said 3D clouds; a decoder for decoding the space-time coding to generate motion information for each point of each of the 3D clouds; To encode a current point of the current 3D point cloud, the encoder extracting features of adjacent points in the current 3D point cloud that are located near the location of the current point to generate a current spatial encoding of the current point in the current frame; extracting features of adjacent points in the previous 3D point cloud that are located near a position in the previous 3D point cloud that corresponds to a position of the current point to generate a previous spatial encoding of the current point in the previous frame; A point cloud motion sensor configured to combine the current spatial encoding and the previous spatial encoding to generate a space-time encoding of the current point.

2. A controller configured to control an automated device moving in the dynamic environment based on the motion information of the at least some of the points determined by the motion sensor of claim 1.

3. 1. An automated device moving in a dynamic environment, comprising: The motion sensor of claim 1; and an actuator configured to control the automated device based on the movement information of the at least some of the points.

4. 13. A traffic assistance system operatively connected to the motion sensor of claim 1, the traffic assistance system comprising a controller configured to execute a control action based on the motion information of the at least some of the points, the control action including one or a combination of traffic estimation, anomaly detection, generation of a pedestrian crossing warning message, and traffic light schedule adjustment.

5. The motion sensor of claim 1 , wherein the adjacent points in the current 3D point cloud and the adjacent points in the previous 3D point cloud are confined to a sphere centered at the position of the current point in the current 3D point cloud.

6. 6. The motion sensor of claim 5, wherein features of the adjacent points in the current 3D point cloud and the adjacent points in the previous 3D point cloud are extracted using a weighted kernel function that extracts the features for a plurality of proxy points different from the adjacent points in the corresponding spheres of the current 3D point cloud and the corresponding spheres of the previous 3D point cloud.

7. 6. The motion sensor of claim 5, wherein the features of the adjacent points in the current 3D point cloud and the features of the adjacent points in the previous 3D point cloud are combined by processing a combination of the extracted features of the current 3D point cloud and the extracted features of the previous 3D point cloud with a multi-layer perceptron.

8. The motion sensor of claim 5 , wherein the weights of the weighted kernel function are trainable parameters of the neural network shared by all points of the current point cloud.

9. 2. The motion sensor of claim 1, wherein the neural network includes a reduction branch that successively downsamples its input and an expansion branch that successively upsamples its input, the reduction branch including one or more pairs of the encoder and a downsampling layer, and the expansion branch including one or more pairs of the decoder and an upsampling layer.

10. 9. The motion sensor of claim 8, wherein the neural network includes at least one skip connection that enables an aggregation layer to fuse features from both one of the encoders and one of the decoders.

11. 9. The motion sensor of claim 8, wherein the reduction branch comprises a plurality of encoders, including a first encoder and a second encoder, arranged to process a downsampled output of the first encoder indicative of a current encoding of the spatio-temporal features of the current point cloud, the second encoder processing the current encoding together with a corresponding encoding of the previous point cloud determined during the previous iteration.

12. The reduction branch has a plurality of encoders arranged to process a downsampled output of the first encoder, the encoders including a first encoder and a second encoder, the encoders being configured to distinguish between the current point cloud and the previous point cloud based on an order of points in the series of point clouds, and the motion estimator is configured to: executing the first encoder to process the current point cloud and the previous point cloud to generate a current encoding; changing the order of the current points and the previous points in the series of points; Executing the first encoder and processing the reordering of the current point cloud and the previous point cloud to generate a previous encoding; The motion sensor of claim 8 , configured to execute the second encoder and process the current encoding and the previous encoding.

13. The motion sensor of claim 1 , wherein the neural network is a multi-headed network that processes the sequence of 3D point clouds and generates predicted motion vectors and estimated motion states for all 3D points in the current point cloud.

14. 2. The motion sensor of claim 1, wherein the neural network is a multi-head network that processes the series of 3D point clouds and generates semantic labels and estimated motion states for all 3D points in the current point cloud, the neural network including a segmentation neural network trained to extract semantic features from each 3D point cloud, a segmentation head trained to estimate the semantic labels, a point motion neural network trained to extract motion features based on the semantic features across multiple timestamps, and a motion state head trained to estimate the motion states.

15. 1. A computer-implemented method for estimating motion information of at least a portion of points of an environment, comprising: sensing a dynamic environment and collecting a time series of three-dimensional (3D) point clouds of the environment, including a current 3D point cloud and a previous 3D point cloud; and iteratively estimating the motion information of the sequence of 3D point clouds with a neural network, the neural network comprising: an encoder providing a space-time encoding of each point in each of said 3D clouds; a decoder for decoding the space-time coding to generate motion information for each point of the 3D image; To encode a current point of the current 3D point cloud, the encoder extracting features of adjacent points in the current 3D point cloud that are located near the location of the current point to generate a current spatial encoding of the current point in the current frame; extracting features of adjacent points in the previous 3D point cloud that are located near a position in the previous 3D point cloud that corresponds to a position of the current point to generate a previous spatial encoding of the current point in the previous frame; configured to combine the current spatial encoding and the previous spatial encoding to generate a space-time encoding of the current point; The computer-implemented method further comprising outputting the estimated motion information.

16. 16. The computer-implemented method of claim 15, wherein the adjacent points in the current 3D point cloud and the adjacent points in the previous 3D point cloud are constrained to a sphere centered at the position of the current point in the current 3D point cloud.

17. 17. The computer-implemented method of claim 16, wherein features of the adjacent points in the current 3D point cloud and the adjacent points in the previous 3D point cloud are extracted using a weighted kernel function that extracts the features for a number of proxy points that are different from the adjacent points in the corresponding spheres of the current 3D point cloud and the corresponding spheres of the previous 3D point cloud.

18. 17. The computer-implemented method of claim 16, wherein features of the adjacent points in the current 3D point cloud and features of the adjacent points in the previous 3D point cloud are combined by combining the extracted features of the current 3D point cloud with the extracted features of the previous 3D point cloud using a multi-layer perceptron.

19. 17. The computer-implemented method of claim 16, wherein the weights of the weighted kernel function are trainable parameters of the neural network shared by all points of the current point cloud.

20. 16. The computer-implemented method of claim 15, wherein the neural network includes a reduction branch that successively downsamples its input and an expansion branch that successively upsamples its input, the reduction branch including one or more pairs of the encoder and a downsampling layer, and the expansion branch including one or more pairs of the decoder and an upsampling layer.

21. 21. The computer-implemented method of claim 20, wherein the neural network includes at least one skip connection that enables an aggregation layer to fuse features from both one of the encoders and one of the decoders.

22. 21. The computer-implemented method of claim 20, wherein the reduction branch has a plurality of encoders, including a first encoder and a second encoder, arranged to process a downsampled output of the first encoder indicative of a current encoding of the spatio-temporal features of the current point cloud, the second encoder processing the current encoding together with a corresponding encoding of the previous point cloud determined during the previous iteration.

23. The reduction branch has a plurality of encoders arranged to process a downsampled output of the first encoder, the encoders including a first encoder and a second encoder, the encoders being configured to distinguish between the current point cloud and the previous point cloud based on an order of points in the series of point clouds, and the motion estimator is configured to: executing the first encoder to process the current point cloud and the previous point cloud to generate a current encoding; changing the order of the current points and the previous points in the series of points; Executing the first encoder and processing the reordering of the current point cloud and the previous point cloud to generate a previous encoding; 21. The computer-implemented method of claim 20, configured to execute the second encoder and process the current encoding and the previous encoding.

24. 16. The computer-implemented method of claim 15, wherein the neural network is a multi-headed network that processes the sequence of 3D point clouds and generates predicted motion vectors and estimated motion states for all 3D points in the current point cloud.

25. 16. The computer-implemented method of claim 15, wherein the neural network is a multi-head network that processes the series of 3D point clouds and generates semantic labels and estimated motion states for all 3D points in the current point cloud, the neural network including a segmentation neural network trained to extract semantic features from each 3D point cloud, a segmentation head trained to estimate the semantic labels, a point motion neural network trained to extract motion features based on the semantic features across multiple timestamps, and a motion state head trained to estimate the motion states.

Images