A convolutional neural network (CNN) is used to process a data stream comprising scans of spatial information provided by 2D or 3D sensors for measuring distances

Abstract

The invention relates to a device (1) for processing a data stream comprising scans of spatial information provided by a 2D or 3D sensor (2) for measuring distances, wherein the device (1) is configured to use a Convolutional Neural Network (CNN) (3) in an inference phase. The CNN comprises a first layer (L1) and one or more further layers (L2a, L2b) below the first layer (L1) and one or more first buffers (4a) for storing output tensors of a respective preceding layer (L1, L2). The device (1) is further configured to input data of a current scan provided by the 2D or 3D sensor (2) in form of a current tensor (epc(t i )) into the CNN (3), to perform one or more convolution operations in each further layer (L2a, L2b) depending on a current output tensor (a1(t i ), a2(t i )) of the preceding layer (L1, L2a) resulting from the data of the current scan and a preceding output tensor (a1(t i–1 ), a2(t i–1 )) of the preceding layer (L1, L2a). The preceding output tensor (a1(t i–1 ), a2(t i–1 )) of the preceding layer (L1, L2a) is the latest tensor stored in a respective first buffer (4a) of the one or more first buffers (4a) and results from data of a preceding scan input to the CNN immediately before the data of the current scan. Furthermore, the device (1) is configured to store the current output tensor (a1(t i ), (a2(t i )) of the preceding layer (L1, L2a) as the latest tensor in the respective first buffer (4a).

Description

TECHNICAL FIELD

[0001] The present invention relates to a device for processing a data stream comprising scans of spatial information provided by 2D or 3D sensors for measuring distances, wherein the device is for using a Convolutional Neural Network (CNN) in an inference phase. The present invention further relates to a ego-vehicle comprising the above-mentioned device and one or more 2D or 3D sensors for measuring distances. The present invention further relates to a hardware implementation of a Convolutional Neural Network (CNN) for processing a data stream comprising scans of spatial information provided by 2D or 3D sensors for measuring distances. Furthermore, the present invention relates to a method of processing a data stream comprising scans of spatial information provided by 2D or 3D sensors for measuring distances using a Convolutional Neural Network (CNN) in an inference phase. In addition, the present invention further relates to a computer program comprising program code for performing the above-mentioned method, a non-transitory storage medium storing executable program code for performing the above-mentioned method when executed by a processor, and a computer comprising a memory and a processor for storing and executing program code for performing the above-mentioned method. BACKGROUND

[0002] A data stream comprising scans of spatial information provided by 2D or 3D distance sensors can be processed by a neural network. The spatial information is spatial information about the environment or the vicinity of the 2D or 3D sensor. Objects in the environment of the 2D or 3D sensor can be detected depending on the processing result of processing the data stream. For example, the 2D or 3D sensor can be installed on an ego-vehicle. In this case, the spatial information provided by the 2D or 3D sensor corresponds to spatial information about the environment of the ego-vehicle, and thus, objects in the environment of the ego-vehicle, such as cars, pedestrians, cyclists, motorcyclists, etc., can be detected depending on the processing result of processing the data stream.

[0003] The term "ego-vehicle" refers to a vehicle equipped with one or more sensors, such as one or more 2D or 3D distance sensors, to sense the environment of the vehicle and to drive depending on the data from these sensors, not necessarily depending on any other data about its environment. In other words, the ego-vehicle drives depending on its own "observations" of its environment. Ego vehicle The term "ego-vehicle" refers to a vehicle equipped with one or more sensors, such as one or more 2D or 3D distance sensors, to sense the environment of the vehicle and to drive depending on the data from these sensors, not necessarily depending on any other data about its environment. In other words, the ego-vehicle drives depending on its own "observations" of its environment. SUMMARY

[0004] The present invention embodiments are further based on the following considerations made by the inventors:

[0005] A convolutional neural network (CNN) can be used to process a data stream of scans including spatial information provided by 2D or 3D sensors for measuring distance.

[0006] Figure 1 FIG. 1 shows an example of the operation performed by the convolutional neural network (CNN) 3 at the current time point ti.

[0007] The terms "present" and "current" can be used as synonyms. That is, for example, the terms "present time point" and "current time point" can be used as synonyms.

[0008] Figure 1 The provided CNN 3 includes four layers L1, L2a, L2b and L2c, wherein one or more convolution operations are performed in each layer.

[0009] At the current time t i The current scanned data pc(t i ) is provided by a 2D or 3D sensor and is derived from the current scan (at time t i The data of the scan provided on pc(t i ) and the current tensor from the previous scan (at time t i–1 The data of the scan provided on pc(t i–1 ), one or more convolution operations are performed in each layer of CNN 3, where the previously scanned data pc(t i–1 ) is input to CNN 3 immediately before the currently scanned data, and the current tensor and the previous tensor are provided to the corresponding layer.

[0010] One or more convolution operations in a layer are as follows Figure 1 The two arrows in derive from the corresponding two tensors on which one or more convolution operations are performed.

[0011] For example, Figure 1 As shown, at the current time point t i On the other hand, the 2D or 3D sensor provides the time t i The scanned data pc(t i ) as the data of the current scan, and the data pc(t i ) tensor epc(t i) is input to CNN 2 and then provided to the first layer L1. At the same time, the image obtained from the 2D or 3D sensor at the previous time point t i–1 Immediately after time point t i The data of the previous scan provided before the current scan pc(t i–1 )’s previous tensor epc(t i–1 ) is provided to the first layer L1, wherein, in the first layer L1, according to the current tensor epc(t i ) and the previous tensor epc(t i–1 ) performs one or more convolution operations of the first layer L1. In order to be able to i The previous tensor epc(t i–1 ) is used for one or more convolution operations in the first layer L1, which must be performed at the current time point t i Regenerate or recalculate the previous tensor epc(t i–1 The first layer L1 needs to be based on the previously scanned data pc(t i–1 ) performs voxelization (VX). Voxelization VX is as follows Figure 1 The single arrows in FIG. 5 are derived from the corresponding data on which the voxelization is performed.

[0012] However, when performing one or more convolution operations in a layer deeper or farther away from the first layer L1 in CNN 3 (e.g., the last layer L2c in CNN 3), in addition to the convolution operation at the current time point t i The corresponding current tensor provided to the corresponding layer (the current tensor epc(t provided to the first layer L1) i ), as the output tensor of the first layer L1 and provided to the current tensor a1(t i ), as the output tensor of the second layer L2a and provided to the current tensor a2(t i ), in addition to performing one or more convolution operations in each previous layer L1, L2a, and L2b, the following recomputation must also be performed:

[0013] – The data collected by the 2D or 3D sensor immediately after the time t must be recalculated i The data of the four subsequent previous scans provided before the current scan pc(t i–1 ), pc(t i–2 ), pc(t i–3 ) and pc(t i–4 )’s voxelized VX,

[0014] – The corresponding layers L1, L2a, L2b must be recalculated based on the data from the previous data pc(t i–1 ), pc(t i–2 ), pc(ti–3 ) and the previous tensors of pc(t i–4 ) to process the tensor epc(t i ) of data pc(t i ), a1(t i ), a2(t i ) and a3(t i ) originating from the current scan, thereby generating the output tensor a4(t Figure 1 ) of the CNN 3 as illustrated in Fig. 3. i

[0015] The terms “ Consecutive ” and “ consecutive ” can be used as synonyms. Subsequent directly subsequent

[0016] According to Figure 1 , the CNN 3 comprises four layers LI, L2a, L2b and L2c. This is just one example. According to one embodiment of the present application, the CNN 3 can comprise more than four layers. The more layers the CNN has, the more reprocessing or recalculation steps as described above have to be performed according to previous data of previous scans when using the CNN 3 for processing data of a current scan at a current point in time t i

[0017] Therefore, using a CNN for processing a data stream of scans comprising spatial information provided by a 2D or 3D sensor for measuring distances requires a large amount of computational resources. This is especially the case when the 2D or 3D sensor provides scans with a frame rate of at least 10 to 20 frames per second. The terms “ Frame ” and “ Scan ” can be used as synonyms.

[0018] In view of the above problems and drawbacks, it is an object of embodiments of the present application to improve the amount of computational resources required for processing a data stream of scans comprising spatial information provided by a 2D or 3D sensor for measuring distances. It is an object to reduce the amount of computational resources required for processing a data stream of scans comprising spatial information provided by a 2D or 3D sensor for measuring distances.

[0019] ​​​​The first aspect of the present application provides a device for processing a data stream comprising scans of spatial information provided by a 2D or 3D sensor for measuring distances, wherein the device is configured to use a Convolutional Neural Network (CNN) in an inference phase. The CNN comprises a first layer and one or more further layers below the first layer and one or more first buffers for storing an output tensor of a respective preceding layer. The device is configured to input a current scanned data provided by the 2D or 3D sensor in a form of a current tensor into the CNN. That is, the device is configured to input the current scanned data into the CNN in the form of the current tensor. Further, the device is configured to perform one or more convolution operations in each of the further layers depending on a current output tensor of the preceding layer resulting from the current scanned data and a preceding output tensor of the preceding layer. That is, the device is configured to perform one or more convolution operations in each of the further layers depending on the current output tensor of the preceding layer and the preceding output tensor of the preceding layer. The preceding output tensor of the preceding layer is the most recent tensor stored in a respective first buffer of the one or more first buffers and results from a preceding scanned data input into the CNN immediately before the current scanned data. Further, the device is configured to store the current output tensor of the preceding layer as the most recent tensor in the respective first buffer.

[0020] The device of the first aspect can reduce the amount of computational resources required for processing a data stream comprising scans of spatial information provided by a 2D or 3D sensor for measuring distances. Since the CNN comprises one or more first buffers for storing an output tensor of a respective preceding layer, the one or more convolution operations performed in each of the further layers at a current point in time do not require a recalculation of one or more preceding tensors, in particular output tensors, of the preceding layer. That is, the device is configured to perform one or more convolution operations in each of the further layers depending on a current output tensor of the preceding layer resulting from the current scanned data and a preceding output tensor of the preceding layer, wherein the preceding output tensor of the preceding layer is the most recent tensor stored in a respective first buffer. The use of the one or more first buffers reduces the inference time required by the CNN in the inference phase. The inference time corresponds to the time required by the CNN to provide output data (an output tensor of a last layer in the CNN) starting from the current tensor resulting from the current scanned data and input into the CNN at the current point in time. In particular, the computational cost can be reduced from K Figure 1 2 / 2 (i.e. O(K 2 ​) is reduced to O(K) in the case of a CNN including a buffer in the device provided by the first aspect. K is the number of aggregated scans (sweeps) provided by the 2D or 3D sensor. If the 2D or 3D sensor is a LiDAR sensor, K is the number of aggregated LiDAR scans. Thus, the device provided by the first aspect can perform real-time inference for large values of K (e.g. between 10 and 100).

[0021] The phrase “ Tensor input into the CNN ” can be understood as “ Tensor that is input into the CNN Tensor input to the CNN ”. The phrases “ Tensor input into the CNN ” and “ Layer Layer ” can be used synonymously.

[0022] In particular, the first layer and the one or more further layers in the CNN are all layers in the CNN. The description herein with respect to a layer referred to by the general term “ 2D sensor configured to measure distance ” applies to the first layer as well as to the one or more further layers.

[0023] In an implementation form of the first aspect, the CNN comprises one or more optional additional layers, and the device is configured to perform one or more convolution operations in each optional additional layer depending on a current output tensor of a previous layer of the CNN, the current output tensor being derived from data of the current scan, and a previous output tensor of the previous layer, the previous output tensor being derived from data of a previous scan, the data of the previous scan being input to the CNN immediately before the data of the current scan. In particular, the first layer, the one or more further layers and the one or more optional additional layers in the CNN are all layers in the CNN. The description herein with respect to a layer referred to by the general term “ 3D sensor configured to measure distance ” also applies to the one or more optional additional layers.

[0024] The 2D or 3D sensor for measuring distances can comprise or correspond to one or more light detection and ranging (LiDAR) sensors, time of flight (TOF) cameras, stereo cameras and / or beamforming radars. That is, the 2D or 3D sensor can comprise or correspond to one or more visual depth sensors. The 2D or 3D sensor can provide scans at a frame rate of at least 10 to 20 frames per second.

[0025] The term “ Buffer storage Rolling buffer ” can be understood as corresponding to a sensor for detecting a scan with two spatial dimensions (two-dimensional scan or frame) and measuring distances. The term “ Rolling buffer storage Activation of a layer" can be understood to correspond to a sensor for detecting a scan with three spatial dimensions (three-dimensional scan or frame) and measuring distances.

[0026] In particular, a CNN is a feed-forward neural network, which can be described using a directed acyclic graph (DAG). One advantage of a DAG neural network is that it has a finite impulse response operator and is related to a finite impulse response filter (FIR filter).

[0027] In particular, a buffer is used for buffering or storing data, such as tensors. A buffer can be a data structure for buffering data, in particular tensors. The terms Layer output "buffering", Buffer "buffering", Convolutional matrix "buffering" can be used to refer to a buffer.

[0028] In one implementation, in one or more of the one or more other layers of the CNN, one or more optional further operations, such as one or more normalization operations and / or one or more activation operations, can be performed in addition to the one or more convolution operations.

[0029] The terms Consecutive "output" or consecutive "output" can be used synonymously with an output tensor of a layer.

[0030] In one implementation of the first aspect, the CNN comprises a second buffer for storing tensors input to the CNN, and the device is configured to perform the one or more convolution operations in the first layer based on the current tensor and a previous tensor generating a current output tensor of the first layer of data originating from the current scan. The previous tensor is the most recent tensor stored in the second buffer and corresponds to data of the previous scan. Further, the device is configured to store the current tensor as the most recent tensor in the second buffer.

[0031] Thus, the device provided by the first aspect can reduce the amount of computational resources required for processing a data stream comprising scans of spatial information provided by a 2D or 3D sensor for measuring distances. Since the CNN comprises a second buffer for storing tensors input to the CNN, the one or more convolution operations performed in the first layer at a current point in time do not require the recalculation of one or more previous tensors input to the CNN based on corresponding data of a corresponding previous scan. That is, the device is configured to perform the one or more convolution operations in the first layer based on the current tensor and the previous tensor, wherein the previous tensor is the most recent tensor stored in the second buffer.

[0032] Herein, the general term "Subsequent The description of the buffer zone referred to in " applies to one or more first buffer zones and the second buffer zone.

[0033] Specifically, the number of first buffers in the CNN corresponds to the number of other layers in the CNN. Specifically, the number of buffers in the CNN corresponds to the number of one or more first buffers and second buffers.

[0034] In an implementation of the first aspect, each buffer is a serial-in and parallel-out buffer, configured to: store a new tensor as the latest tensor; and simultaneously discard the oldest tensor stored in the buffer if the buffer is full.

[0035] Specifically, each buffer is a serial-in parallel-out (SIPO) shift register.

[0036] In an implementation of the first aspect, the time size of each buffer in the one or more first buffers for storing the output tensor of the corresponding previous layer is 1 less than the time size of the convolution kernel of the corresponding previous layer.

[0037] Specifically, the one or more buffers have different temporal sizes.

[0038] the term" directly subsequent " can be used to refer to a convolution kernel. A convolution kernel is used in a layer of a CNN to perform one or more convolution operations in that layer.

[0039] The temporal size of the first buffer corresponds to the number of consecutive or subsequent time points when the first buffer is used to store the output tensor of the corresponding layer (previous layer). Therefore, the temporal size can also be defined according to the number of subsequent scans of the 2D or 3D sensor when the first buffer can store the output tensor of the corresponding layer (previous layer). For example, if the temporal size of the first buffer is 1 (" 1 ”), the first buffer can only store one output tensor of the corresponding layer's data originating from one scan at one time point. This minimizes storage consumption (e.g., RAM consumption). Therefore, in this case, when the current output tensor (data originating from the current scan at the current time point) of a layer (previous layer) is stored as the latest tensor in the corresponding first buffer, since the first buffer is full, the previous output tensor (data originating from the previous scan input to the CNN immediately before the data of the current scan) of this layer that has already been stored in the corresponding first buffer is discarded or deleted. For example, if the time size of the first buffer is 3 (“ 3 ”), the first buffer can store three output tensors of the corresponding layer (previous layer) originating from three consecutive scans of data at three consecutive time points.

[0040] Thus, the temporal size of the buffer corresponds to the number of subsequent time points at which the buffer can store a tensor. Thus, the temporal size of the buffer corresponds to the number of subsequent scans of the 2D or 3D sensor at which the first buffer can store a corresponding tensor.

[0041] The term “tensor” can be used synonymously with the terms “data tensor” and “tensor data”. Ego-vehicle The terms “tensor data” and “data tensor” can be used synonymously. Local navigational frame The terms “tensor” and “tensor data” can be used synonymously. Figure 7 The terms “tensor” and “tensor data” can be used synonymously. Local navigational coordinate frame The terms “tensor” and “tensor data” can be used synonymously.

[0042] In an implementation form of the first aspect, each tensor has two or more dimensions.

[0043] In particular, each tensor has one or more spatial dimensions and one channel dimension. The number of spatial dimensions of a tensor can be equal to the spatial dimensions of the data of the corresponding scan provided by the 2D or 3D sensor used to measure the distance at which the tensor starts.

[0044] The channel dimension corresponds to the number of channels and is greater than or equal to one (channel dimension 1).

[0045] In particular, the data of a scan (e.g. the current scan) corresponds to a point cloud, wherein a point cloud is a set of points in an N-dimensional space and their attributes, N corresponding to the number of spatial dimensions. In particular, the number of spatial dimensions N can correspond to one spatial dimension, two spatial dimensions or three spatial dimensions.

[0046] For example, if the data of a scan (e.g. the current scan) corresponds to a point cloud having two spatial dimensions, the data having two spatial dimensions can correspond to a bird’s eye view (BEV) representation of the point cloud. If the data of a scan (e.g. the current scan) corresponds to a point cloud having three spatial dimensions, the data having three spatial dimensions can correspond to a volumetric representation of the point cloud, e.g. for a flying vehicle, e.g. a flying robot, a flying drone, an airplane, etc.

[0047] For example, a point cloud having three spatial dimensions can be produced by a 2D or 3D sensor (e.g. a LiDAR sensor) used to measure distances, wherein a wave or beam (e.g. a light beam) is reflected back from an obstacle in the environment of the 2D or 3D sensor, producing a point having three spatial dimensions and a scalar reflected brightness attribute, the position of the point being in meters. Such a point cloud comprising a plurality (e.g. thousands) of points having three spatial dimensions and a scalar reflected brightness attribute can correspond to a tensor having four dimensions. The four dimensions correspond to the three spatial dimensions and one channel dimension for the scalar reflected brightness attribute.

[0048] In an implementation form of the first aspect, each tensor has two dimensions, in particular one spatial dimension and one channel dimension. Optionally, each tensor has three dimensions, in particular two spatial dimensions and one channel dimension. Optionally, each tensor has four dimensions, in particular three spatial dimensions and one channel dimension.

[0049] In an implementation form of the first aspect, the channel dimension of each first buffer for storing an output tensor of a respective previous layer corresponds to the channel dimension of the respective previous layer.

[0050] In an implementation form of the first aspect, the device is configured to generate output data for a navigation process of a ego vehicle in which the 2D or 3D sensor is arranged, the device is configured to: input current position data of the ego vehicle in a grid of a local navigation coordinate system together with the data of the current scan to the CNN; fill and crop the stored latest tensors in each buffer if the current position data of the ego vehicle does not match the previous position data input together with the data of the previous scan.

[0051] The term “ego vehicle” can be understood as a mobile platform, in particular a mobile robot platform, carrying one or more sensors (e.g. one or more 2D or 3D sensors for measuring distances), such a mobile platform performing calculations or operations from the perspective of perceiving the world or environment. Ego-vehicle

[0052] The ego vehicle can correspond to:

[0053] a vehicle, e.g. a car, a truck, a motorcycle, etc.;

[0054] an autonomous vehicle, e.g. an autonomous car, an autonomous truck, etc.;

[0055] a robot, e.g. a delivery robot;

[0056] an autonomous robot, e.g. an autonomous delivery robot;

[0057] a flying vehicle, e.g. a flying drone, a flying robot, an airplane, etc.; or

[0058] an autonomous flying vehicle, e.g. an autonomous flying drone, an autonomous flying robot, an autonomous airplane, etc.

[0059] ​The ego vehicle may include a positioning unit for determining the current position of the ego vehicle, thereby determining the current position data of the ego vehicle. The positioning unit may be configured for short-term positioning, such as positioning within a range of 1 second to 10 seconds. The positioning unit may include or correspond to one or more inertial measurement units (IMUs). Specifically, the positioning unit is used to perform an odometer measurement process, such as an inertial odometer measurement process, a wheel odometer measurement process, and / or an optical or visual odometer measurement process. The odometer measurement process is a process of understanding the position of the ego vehicle (i.e., the position of the ego vehicle) based on sensory (e.g., wheel, inertial) information.

[0060] the term" 2D sensor configured to measure distance " can be understood as a coordinate system connected to the ground. Specifically, the local navigation coordinate system can correspond to a two-dimensional coordinate system with a top-down view on the ground (such as 3D sensor configured to measure distance shown).

[0061] Specifically, the grid of the local navigation coordinate system is a regular grid including a plurality of cells.

[0062] the term" Figure 1 " and local navigation coordinate system can be used as synonyms.

[0063] Specifically, the device is configured to: if the current position data of the ego vehicle does not match the previous position data input together with the previously scanned data, zero-fill and crop the latest tensor stored in each buffer.

[0064] Specifically, the local navigation coordinate system grid is a regular grid consisting of multiple cells, and if the ego vehicle moves within the same cell of the grid, the device does not perform padding or clipping. In other words, if the ego vehicle's current position data does not match the previous position data input with the previously scanned data, but the ego vehicle's current position data and the previous position data are or are within the same cell of the grid, the device does not perform padding or clipping.

[0065] In an implementation of the first aspect, the device is used to control the autonomous movement of the vehicle by processing the data stream.

[0066] In one implementation of the first aspect, the device is used to: store the position data in a positioning field of one or more buffers in the one or more first buffers and the second buffer; if the current position data of the vehicle does not match the previous position data, use the current position data to update the positioning field of the corresponding one or more buffers.

[0067] In particular, the grid of the local navigation coordinate system is a regular grid comprising a plurality of cells, and the positioning field of the respective buffer(s) is not updated if the ego vehicle moves within the same cell of the grid. In other words, if the current position data of the ego vehicle does not match the previous position data entered together with the previously scanned data, but the current position data of the ego vehicle and the previous position data are within or located in the same cell of the grid, the device does not update the positioning field of the respective buffer(s) using the current position data.

[0068] In an implementation form of the first aspect, the device is configured to generate, from the currently scanned data and the current position data of the ego vehicle in the grid of the local navigation coordinate system, the current tensor input to the CNN by transforming coordinates of the currently scanned data into the local navigation coordinate system.

[0069] In an implementation form of the first aspect, the device is configured to generate, from the currently scanned data and the current position data of the ego vehicle in the grid of the local navigation coordinate system, the current tensor input to the CNN by additionally performing voxelization.

[0070] In an implementation form of the first aspect, the device is configured to generate, from the currently scanned data, the current tensor input to the CNN by performing voxelization.

[0071] In particular, the currently scanned data corresponds to a point cloud, wherein the point cloud is a set of points and their attributes in an N-dimensional space, N corresponding to the number of spatial dimensions. Thus, the above-described device can be configured to generate, from the currently scanned point cloud and the current position data of the ego vehicle in the grid of the local navigation coordinate system, the current tensor input to the CNN by transforming coordinates of the currently scanned point cloud into the local navigation coordinate system.

[0072] The current tensor input to the CNN can correspond to an encoded point cloud. The encoded point cloud can be understood as a result of transforming an original or unordered point cloud into a voxelized or ordered point cloud.

[0073] Voxelization is a transformation of a set of unordered points (e.g. unordered points) of a point cloud into a regular grid. In particular, voxelization is a transformation of a set of unordered points having N spatial dimensions (e.g. unordered points in a point cloud having N spatial dimensions) into a regular N-dimensional grid. Voxelization can be performed by pillar encoding or voxel feature encoding. In particular, the number N of spatial dimensions can correspond to one spatial dimension, two spatial dimensions or three spatial dimensions.

[0074] In an implementation form of the first aspect, the device is configured to detect, based on the output tensor of the last one of the one or more further layers of the CNN, an object, in particular a moving object, in a vicinity of the 2D or 3D sensor. Optionally or in addition, the device is configured to perform, based on the output tensor of the last one of the one or more further layers of the CNN, a point cloud semantic segmentation. Optionally or in addition, the device is configured to perform, based on the output tensor of the last one of the one or more further layers of the CNN, a free space estimation.

[0075] In particular, the device is configured to detect, based on the output tensor of the last one of the one or more further layers of the CNN, a weak target object in a vicinity of the 2D or 3D sensor. A weak target object is an object with less spatial information but important temporal information. In particular, a weak target object is a moving object that can be detected by the 2D or 3D sensor, such as a car, a pedestrian including a child, a cyclist including a child, a motorcyclist, etc., which has a small number of points, in particular between 1 and 5 points, more particularly between 1 and 10 points. For example, if the 2D or 3D sensor is a LiDAR sensor, a weak target object is an object with a small number of LiDAR return points, in particular between 1 and 5 LiDAR return points, more particularly between 1 and 10 LiDAR return points. In the case of fog, rain and snow, and severe occlusion, objects that are usually detectable or visible by the 2D or 3D sensor, such as a LiDAR sensor, can also become weak target objects.

[0076] In an implementation form of the first aspect, if the 2D or 3D sensor is a LiDAR sensor, a weak target object can be defined as a moving object in a known object class (category) list, etc., with 1 to 5 (in particular 1 to 10) LiDAR return points falling into it.

[0077] For implementing the device of the first aspect of the present application, the above-mentioned implementation forms and optional features of the first aspect, in part or all, can be combined with each other.

[0078] A second aspect of the present application provides a hardware implementation of a Convolutional Neural Network (CNN) for processing a data stream of scans comprising spatial information provided by a 2D or 3D sensor for measuring distances. The CNN comprises a first layer and at least one or more further layers below the first layer and one or more first buffers for storing an output tensor of a respective preceding layer. The hardware implementation of the CNN is configured to input data of a current scan provided by the 2D or 3D sensor in form of a current tensor into the CNN. Further, the hardware implementation of the CNN is configured to perform one or more convolution operations in each further layer depending on a current output tensor of the preceding layer resulting from the data of the current scan and a preceding output tensor of the preceding layer. The preceding output tensor of the preceding layer is the most recent tensor stored in a respective first buffer of the one or more first buffers and results from data of a preceding scan input into the CNN immediately before the data of the current scan. The hardware implementation of the CNN is configured to store the current output tensor of the preceding layer as the most recent tensor in the respective first buffer.

[0079] In an implementation form of the second aspect, the CNN comprises one or more optional additional layers, and the hardware implementation of the CNN is configured to perform one or more convolution operations in each optional additional layer depending on a current output tensor of a preceding layer resulting from the data of the current scan and a preceding output tensor of the preceding layer resulting from data of a preceding scan input into the CNN immediately before the data of the current scan. In particular, the first layer, the one or more further layers and the one or more optional additional layers in the CNN are all layers in the CNN.

[0080] In an implementation form of the second aspect, the CNN comprises a second buffer for storing tensors input into the CNN, and the hardware implementation of the CNN is configured to perform one or more convolution operations in the first layer depending on the current tensor and a preceding tensor generating the current output tensor of the first layer resulting from the data of the current scan. The preceding tensor is the most recent tensor stored in the second buffer and corresponds to the preceding scan data. Further, the hardware implementation of the CNN is configured to store the current tensor as the most recent tensor in the second buffer.

[0081] In an implementation form of the second aspect, each buffer is a serial push-pop buffer configured to store a new tensor as the most recent tensor and, if the buffer is full, to discard simultaneously the oldest tensor stored in the buffer.

[0082] In an implementation form of the second aspect, a time size of each of the one or more first buffers for storing an output tensor of a respective previous layer is smaller than a time size of a convolution kernel of the respective previous layer by one.

[0083] In particular, the one or more buffers have different time sizes.

[0084] In an implementation form of the second aspect, each tensor has two or more dimensions.

[0085] In particular, each tensor has one or more spatial dimensions and one channel dimension.

[0086] In an implementation form of the second aspect, each tensor has two dimensions, in particular one spatial dimension and one channel dimension. Alternatively, each tensor has three dimensions, in particular two spatial dimensions and one channel dimension. Alternatively, each tensor has four dimensions, in particular three spatial dimensions and one channel dimension.

[0087] In an implementation form of the second aspect, a channel dimension of each first buffer for storing an output tensor of a respective previous layer corresponds to a channel dimension of the respective previous layer.

[0088] In an implementation form of the second aspect, the hardware implementation of the CNN is configured to generate output data for a navigation process of a ego vehicle in which the 2D or 3D sensor is arranged, the hardware implementation of the CNN being configured to: input current position data of the ego vehicle in a grid of a local navigation coordinate system together with the data of the current scan to the CNN; if the current position data of the ego vehicle does not match previous position data input together with the data of the previous scan, pad and crop the stored latest tensor in each buffer.

[0089] In an implementation form of the second aspect, the hardware implementation of the CNN is configured to control an autonomous movement of the ego vehicle by processing the data stream.

[0090] In an implementation form of the second aspect, the hardware implementation of the CNN is configured to: store the position data in a positioning field of one or more of the one or more first buffers and the second buffer; if the current position data of the ego vehicle does not match the previous position data, update the positioning field of the respective one or more buffers with the current position data.

[0091] In an implementation form of the second aspect, the hardware implementation of the CNN is configured to generate the current tensor input to the CNN by transforming coordinates of the data of the current scan into a local navigation coordinate system according to the data of the current scan and current position data of the ego vehicle in a grid of the local navigation coordinate system.

[0092] In an implementation form of the second aspect, the hardware implementation of the CNN is configured to generate the current tensor input to the CNN by additionally performing voxelization according to the data of the current scan and current position data of the ego vehicle in a grid of a local navigation coordinate system.

[0093] In an implementation form of the second aspect, the hardware implementation of the CNN is configured to detect objects, in particular moving objects, in a vicinity of the 2D or 3D sensor according to an output tensor of a last one of the one or more further layers of the CNN. Optionally or additionally, the hardware implementation of the CNN is configured to perform point cloud semantic segmentation according to an output tensor of a last one of the one or more further layers of the CNN. Optionally or additionally, the hardware implementation of the CNN is configured to perform free space estimation according to an output tensor of a last one of the one or more further layers of the CNN.

[0094] The hardware implementation of the CNN of the second aspect and implementation forms and optional features thereof achieve the same advantages as the device of the first aspect and corresponding implementation forms and optional features thereof.

[0095] The implementation forms and optional features of the device provided by the first aspect correspondingly apply to the hardware implementation of the CNN provided by the second aspect.

[0096] For implementing the hardware implementation of the CNN provided by the second aspect of the present application, the implementation forms and optional features of the second aspect described above can be combined with each other.

[0097] A third aspect of the present application provides an ego vehicle. The ego vehicle comprises one or more 2D or 3D sensors for measuring distances and a device provided by the first aspect or any implementation form thereof. The one or more 2D or 3D sensors are configured to provide a scan comprising spatial information of a vicinity of the ego vehicle to the device in a form of a data stream, and the device is configured to process the data stream.

[0098] The term “ego vehicle” can be understood as a mobile platform, in particular a mobile robot platform, carrying one or more sensors (e.g. one or more 2D or 3D sensors for measuring distances), such a mobile platform performing calculations or operations from a perspective of perceiving the world or environment. Figure 2 The term “ego vehicle” can be understood as a mobile platform, in particular a mobile robot platform, carrying one or more sensors (e.g. one or more 2D or 3D sensors for measuring distances), such a mobile platform performing calculations or operations from a perspective of perceiving the world or environment.

[0099] The ego car can correspond to:

[0100] – Vehicles, such as cars, trucks, motorcycles, etc.;

[0101] – Autonomous vehicles, such as self-driving cars and trucks;

[0102] – Robots, such as delivery robots;

[0103] – Self-driving robots, such as self-driving delivery robots;

[0104] – flying vehicles, such as flying drones, flying robots, airplanes, etc.; or

[0105] – Autonomous flying vehicles, such as autonomous flying drones, autonomous flying robots, autonomous airplanes, etc.

[0106] The ego vehicle may include a positioning unit for determining the ego vehicle's current position, thereby determining the ego vehicle's current position data. The positioning unit may be configured for short-term positioning, such as positioning within a range of 1 to 10 seconds. The positioning unit may include or correspond to one or more inertial measurement units (IMUs). Specifically, the positioning unit is configured to perform odometry processes, such as inertial odometry processes, wheel odometry processes, and / or optical or visual odometry processes.

[0107] The one or more 2D or 3D sensors for measuring distance may include or correspond to one or more light detection and ranging (LiDAR) sensors, time of flight (TOF) cameras, stereo cameras, and / or beamforming radars. In other words, the one or more 2D or 3D sensors may include or correspond to one or more visual depth sensors. The 2D or 3D sensors may provide scanning at a frame rate of at least 10 to 20 frames per second.

[0108] the term" Figure 3A Figure 3B " can be understood as corresponding to a sensor for detecting a scan with two spatial dimensions (a two-dimensional scan or frame) and measuring distance. The term " Figure 4 Figure 5 ” can be understood as corresponding to a sensor for detecting a scan with three spatial dimensions (3D scan or frame) and measuring distances.

[0109] In an implementation of the third aspect, the device is used to control the autonomous movement of the vehicle by processing the data stream.

[0110] The self-driving vehicle of the third aspect and its implementation forms and optional features achieve the same advantages as the device of the first aspect and its corresponding implementation forms and optional features.

[0111] In order to achieve the self-driving vehicle of the third aspect of the present application, part or all of the implementation forms and optional features of the third aspect described above can be combined with each other.

[0112] The fourth aspect of the present application provides a method for processing a data stream of scans comprising spatial information provided by a 2D or 3D sensor for measuring distances using a Convolutional Neural Network (CNN) in an inference phase. The CNN comprises a first layer and at least one or more further layers below the first layer and one or more first buffers for storing an output tensor of a respective preceding layer. The method comprises the following steps:

[0113] - inputting data of a current scan provided by the 2D or 3D sensor in form of a current tensor into the CNN;

[0114] - performing one or more convolution operations in each further layer depending on a current output tensor of the preceding layer resulting from the data of the current scan and a previous output tensor of the preceding layer, wherein

[0115] the previous output tensor of the preceding layer is the latest tensor stored in a respective first buffer of the one or more first buffers and results from data of a preceding scan inputted into the CNN immediately before the data of the current scan;

[0116] - storing the current output tensor of the preceding layer as the latest tensor in the respective first buffer.

[0117] In an implementation form of the fourth aspect, the CNN comprises one or more optional additional layers, and the method comprises the following steps:

[0118] - performing one or more convolution operations in each optional additional layer depending on a current output tensor of the preceding layer resulting from the data of the current scan and a previous output tensor of the preceding layer resulting from data of a preceding scan inputted into the CNN immediately before the data of the current scan.

[0119] In particular, the first layer, the one or more further layers and the one or more optional additional layers in the CNN are all layers in the CNN.

[0120] In an implementation form of the fourth aspect, the CNN comprises a second buffer for storing tensors input to the CNN, and the method comprises the steps of:

[0121] performing one or more convolution operations in the first layer according to the current tensor and a previous tensor generating a current output tensor of the first layer from data of the current scan, wherein the previous tensor is the most recent tensor stored in the second buffer and corresponds to data of the previous scan;

[0122] storing the current tensor as the most recent tensor in the second buffer.

[0123] In an implementation form of the fourth aspect, each buffer is a ring-in and ring-out buffer, and the method comprises the steps of:

[0124] storing a new tensor as the most recent tensor;

[0125] if the buffer is full, discarding the oldest tensor stored in the buffer at the same time.

[0126] In an implementation form of the fourth aspect, each buffer of the one or more first buffers for storing output tensors of a respective previous layer has a time size which is one less than a time size of a convolution kernel of the respective previous layer.

[0127] In particular, the one or more buffers have different time sizes.

[0128] In an implementation form of the fourth aspect, each tensor has two or more dimensions.

[0129] In particular, each tensor has one or more spatial dimensions and one channel dimension.

[0130] In an implementation form of the fourth aspect, each tensor has two dimensions, in particular one spatial dimension and one channel dimension. Optionally, each tensor has three dimensions, in particular two spatial dimensions and one channel dimension. Optionally, each tensor has four dimensions, in particular three spatial dimensions and one channel dimension.

[0131] In an implementation form of the fourth aspect, each first buffer for storing output tensors of a respective previous layer has a channel dimension corresponding to a channel dimension of the respective previous layer.

[0132] In an implementation form of the fourth aspect, the output data is generated for a navigation process of a ego vehicle in which the 2D or 3D sensor is arranged, and the method comprises the steps of:

[0133] - inputting current position data of the ego vehicle in a grid of a local navigation coordinate system together with the data of the current scan into the CNN;

[0134] - filling and cropping the stored latest tensor in each buffer if the current position data of the ego vehicle does not match the previous position data input together with the data of the previous scan.

[0135] In an implementation form of the fourth aspect, the method comprises the following step:

[0136] - controlling autonomous motion of the ego vehicle by processing the data stream.

[0137] In an implementation form of the fourth aspect, the method comprises the following step:

[0138] - storing the position data in a positioning field of one or more of the one or more first buffers and the second buffer.

[0139] - updating the positioning field of the respective one or more buffers with current position data of the ego vehicle if the current position data of the ego vehicle does not match the previous position data.

[0140] In an implementation form of the fourth aspect, the method comprises the following step:

[0141] - generating the current tensor input to the CNN by transforming coordinates of the data of the current scan into a local navigation coordinate system according to the data of the current scan and the current position data of the ego vehicle in a grid of the local navigation coordinate system.

[0142] In an implementation form of the fourth aspect, the method comprises the following step:

[0143] - generating the current tensor input to the CNN by additionally performing voxelization according to the data of the current scan and the current position data of the ego vehicle in a grid of a local navigation coordinate system.

[0144] In an implementation form of the fourth aspect, the method comprises the following step: detecting objects, in particular moving objects, in a vicinity of the 2D or 3D sensor according to an output tensor of a last one of the one or more further layers of the CNN. Optionally or additionally, the method comprises the following step: performing point cloud semantic segmentation according to an output tensor of a last one of the one or more further layers of the CNN. Optionally or additionally, the method comprises the following step: performing free space estimation according to an output tensor of a last one of the one or more further layers of the CNN.

[0145] The method of the fourth aspect and implementation forms and optional features thereof achieve the same advantages as the device of the first aspect and corresponding implementation forms and optional features thereof.

[0146] The implementation forms and optional features of the device provided by the first aspect apply correspondingly to the method provided by the fourth aspect.

[0147] In order to implement the method provided by the fourth aspect of the application, some or all of the implementation forms and optional features of the fourth aspect described above can be combined with each other.

[0148] The fifth aspect of the application provides a computer program comprising program code. The program code is used to execute the method provided by the fourth aspect or any implementation form thereof.

[0149] In particular, the fifth aspect of the application provides a computer program comprising program code. When the program code is implemented on a processor, it is used to execute the method provided by the fourth aspect or any implementation form thereof.

[0150] The sixth aspect of the application provides a computer program comprising program code. When the program code is implemented on a processor, it is used to execute the method provided by the fifth aspect or any implementation form thereof.

[0151] The seventh aspect of the application provides a non-transitory storage medium storing executable program code. When the executable program code is executed by a processor, the method provided by the fourth aspect or any implementation form thereof is executed.

[0152] The eighth aspect of the application provides a computer. The computer comprises a memory and a processor, and the memory and the processor are used to store and execute program code to execute the method provided by the fourth aspect or any implementation form thereof.

[0153] The memory can be distributed on multiple physical devices. Multiple processors that cooperate when executing the program code can be referred to as a kind of processor.

[0154] It should be noted that all the devices, elements, units and modules described in the present application can be implemented in software or hardware elements or any type of combination thereof. All the steps performed by various entities described in the present application and the functions described by various entities are intended to indicate that the corresponding entity is used to perform the corresponding steps and functions. Although in the description of the following specific embodiments, the specific functions or steps performed by external entities are not reflected in the description of the specific detailed elements of the entities performing the specific steps or functions, but those skilled in the art should know that these methods and functions can be realized by corresponding hardware or software elements or any combination thereof. BRIEF DESCRIPTION OF DRAWINGS

[0155] The following description of specific embodiments will illustrate the various aspects and implementations described above in connection with the accompanying drawings.

[0156] Figure 6 An example of the operation of a Convolutional Neural Network (CNN) at a current time point t i is shown.

[0157] Figure 7 An apparatus provided by one embodiment of the application and a self-driving vehicle provided by one embodiment of the application are shown.

[0158] Figure 8 An example of the operation of a Convolutional Neural Network (CNN) at a current time point t i is shown.

[0159] Figure 9 An example of the operation of a Convolutional Neural Network (CNN) at a current time point t i is shown.

[0160] Figures 1 to 1 An example of the operation of a Convolutional Neural Network (CNN) at a current time point t i is shown.

[0161] Figure 2 An example of the operation of a Convolutional Neural Network (CNN) at a current time point t

[0162] Figure 2 An example of the operation of a Convolutional Neural Network (CNN) at a current time point t

[0163] Figure 2 A grid of a local navigation coordinate system and positions of a self-driving vehicle provided by one embodiment of the application at two subsequent time points are shown.

[0164] Figure 2is a schematic diagram of a method for storing a tensor of data of a current scan originating from a current point in time ti in a buffer of a Convolutional Neural Network, CNN, according to an embodiment of the invention

[0165] Figure 2 is a schematic diagram of a method for processing a data stream of a scan of spatial information provided by a 2D or 3D sensor for measuring distances using a Convolutional Neural Network, CNN, in an inference phase according to an embodiment of the invention. DETAILED DESCRIPTION

[0166] In Figure 3A 0, corresponding elements are labeled with the same reference signs.

[0167] In Figure 3B , the left side shows a device 1 according to an embodiment of the invention and the right side shows a self-driving car 5 according to an embodiment of the invention.

[0168] The above description of the device according to the first aspect and its implementation forms applies correspondingly to the device 1 in Figure 4 . The above description of the self-driving car according to the third aspect and its implementation forms applies correspondingly to the self-driving car 5 in Figure 5 .

[0169] As shown on the left side of Figure 6 , the device 1 comprises a Convolutional Neural Network, CNN, 3 and is configured to use the CNN in an inference phase. The device 1 is configured to receive and process a data stream comprising a scan of spatial information provided by a 2D or 3D sensor 2 for measuring distances.

[0170] The CNN 3 in the device 1 comprises a first layer and one or more further layers below the first layer (not shown in Figure 2 ). The CNN 3 in the device 1 further comprises one or more first buffers 4a for storing output tensors of the respective preceding layer.

[0171] The device 1 is configured to:

[0172] - input data of a current scan provided by the 2D or 3D sensor 2 in form of a current tensor into the CNN 3;

[0173] - perform one or more convolution operations in each of the further layers of the CNN 3 depending on a current output tensor of the preceding layer originating from the data of the current scan and a previous output tensor of the preceding layer, wherein

[0174] The previous output tensor of the previous layer is the latest tensor stored in the corresponding first buffer 4a in the one or more first buffers 4a, and is derived from the previously scanned data input to the CNN 3 immediately before the currently scanned data;

[0175] – Store the current output tensor of the previous layer as the latest tensor in the corresponding first buffer 4a.

[0176] Figure 2 、 Figure 2 、 Figure 3A 、 Figure 3A and Figure 3A An embodiment of CNN 3 is shown, and in particular the operation of the embodiment of CNN 3 is shown.

[0177] Figure 2 The device 1 in can reduce the amount of computing resources required to process a scanned data stream including spatial information provided by a 2D or 3D sensor 2. Since the CNN 3 includes one or more first buffers 4a for storing the output tensors of the corresponding previous layer, performing one or more convolution operations in each other layer at the current point in time does not require recalculating one or more previous tensors, specifically output tensors, of the corresponding previous layer. That is, the device 1 is used to perform one or more convolution operations in each other layer based on the current output tensor of the previous layer derived from the currently scanned data and the previous output tensor of the previous layer, wherein the previous output tensor of the previous layer is the latest tensor stored in the corresponding first buffer 4a. The use of one or more first buffers 4a reduces the inference time required by the CNN 3 in the inference phase.

[0178] like Figure 3A As shown on the right side of FIG, device 1 and 2D or 3D sensor 2 may be part of ego vehicle 5. That is, ego vehicle 5 includes 2D or 3D sensor 2 and device 1. Ego vehicle 5 may also include more than one 2D or 3D sensor 2. Thus, ego vehicle 5 includes device 1 and one or more 2D or 3D sensors 2 for measuring distance. The one or more 2D or 3D sensors 2 are configured to provide scans, including spatial information of the neighborhood or environment of ego vehicle 5, to device 1 in the form of a data stream, and device 1 is configured to process the data stream.

[0179] The one or more 2D or 3D sensors 2 for measuring distances can comprise or correspond to one or more light detection and ranging (LiDAR) sensors, time of flight (TOF) cameras, stereo cameras and / or beamforming radars, respectively. That is, the one or more 2D or 3D sensors 2 can comprise or correspond to one or more visual depth sensors, respectively. The one or more 2D or 3D sensors 2 can provide scans at a frame rate of at least 10 to 20 frames per second.

[0180] The ego vehicle 5 can correspond to:

[0181] a vehicle, e.g., a car, a truck, a motorcycle, etc.;

[0182] an autonomous vehicle, e.g., an autonomous car, an autonomous truck, etc.;

[0183] a robot, e.g., a delivery robot;

[0184] an autonomous robot, e.g., an autonomous delivery robot;

[0185] a flying vehicle, e.g., a flying drone, a flying robot, an airplane, etc.; or

[0186] an autonomous flying vehicle, e.g., an autonomous flying drone, an autonomous flying robot, an autonomous airplane, etc.

[0187] The ego vehicle 5 can comprise a positioning unit (not shown in Figure 3A ) for determining a current position of the ego vehicle 5, thereby determining current position data of the ego vehicle 5. The positioning unit can be configured for short-term positioning, e.g., positioning in the range of 1 s to 10 s. The positioning unit can comprise or correspond to one or more inertial measurement units (IMUs). In particular, the positioning unit is configured to perform a odometry process, e.g., an inertial odometry process, a wheel odometry process, and / or an optical or visual odometry process, etc.

[0188] Figure 3A An operation of a Convolutional Neural Network (CNN) at a current time point ti is shown.

[0189] The above description of the CNN in the device of the first aspect and implementations thereof, the above description of the CNN of the hardware implementation of the CNN of the second aspect and implementations thereof, and the above description of the method of the fourth aspect and implementations thereof apply correspondingly to Figure 3ACNN 3 in. Figure 3A The CNN 3 in can be Figure 3A An embodiment of a CNN 3 in the device 1.

[0190] according to Figure 3A In the embodiment of the present invention, the CNN 3 comprises a first layer L1, a further layer L2a and a first buffer 4a, wherein the further layer L2a is a continuous layer below the first layer L1. That is, according to Figure 3A In the embodiment of the invention, the first layer L1 is a layer above the other layer L2a. As mentioned above, the CNN 3 may include one or more other layers. Additionally or alternatively, the CNN 3 may include one or more optional additional layers ( Figure 3A not shown).

[0191] like Figure 3B As shown, at the current time point t i (Time T is equal to time point t i ) will be used to measure the distance by a 2D or 3D sensor 2 ( Figure 3B The data of the current scan provided by the current time point t i The current tensor epc(t i ) is input to CNN 3. In the other layer L2a, one or more convolution operations are performed according to the following:

[0192] –The current output tensor a1(t i ), where the current output tensor a1(t i ) originates from the current time point t i The currently scanned data;

[0193] – The previous output tensor a1(t i–1 ), where the previous output tensor a1(t i–1 ) is the latest tensor stored in the first buffer 4a and is derived from the previously scanned data input to the CNN immediately before the currently scanned data ( Figure 2 not shown).

[0194] The previous scan is the scan immediately before the current time point t i The previous time point t i–1 Therefore, the previous scan can be called the previous time point t i–1 Scan.

[0195] The current output tensor a2(t i ) can be stored as the latest tensor in the first buffer 4a ( Figure 3B (not shown). Therefore, immediately after the current time point ti a subsequent time point t i+1 The current output tensor a1(t i+1 ) provided above is a 2D or 3D tensor. i+1 The subsequent output tensor a1(t i+1 ) provided above is a 2D or 3D tensor. i The current output tensor a1(t i ) provided above is a 2D or 3D tensor. i–1 The subsequent output tensor a1(t i ) provided above is a 2D or 3D tensor. i–1 The subsequent output tensor a1(t i–1 ) provided above is a 2D or 3D tensor. i The subsequent output tensor a1(t i ) provided above is a 2D or 3D tensor. i The subsequent output tensor a1(t i ) provided above is a 2D or 3D tensor. i–1 The subsequent output tensor a1(t i–1 ) provided above is a 2D or 3D tensor.

[0196] Figure 3A The CNN 3 in the above can reduce the amount of computational resources required to process a data stream of a scan including spatial information provided by the 2D or 3D sensor 2. Since the CNN 3 includes the first buffer 4a for storing the output tensor of the first layer L1, the re-computation of the previous output tensor a1(t i–1 ) of the first layer L1 is not required to perform one or more convolution operations in the other layers L2a at the current time point t i . That is, one or more convolution operations are performed in the other layers L2a based on the current output tensor a1(t i ) of the first layer L1 derived from the current scan data and the previous output tensor a1(t i–1 ) of the first layer L1, wherein the previous output tensor a1(t i–1 ) of the first layer L1 is the latest tensor stored in the first buffer 4a. The use of the first buffer 4a reduces the inference time required by the CNN 3 in the inference phase.

[0197] According to Figure 3B , since the first buffer 4a can only store the output tensor of the first layer L1 at one time point, the temporal size of the first buffer 4a is equal to 1 (“ 1 ”). Therefore, when the current output tensor a1(t i ) of the previous layer L1 is stored in the first buffer 4a as the latest tensor at the current time point t i , the previous output tensor a1(t i–1 ) of the previous layer L1 stored in the first buffer 4a as the latest tensor at the previous time point t i immediately preceding the current time point t i is discarded or deleted at the current time point t i . Specifically, the temporal size of the first buffer 4a is smaller than the temporal size of the convolution kernel of the first layer L1 by 1. Therefore, according to an embodiment of Figure 3A , the temporal size of the convolution kernel of the first layer L1 can be equal to 2 (“ 2 ”).

[0198] The temporal size of the first buffer can optionally be larger than 1 ("1"), so that the first buffer can be used to store the output tensor of the previous layer L1 at more than one point in time (i.e. two or more subsequent points in time) or more than one scan (i.e. two or more subsequent scans).

[0199] Figure 3B An operation performed by a Convolutional Neural Network (CNN) according to an embodiment of the application at a current point in time t is shown.

[0200] Figure 3B The CNN 3 in the device 1 of Figure 3B may be an embodiment of the CNN 3 in the device 1 of Figure 3A The CNN 3 in the device 1 of Figure 3B differs from the CNN 3 in the device 1 of Figure 3B in that the CNN 3 in the device 1 of Figure 3B includes a second buffer 4b for storing tensors input to the CNN 3. Thus, Figure 3A the description of the CNN 3 in the device 1 of Figure 3B applies to the CNN 3 in the device 1 of Figure 3B The description of the CNN 3 in the device 1 of the first aspect and its implementation manners, the description of the CNN 3 in the hardware implementation of the CNN of the second aspect and its implementation manners, and the description of the method of the fourth aspect and its implementation manners above apply to the CNN 3 in the device 1 of Figure 4 respectively. The following mainly describes one or more additional features of the CNN 3 in the device 1 of Figure 4 and the differences between the CNN 3 in the device 1 of

[0201] As shown in Figure 2 , in addition to the first layer L1, one other layer L2a and the first buffer 4a, the CNN 3 also includes a second buffer 4b for storing tensors input to the CNN 3. According to the current tensor epc(t i derived from the data of the current scan at the current point in time t i and the previous tensor epc(t i ) generating the current output tensor a1(t i–1 ) of the first layer L1 derived from the data of the current scan, one or more convolution operations are performed in the first layer L1, wherein the previous tensor epc(t i–1 ) is the latest tensor stored in the second buffer 4b at the point in time t i–1 and corresponds to the point in time t i–1 immediately preceding the current point in time t iThe currently scanned data input to CNN 3 precedes the previously scanned data input to CNN 3.

[0202] The current output tensor a1(t i ) can be stored as the latest tensor in the second buffer 4b ( Figure 4 (not shown). Therefore, immediately after the current time point t i At the subsequent time point t i+1 On the other hand, according to the source from the 2D or 3D sensor at the subsequent time point t i+1 The subsequent tensor epc(t i+1 ) and at subsequent time point t i+1 The current tensor epc(t i ), performing one or more convolution operations.

[0203] Therefore, with Figure 3A Compared with CNN 3 in Figure 4 The CNN 3 in can further reduce the amount of computing resources required to process the data stream including the scanned spatial information provided by the 2D or 3D sensor for measuring distance. Since the CNN 3 includes a second buffer 4b for storing tensors input to the CNN 3, at the current time point t i Performing one or more convolution operations in the first layer L1 does not require recalculating the previous tensor epc(t–1) input to CNN 3 based on the corresponding data of the previous scan. That is, based on the current tensor epc(t i ) and the previous tensor epc(t i–1 ), one or more convolution operations are performed in the first layer L1, where the previous tensor epc(t i–1 ) is the latest tensor stored in the second buffer 4b.

[0204] according to Figure 3A , since the second buffer 4b can only store tensors input to the CNN 3 at one time point or one scan, the time size of the second buffer 4b is equal to 1 (" 1 ”). Therefore, in the current tensor epc(t i ) at the current time t i When the tensor is stored in the second buffer 4b as the latest tensor, at the current time point t i At the same time, discard or delete the i The previous time point t i–1 The previous tensor epc(t i–1 ).

[0205] The temporal size of the second buffer 4b can optionally be larger than 1 (“ 1 ”), thus the second buffer 4b can be used to store tensors input to the CNN 3 at more than one time point (i.e. two or more candidate time points) or more than one scan (i.e. two or more scans).

[0206] In one embodiment of the application, the first buffer 4a and the second buffer 4b have different temporal sizes. Optionally, they can have the same temporal size.

[0207] Figure 4 An operation of a Convolutional Neural Network (CNN) at a current time point ti provided by one embodiment of the application is shown.

[0208] Figure 4 The CNN 3 in Figure 4 is one embodiment of the CNN 3 in the device 1 of Figure 3A The CNN 3 in Figure 4 differs from the CNN 3 in Figure 4 The CNN 3 in 2 comprises a first buffer 4a having a temporal size of 2 (“ Figure 4 The description of the CNN 3 in Figure 4 applies accordingly to the CNN 3 in Figure 5 The description of the CNN 3 in Figure 5 describes one or more additional features of the CNN 3 in Figure 2 or differences between the CNN 3 in Figure 5 and the CNN 3 in

[0209] According to Figure 4 , the temporal size of the first buffer 4a equals 2 (“ 2 ”) since the first buffer 4a can store the output tensor of the first layer L1 (i.e. the preceding layer of the other layer L2a) at two subsequent time points or two subsequent scans. As shown in Figure 5 , at the current time point t i , the first buffer 4a stores as the latest tensor:

[0210] - the output tensor of the first layer L1 at the preceding time point t i–1The previous output tensor a1(t i–1 ), where the currently scanned data is at the current time point t i The current tensor epc(t i ) is input into CNN 3;

[0211] – The first layer L1 originates from the immediately previous time point t i–1 The second previous time point t i–2 The second previous output tensor a1(t i–2 ).

[0212] Therefore, when the current output tensor a1(t i ) at the current time t i When the tensor is stored in the first buffer 4a as the latest tensor, at the current time point t i At the same time, discard or delete the second previous output tensor a1(t i–2 )(like Figure 3B That is, the time size of the first buffer 4a is only 2, so the first buffer 4a can only store the first layer L1 at two subsequent time points (for example, two candidate time points t i–2 and t i–1 or t i–1 and t i ).

[0213] The time size of the first buffer can optionally be greater than 2 (" 2 ”), therefore, the first buffer can be used to store the output tensors of the first layer L1 at more than two time points (i.e., three or more subsequent time points) or more than two scans (i.e., three or more subsequent scans).

[0214] Figure 4 The diagram shows the operation of a convolutional neural network (CNN) at the current time point ti provided by one embodiment of the present invention.

[0215] Figure 5 The CNN 3 in can be Figure 5 An embodiment of a CNN 3 in the device 1. Figure 5 CNN 3 in Figure 4 The difference between CNN 3 in is that Figure 5 The CNN 3 in includes a second buffer 4b for storing tensors input to the CNN 3. Therefore, Figure 5 andFigure 5 The description of the CNN 3 in Figure 5 applies accordingly to the CNN in the device of the first aspect and its implementation forms, to the CNN of the hardware implementation of the CNN provided above for the second aspect and its implementation forms, and to the method of the fourth aspect and its implementation forms. Figure 5 The description of the CNN 3 in Figure 6 applies accordingly to the CNN in the device of the first aspect and its implementation forms, to the CNN of the hardware implementation of the CNN provided above for the second aspect and its implementation forms, and to the method of the fourth aspect and its implementation forms. Figure 6 The description of the CNN 3 in Figure 2 applies accordingly to the CNN in the device of the first aspect and its implementation forms, to the CNN of the hardware implementation of the CNN provided above for the second aspect and its implementation forms, and to the method of the fourth aspect and its implementation forms.

[0216] As shown in Figure 6 , the CNN 3 comprises, in addition to the first layer L1, one further layer L2a and the first buffer 4a, a second buffer 4b for storing tensors input into the CNN 3.

[0217] According to Figure 3A , since the second buffer 4b can store tensors input into the CNN 3 at two subsequent time points, the temporal size of the second buffer 4b is equal to 2 (“ 2 ”). As shown in Figure 6 , at a current time point t i , the second buffer 4a stores as the latest tensor:

[0218] - a previous tensor epc(t i–1 ) corresponding to previously scanned data input into the CNN 3 immediately before the current scanned data at a previous time point t i–1 , wherein the current scanned data is input into the CNN 3 at the current time point t i in the form of the current tensor epc(t i );

[0219] - a second previous tensor epc(t i–2 ) corresponding to second previously scanned data input into the CNN 3 at a second previous time point t i–2 immediately before the previous time point t i–1 .

[0220] Thus, when the current tensor a1(t i ) is stored in the second buffer 4b at the current time point t i as the latest tensor, the second previous tensor epc(t i–2 ) is discarded or deleted at the same time at the current time point t i (as shown in ​the dashed box on the left). That is, the temporal size of the second buffer 4b is only 2, and thus the second buffer 4b can only store tensors input to the CNN 3 at two subsequent time points (e.g. two subsequent time points t i–2 and t i–1 or t i–1 and t i .

[0221] The temporal size of the second buffer 4b can optionally be greater than 2 (“2”), and thus the second buffer 4b can be used to store tensors input to the CNN 3 at more than two time points (i.e. three or more subsequent time points).

[0222] ​ An operation of a Convolutional Neural Network (CNN) at a current time point t, provided by an embodiment of the application, is shown.

[0223] ​ The CNN 3 in the device 1 of ​ is one embodiment of the CNN 3 in the device 1 of ​ The CNN 3 in the device 1 of ​ differs from the CNN 3 in the device 1 of ​ The CNN 3 in the device 1 of Figure 3A and Figure 3B applies. The description of the CNN 3 in the device 1 of Figure 6 applies accordingly. The description of the CNN in the device of the first aspect and its implementation forms, the description of the CNN of the hardware implementation of the CNN of the second aspect and its implementation forms, and the description of the method of the fourth aspect and its implementation forms apply accordingly to the CNN 3 in the device 1 of Figure 6 . The following mainly describes one or more additional features of the CNN 3 in the device 1 of Figure 6 or Figure 3A the CNN 3 in the device 1 of Figure 6 and the differences between the CNN 3 in the device 1 of

[0224] According to an embodiment of the device 1 of Figure 6 , the CNN 3 comprises a first layer L1, three further layers L2a, L2b and L2c, three first buffers 4a and one optional second buffer 4b. The first further layer L2a is a consecutive layer below the first layer L1, the second further layer L2b is a consecutive layer below the first further layer L2a, and the third further layer L2c is a consecutive layer below the second further layer L2b. That is, according toFigure 6 In the embodiment, the first layer L1 is an upper layer of the first other layer L2a, the first other layer L2a is an upper layer of the second other layer L2b, and the second other layer L2b is an upper layer of the third other layer L2c.

[0225] As described above, CNN 3 may include only one other layer, or only two other layers, or more than three other layers. Additionally or alternatively, CNN 3 may include one or more optional additional layers ( Figure 6 In one embodiment, one or more of the one or more optional additional layers may be disposed between the other layers and / or between the first layer and the first other layer.

[0226] like Figure 6 As shown, at the current time point t i On the other hand, a 2D or 3D sensor ( Figure 6 not shown) (e.g. Figure 2 The current scan data pc(t) provided by the 2D or 3D sensor 2) i ) at the current time point t i The current tensor epc(t i ) is input into CNN 3. According to the current scanned data pc(t i ), generating the current tensor epc(t by optionally performing voxelization VX i ).

[0227] In one embodiment of the present invention, if the current scan pc(t i ) is set on the vehicle, then according to the current scanning pc(t i ) data and the self-driving car (e.g. Figure 2 The current position data of the ego vehicle 5) in the local navigation coordinate system is generated by performing voxelization VX to generate the current tensor epc(t i ).

[0228] Specifically, the current scan pc(t i ) corresponds to a point cloud, where a point cloud is a set of points and their attributes in N-dimensional space, where N corresponds to the number of spatial dimensions.

[0229] In each other layer L j (j can be 2a, 2b, or 2c), one or more convolution operations are performed according to the following:

[0230] - Upper level L v The current output tensor a w (t i ), where the current output tensor a w (ti ) originates from the current time point t i The current scanned data pc(t i );

[0231] - Upper level L v The previous output tensor a of w (t i–1 ), where the previous layer L v The previous output tensor a of w (t i–1 ) is the latest tensor stored in the corresponding first buffer 4a and is derived from the data pc(t immediately preceding the current scan i )The previously scanned data previously input to the CNN ( Figure 6 (not shown in ) (if j = 2a, then w = 1, v = 1; if j = 2b, then w = 2, v = 2a; if j = 2c, then w = 3, v = 2b).

[0232] The previous scan is the scan immediately before the current time point t i The previous time point t i–1 Therefore, the previous scan can be called the previous time point t i–1 Scan.

[0233] Previous layer L v The current output tensor a w (t i ) can be stored as the latest tensor in the corresponding first buffer 4a (such as Figure 6 Therefore, immediately after the current time point t i At the subsequent time point t i+1 On the basis of the previous layer L v The source of the 2D or 3D sensor at a subsequent time point t i+1 The subsequent output tensor a of the scanned data provided above w (t i+1 ) and at subsequent time point t i+1 The current output tensor a is stored in the corresponding first buffer 4a as the latest tensor w (t i ), in each other layer L j (j can be 2a, 2b or 2c) to perform one or more convolution operations.

[0234] according to Figure 6 , since the first buffer 4a can only store the output tensor of the corresponding layer (previous layer) at one time point or one scan, the time size of the first buffer 4a is equal to 1 (" 1 ”). Therefore, when the corresponding layer L vthe current output tensor a of the (upper) layer w (t i ) stored in the corresponding first buffer 4a at the current time point t i , the previous output tensor a of the corresponding layer L i at a previous time point t v immediately preceding the current time point t i is discarded or deleted at the current time point t i–1 , and the previous output tensor a of the corresponding layer L w (t i–1 ) is stored in the corresponding first buffer 4a at the current time point t i (as shown in the dashed box on the left; if v = 1, then w = 1; if v = 2a, then w = 2; if v = 2b, then w = 3). Specifically, the temporal size of the first buffer 4a is one less than the temporal size of the convolution kernel of the corresponding layer (the upper layer). Thus, according to embodiments of the application, the temporal size of the convolution kernel of the layers L1, L2a and L2b can be equal to 2 (“ Figure 6 ”). 2

[0235] The temporal size of the one or more first buffers 4a can optionally be greater than one (“ 1 ”), and thus, the one or more first buffers 4a can be used to store output tensors of the corresponding upper layer at more than one time point (i.e. two or more subsequent time points) or more than one scan (i.e. two or more subsequent scans).

[0236] According to embodiments of the application, one or more convolution operations are performed in the first layer L1 based on a current tensor epc(t i ) derived from the data pc(t i ) of the current scan at the current time point (t i ) and a previous tensor epc(t i–1 ) of a current output tensor a1(t i–1 ) of the first layer L1 generated from the data pc(t i–1 ) of the current scan. The previous tensor epc(t i ) is the most recent tensor stored in the optional second buffer 4b and corresponds to the data of a previous scan input to the CNN 3 at a previous time point t i immediately preceding the data pc(t i ) of the current scan.

[0237] The current output tensor a1(t i+1 ) of the first layer L1 can be stored as the most recent tensor in the optional second buffer 4b (as shown in the dashed box on the left). Thus, the previous output tensor a1(t i+1 ) of the first layer L1 is discarded or deleted at the current time point t i+1 .At the subsequent time point t i+1 On the other hand, according to the source from the 2D or 3D sensor at the subsequent time point t i+1 The subsequent tensor epc(t i+1 ) and at subsequent time point t i+1 The current tensor epc(t i ), performing one or more convolution operations.

[0238] according to Figure 6 , since the optional second buffer 4b can only store tensors input to CNN3 at one time point or one scan, the temporal size of the optional second buffer 4b is equal to 1 (" 1 ”). Therefore, in the current tensor epc(t i ) at the current time t i When the tensor is stored in the second buffer 4b as the latest tensor, at the current time point t i At the same time, discard or delete the i The previous time point t i–1 The previous tensor epc(t i–1 )(like Figure 6 dashed box on the left).

[0239] The time size of the optional second buffer 4b may optionally be greater than 1 (" 1 ”), therefore, the second buffer 4b can be used to store tensors that are input to the CNN 3 at more than one time point (i.e., two or more subsequent time points) or more than one scan (i.e., two or more subsequent scans).

[0240] In one embodiment of the present invention, one or more of the first buffer 4a and the optional second buffer 4b have different temporal sizes. Alternatively, the first buffer 4a and the optional second buffer 4b may have the same temporal size.

[0241] Figure 7 The figure shows a grid G ​​of a local navigational frame (LNF) and positions of the ego-vehicle at two subsequent time points according to an embodiment of the present invention.

[0242] Self-driving car ( Figure 7 ) may correspond to Figure 2 The above description of the device provided by the first aspect of the present invention and its implementation mode and the above description of the vehicle provided by the third aspect of the present invention and its implementation mode apply accordingly. Figure 7 Description, specifically applicable toFigure 7 Description of the self-driving car 5.

[0243] References below Figure 2 The vehicle 5 is shown. The vehicle 5 can be configured to move autonomously. Specifically, the device 1 in the vehicle 5 is used to control the autonomous movement of the vehicle 5 by processing a data stream of scans provided by one or more 2D or 3D sensors 2 for measuring distance and provided on the vehicle 5.

[0244] During the autonomous movement of the ego vehicle 5 , one or more 2D or 3D sensors 2 perform detection in a local navigation frame (LNF). Figure 7 The local navigational frame (LNF) shown is a coordinate system with two spatial dimensions x, y, which has a top-down view on the ground and is relative to the position of the ego vehicle 5. In other words, the position or orientation of the ego vehicle 5 is within the local navigational frame (LNF). The term " Location "and" Position " can be used as a synonym.

[0245] In a local navigational frame (LNF), data (e.g., a point cloud) of a current scan (a scanned data stream provided by one or more 2D or 3D sensors 2) is registered according to the current position of the ego vehicle 5. As the ego vehicle 5 moves, a region of interest (ROI) for processing the scanned data stream provided by the one or more 2D or 3D sensors 2 moves with the position of the ego vehicle 5 in the local navigational frame (LNF). Specifically, the ROI corresponds to a predefined regular grid or area of ​​the local navigational frame (LNF), with the ego vehicle 5 at the center, and the regular grid includes a plurality of cells. Therefore, Figure 7 Shows the current time point t i Current ROI R i Grid G i (area) and immediately after the current time point t i–1 The previous time point t i–1 Previous ROI R i–1 The grid G i–1(area). For example, the ROI can be selected to correspond to a grid of size 140 m x 140 m in a local navigational frame (LNF). The size of each cell is 0.5 m x 0.5 m. In particular, the ROI can be selected to correspond to a grid of size between 70 m x 70 m and 250 m x 250 m in a local navigational frame (LNF). In particular, the size of each cell can be between 0.05 m x 0.05 m and 0.5 m x 0.5 m.

[0246] According to Figure 7 , the data of the current scan (e.g. all points in a point cloud such as a LiDAR point cloud when the 2D or 3D sensor is a LiDAR sensor) is mapped into a regular grid G of a local navigational frame (LNF). That is, the device 1 is configured to generate a current tensor input to the CNN from the data of the current scan and current position data px i and py i of the ego vehicle in the grid G of the local navigational frame (LNF). The current position data px i and py i describe the current position p i or bearing of the ego vehicle 5 in the grid G of the local navigational frame (LNF) at the current point in time t i . Thus, the tensors stored in the buffers of the CNN 3 correspond to the respective cells in the regular grid G of the local navigational frame (LNF).

[0247] Figure 7 The grid G i (area) in the current tensor corresponds to the current tensor stored in the respective buffer of the CNN 3 as the most recent tensor at the current point in time t i . The grid G i corresponds to the predefined grid (area) of the current ROI R i at the current point in time t i . In Figure 7 , the grid G i includes blank cells and point-like cells within the corresponding coarse frame. Figure 7 The grid G i–1 in the current tensor corresponds to the current tensor stored in the respective buffer of the CNN 3 as the most recent tensor immediately preceding the current point in time ti The previous time point t i–1 The previous tensor stored in the same buffer of CNN 3 as the latest tensor. Grid G i–1 Corresponding to the previous time point t i–1 Previous ROI R i–1 A predefined grid (area) of Figure 7 In the grid G i–1 Including the diagonal cells and blank cells in the corresponding thick boxes.

[0248] In the local navigational frame (LNF) (grid G), since the ego vehicle 5 has been moved from the previous time point t i–1 The previous position p i–1 Move to the current time point t i Current location p i , so the current time point t i Grid G i Compared with the previous time point t i–1 The grid G i–1 Inconsistent. That is, at the previous time point t i–1 On the other hand, the vehicle 5 is at the previous position p i–1 , and at the current time point t i On the left, the vehicle 5 is at the current position p i Due to this movement, the ROI changes from the previous ROIR i–1 Change to the current ROI R i The previous position p of vehicle 5 i–1 From the previous position data px i–1 、py i–1 Description, the current position p of vehicle 5 i By the current position data px i 、py i describe.

[0249] If the current position data px of the vehicle 5 i 、py i Does not match the previous position data px entered with the previously scanned data i–1 、py i–1 , then fill and clip the latest tensor stored in each buffer of CNN. That is, the device 1 in the ego vehicle 5 can be used to: convert the current position data px of the ego vehicle 5 in the grid G ​​of the local navigation frame (LNF) i 、py i Together with the currently scanned data, it is input to CNN 3 (CNN Figure 7the current position data px i , py i of the ego vehicle 5 does not match the previous position data px i–1 , py i–1 inputted together with the data of the previous scan, the padding and cropping (in particular, zero padding and cropping) of the stored latest tensor in each buffer of the CNN is performed.

[0250] Thus, according to Figure 7 , the blank cells in the coarse frame correspond to the data of the latest tensor stored in the buffer, although the position of the ego vehicle 5 has changed, thus the ROI has changed from the previous ROI R i–1 to the current ROI R i , the data of the latest tensor can also be reused. The diagonal cells correspond to the data of the cropped latest tensor stored in the buffer. The terms “ Dropped Deleted ” or “ Cropped ” can be used as synonyms for “ Figure 7 ”. The dot-like cells correspond to the data of the padded (in particular, zero-padded) latest tensor stored in the buffer. Padded data can be understood as the value of the data being overwritten with a predefined value. Thus, zero-padded data can be understood as the value of the data being overwritten with zero.

[0251] If the ego vehicle 5 moves within the same cell of the grid G (not shown in Figure 7 ), no padding and cropping is performed. In other words, if the current position data px i , py i of the ego vehicle 5 does not match the previous position data px i–1 , py i–1 inputted together with the data of the previous scan, but the current position data px i , py i of the ego vehicle 5 and the previous position data px i–1 , py i–1 are within or located in the same cell of the grid G in the local navigational frame (LNF), no padding and cropping is performed.

[0252] The device 1 in the ego vehicle 5 can be configured to generate the current tensor inputted to the CNN 3 from the data of the current scan and the current position data px i , py i of the ego vehicle 5 in the grid G in the local navigational frame (LNF) by transforming the coordinates of the data of the current scan into the local navigational frame (LNF).

[0253] Furthermore, the device 1 in the ego vehicle 5 can be configured to store the tensor of the data of the current scan in a buffer of the CNN provided by an embodiment of the invention in dependence on the current position data px i , py i of the ego vehicle 5 in the grid G of the local navigational frame (LNF) at the time point t Figure 7 at which the data of the current scan is acquired.

[0254] The position data can be stored in a positioning field of one or more first buffers 4a and optionally one or more buffers of a second buffer of the CNN, if the current position data px i , py i of the ego vehicle 5 does not match the previous position data px i–1 , py i–1 , the positioning field of the respective one or more buffers is updated (not shown in Figure 8 ) with the current position data px i , py i .

[0255] If the ego vehicle 5 moves within the same cell of the grid G, the positioning field of the respective one or more buffers is not updated. In other words, if the current position data px i , py i of the ego vehicle 5 does not match the previous position data px i–1 , py i–1 , which is input together with the data of the previous scan, but the current position data px i , py i of the ego vehicle 5 and the previous position data px i–1 , py i–1 are within or located in the same cell of the grid G, the positioning field of the respective one or more buffers is not updated with the current position data px i , py i .

[0256] Figure 2 is a schematic diagram of a method provided by an embodiment of the invention for storing a tensor of data of a current scan originating from a current time point t i in a buffer of a CNN provided by an embodiment of the invention.

[0257] The following description refers to the ego vehicle 5 in Figure 7 and to the local navigational frame in Figure 8 .

[0258] In Figure 8In a first step S81 of the method, the device 1 in the ego vehicle 5 acquires data of a current scan provided by one or more 2D or 3D sensors 2 in the form of a current tensor of shape [C, H, W] and a current position data px i , py i in a grid G of a local navigational frame (LNF) at a current time point t i “C” is the number of channels. The current tensor is generated by transforming the coordinates of the data of the current scan into a local navigational frame (LNF). Thus, “H” is the size of the current tensor along the y coordinate and “W” is the size of the current tensor along the x coordinate.

[0259] In a second step S82 following the first step S81, the device 1 determines whether the current position data px i , py i matches previous position data px i–1 , py i–1 of the ego vehicle 5. That is, the device 1 determines whether the position of the ego vehicle 5 has changed. Specifically, the device 1 determines whether the current position data px i , py i matches the previous position data px i–1 , py i–1 and whether the current position data px i , py i and the previous position data px i–1 , py i–1 are not within the same cell of the grid G of the local navigational frame (LNF). That is, the device 1 specifically determines whether the position of the ego vehicle 5 has changed such that the current position of the ego vehicle is within a different cell than the previous position. If the determination of the second step S82 is “yes” (the position of the ego vehicle has changed), the method proceeds to a third step S83. If the determination of the second step S82 is “no” (the position of the ego vehicle has not changed or the position of the ego vehicle is still within the same cell), the method proceeds to a fifth step S85.

[0260] In the third step S83, the device 1 pads and crops (specifically zero-pads and crops) the stored latest tensor in each buffer of the CNN 3 to match the current position data px i , py i , thereby matching the current ROI R i . That is, the padding and cropping matches the current ROI R i immediately preceding the current time point ti–1 The tensors stored in the respective buffer as the latest tensors and still stored in the respective buffer as the latest tensors in the third step S83.

[0261] In a fourth step S84 following the third step S83, the device 1 updates the localization field in the buffer comprising the localization field using current position data px i , py i of the ego vehicle 5.

[0262] In a fifth step S85, the device 1 stores the respective tensors stemming from the data of the current scan as the latest tensors in each buffer and simultaneously discards or deletes the oldest tensors in the full buffers. Specifically, the device 1 stores the output tensors stemming from the data of the current scan of the corresponding layer (the layer above) as the latest tensors in each first buffer 4a. Furthermore, if the CNN 3 in the device 1 comprises a second buffer, the device 1 stores the current tensors stemming from the data of the current scan as the latest tensors in the second buffer.

[0263] Specifically, the device 1 in the ego vehicle 5 repeats the method in Figure 9 at each new time point or new scan.

[0264] Figure 9 is a schematic diagram of a method for processing a data stream of scans of spatial information provided by one or more 2D or 3D sensors for measuring distances using a Convolutional Neural Network (CNN) in an inference phase according to an embodiment of the invention.

[0265] Figure 2 The method in Figure 9 may be performed by the device 1 in Figure 9 . The above description of the device of the first aspect and implementation forms thereof, the above description of the hardware implementation of the CNN of the second aspect and implementation forms thereof, and the above description of the method of the fourth aspect and implementation forms thereof apply accordingly to the method in

[0266] According to Figure 2 , the method steps S101 and optionally the method step S102 can be performed on the basis of a current scan of data pc(t i ) provided by a 2D or 3D sensor for measuring distances (e.g. a LiDAR sensor) at a current time point t i on. The 2D or 3D sensor is arranged on an ego vehicle (e.g. the ego vehicle 5 in Figure 7 ). In step S101, the current scan of data pc(t i ) and the current position data px i , py i of the ego vehicle in a local navigation coordinate system (e.g. the local navigation coordinate system in Figure 9current position data px i , py i in the grid of the local navigation coordinate system, a current tensor epc(t i ) is generated which is input to the convolutional neural network (CNN) 3 by transforming the coordinates of the data pc(t i ) into the local navigation coordinate system. From the data pc(t i ) of the current scan and the current position data px i , py i of the ego vehicle in the grid of the local navigation coordinate system, a current tensor epc(t i ) is generated which is input to the convolutional neural network (CNN) 3 by additionally performing the voxelization of the optional method step S102.

[0267] The CNN 3 is used to generate a current output data OUT(t i ) from the current tensor epc(t i ), and the navigation process of the ego vehicle is performed in accordance with the current output data OUT(t i ).

[0268] Figure 2 The CNN 3 shown in Fig. 3 can correspond to any of the CNNs 3 in Figure 3A , Figure 3B , Figure 4 , Figure 5 , Figure 6 and Figure 9 . The above description of the CNN in the device of the first aspect and its implementation, the above description of the CNN of the hardware implementation of the CNN of the second aspect and its implementation, and the above description of the method of the fourth aspect and its implementation apply accordingly to the CNN 3 in Figure 9 .

[0269] Optionally, in a method step S103, the current output data OUT(t i ) can be decoded, and a non-maximum suppression can be performed on the current output data OUT(t i ). From the result of the processing according to the method step S103, a detection object frame can be provided.

[0270] The size of the data pc(t i ) as a point cloud is “ wherein "M" is the number of points in the point cloud, and "4" indicates that each point in the point cloud comprises three spatial dimensions (the point cloud comprises three spatial dimensions) and one attribute, e.g. a scalar reflectance intensity attribute.

[0271] Since the position of the ego vehicle in the local navigation coordinate system (i.e. according to the two-dimensional coordinate system of the embodiment ​ ) is described by two coordinates (x and y coordinates), the current position data px i , py i has a size of 2 (" 2 ").

[0272] The size of the current tensor epc(t i ) encoding the point cloud is "C*H*W". "C" is the number of channels of the current tensor epc(t i ) (the number of channels in the encoded point cloud), wherein C is greater than or equal to 1 ( ). "H" is the size of the current tensor epc(t i ) along the y coordinate of the local navigation coordinate system, and "W" is the size of the current tensor along the x coordinate of the local navigation coordinate system. "H" and "W" define the spatial resolution of the area of the region of interest (ROI) in the grid of the local navigation coordinate system. For example, "H" and "W" can both be 280 cells in the grid of the local navigation coordinate system, such that if the size of the cells in the regular grid of the local navigation coordinate system is 0.5 m x 0.5 m, then the area of the ROI is 140 m x 140 m.

[0273] The size of the output data OUT(t i ) is "C2*H*W", wherein "H" and "W" are as described above. "C2" is the number of channels of the output data OUT(t i ), which can be the same or different from the number of channels of the current tensor epc(t i ) ("C").

[0274] "B" is the number of detected objects, and "S" is the size of the metadata related to one object.

[0275] The application has been described in relation to different embodiments and implementations as examples. However, other variants can be understood and implemented by a person skilled in the art in the practice of the claimed application, in the light of the attached drawings, the application and the independent claims. In the claims and in the description, the word "comprising" does not exclude other elements or steps, and the terms "a" or "an" do not exclude a plurality. A single element or other unit can fulfill the functions of several entities or items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. A device (1) for processing a data stream of scans comprising spatial information provided by a 2D or 3D sensor (2) for measuring distances, characterized in that, The device (1) is configured to: use a Convolutional Neural Network, CNN, (3) in an inference phase, wherein the CNN comprises a first layer (LI) and one or more further layers (L2a, L2b, L2c) below the first layer (LI) and one or more first buffers (4a) for storing output tensors of a respective preceding layer (LI, L2a, L2b); the data (pc(t i ) of the current scan provided by the 2D or 3D sensor (2) in the form of the current tensor (epc(t i )) is input into the CNN (3);​​​​ According to the data (pc(t i )) of the current output tensor (a1(t i )、a2(t i )、a3(t i )) and the previous output tensor (a1(t i–1 )、a2(t i–1 )、a3(t i–1 )), one or more convolution operations are performed in each of the other layers (L2a, L2b, L2c), where the previous output tensors (a1(t i–1 ), a2(t i–1 ), a3(t i–1 )) of the upper layer (L1, L2a, L2b) are the most recent tensors stored in the respective first buffer (4a) of the one or more first buffers (4a) and originate from the data of a previous scan input to the CNN immediately before the data (pc(t i ) of the current scan; storing the current output tensors (a1(t i ), a2(t i ), a3(t i )) of the last layer (L1, L2a, L2b) as latest tensors in the respective first buffer (4a).

2. The device (1) according to claim 1, wherein The CNN (3) comprises a second buffer (4b) for storing tensors (epc(t i–1 ), epc(t i )) input to the CNN (3). The device (1) is configured to: from the current scan (pc(t i ) ) of the first layer (L1), wherein, in the first layer (L1), one or more convolution operations are performed on the basis of the current tensor (epc(t i ) ) and a previous tensor (epc(t i–1 ) ) of a current output tensor (a1(t i ) ) of the first layer (L1) which is generated from the data (pc(t i ) ) of the current scan, wherein, said previous tensor (epc(t i–1 )) is the most recent tensor stored in said second buffer (4b) and corresponds to said previous scanned data; storing the current tensor (epc(t i ) ) as a latest tensor in the second buffer (4b).

3. The device (1) according to claim 1 or 2, wherein Each buffer (4a, 4b) is a First-In-Last-Out buffer configured to store a new tensor as the newest tensor and, if the buffer is full, to discard the oldest tensor stored in the buffer at the same time.

4. The device (1) according to claim 1 or 2, wherein Each of the one or more first buffers (4a) for storing output tensors of a respective preceding layer (LI, L2a, L2b) has a temporal size which is smaller by one than a temporal size of a convolution kernel of the respective preceding layer (LI, L2a, L2b); The one or more buffers (4a, 4b) in particular have different temporal sizes.

5. The device (1) according to claim 1 or 2, wherein Each tensor has two or more dimensions, in particular one or more spatial dimensions and one channel dimension.

6. The device (1) according to claim 1 or 2, wherein Each tensor has two dimensions, in particular one spatial dimension and one channel dimension; Each tensor has three dimensions, in particular two spatial dimensions and one channel dimension; or Each tensor has four dimensions, in particular three spatial dimensions and one channel dimension.

7. The device (1) according to claim 1 or 2, wherein Each first buffer (4a) for storing output tensors of a respective preceding layer (LI, L2a, L2b) has a channel dimension which corresponds to a channel dimension of the respective preceding layer (LI, L2a, L2b).

8. The apparatus (1) according to claim 1 or 2, characterized in that The device (1) is configured to generate output data for a navigation process of a ego vehicle (5) in which the 2D or 3D sensor (2) is located, wherein the device (1) is configured to: The current position data (p) of the vehicle in the grid (G) of the local navigational frame (LNF) is Xi 、py i ) and the currently scanned data pc(t i ) are input together into the CNN (3); If the current position data (px Xi , py i ) of the ego vehicle does not match the previous position data (px i–1 , py i–1 ) entered together with the data of the previous scan, the stored latest tensor is padded and cropped (S83) in each buffer (4a, 4b).

9. The device (1) according to claim 8, characterized in that The device is configured to: store the position data in a position field of one or more of the one or more first buffers (4a) and the second buffer (4b); if the current position data (px Xi , py i ) of the ego vehicle does not match the previous position data (px i–1 , py i–1 ), update (S84) the position field of the respective one or more buffer using the current position data.

10. The device (1) according to claim 8, wherein The device (1) is used to detect the current scanned data (pc(t i )) and the current position data (p) of the vehicle (5) in the grid (G) of the local navigational frame (LNF) Xi 、py i ), by adding the currently scanned data (pc(t i )) is converted into the local navigational frame (LNF), generating (S101) the current tensor (epc(t i )).

11. The device (1) according to claim 10, wherein The device (1) is configured to generate the current tensor (epc(t i ) ) input to the CNN (3) by additionally performing voxelization (S102) from the current scanned data (pc(t Xi ) ) and the current position data (px i , py i ) of the ego vehicle (5) in a grid (G) of a local navigational frame (LNF).

12. The device (1) according to claim 11, wherein The device (1) is configured to use an output tensor of a last layer of the one or more further layers of the CNN (3) for: detecting objects, in particular moving objects, in a vicinity of the 2D or 3D sensor (2); performing a point cloud semantic segmentation; and / or performing a free space estimation.

13. A device for processing a data stream of scans comprising spatial information provided by a 2D or 3D sensor (2) for measuring distances, characterized in that, The device comprises a Convolutional Neural Network, CNN, (3); The CNN (3) comprises a first layer (LI) and at least one or more further layers (L2a, L2b, L2c) below the first layer (LI) and one or more first buffers (4a) for storing output tensors of the respective preceding layer (LI, L2a, L2b); The hardware implementation of the CNN is for: the data (pc(t i ) of the current scan provided by the 2D or 3D sensor (2) in the form of the current tensor (epc(t i ) is input into the CNN (3);​​​​ a current output tensor (a1(t i ), a2(t i ), a3(t i )) of the upper layer (L1, L2a, L2b) resulting from data (pc(t i )) of the current scan and a previous output tensor (a1(t i–1 ), a2(t i–1 ), a3(t i–1 )) of the upper layer (L1, L2a, L2b), wherein,​​​​​​ the previous output tensors (a1(t i–1 ), a2(t i–1 ), a3(t i–1 )) of the upper layer (L1, L2a, L2b) are the most recent tensors stored in the respective first buffer (4a) of the one or more first buffers (4a) and originate from the data of a previous scan input to the CNN (3) immediately before the data (pc(t i ) of the current scan; The current output tensors (a1(t i ), a2(t i ), a3(t i )) of the upper layer (L1, L2a, L2b) are stored as latest tensors in the respective first buffer (4a).

14. An ego vehicle (5) comprising one or more 2D or 3D sensors (2) for measuring distances and a device (1) according to any one of claims 1 to 12, characterized in that The one or more 2D or 3D sensors (2) are for providing a scan comprising spatial information of a neighborhood of the ego vehicle (5) to the device (1) in a data stream, the device (1) being for processing the data stream.

15. The ego vehicle (5) according to claim 14, characterized in that The device (1) is for controlling autonomous movement of the ego vehicle (5) by processing the data stream.

16. A method of processing a data stream comprising a scan of spatial information provided by a 2D or 3D sensor (2) for measuring distances in an inference phase using a Convolutional Neural Network, CNN, (3), characterized in that The CNN (3) comprises a first layer (LI) and at least one or more further layers (L2a, L2b, L2c) below the first layer (LI) and one or more first buffers (4a) for storing output tensors of the respective preceding layer (LI, L2a, L2b); The method comprises the following steps: the data (pc(t i )) of the current scan provided by the 2D or 3D sensor (2) in the form of the current tensor (epc(t i )) is input into the CNN (3);​​​​ a current output tensor (a1(t i ), a2(t i ), a3(t i )) of the upper layer (L1, L2a, L2b) resulting from data (pc(t i )) of the current scan and a previous output tensor (a1(t i–1 ), a2(t i–1 ), a3(t i–1 )) of the upper layer (L1, L2a, L2b), wherein,​​​​​​​ the previous output tensors (a1(t i–1 ), a2(t i–1 ), a3(t i–1 )) of the upper layer (L1, L2a, L2b) are the most recent tensors stored in the respective first buffer (4a) of the one or more first buffers (4a) and originate from a previous scan of data that was input into the CNN (3) immediately before the current scan of data; The current output tensors (a1(t i ), a2(t i ), a3(t i )) of the upper layer (L1, L2a, L2b) are stored as latest tensors in the respective first buffer (4a).

17. A computer program comprising program code for, When the program code is implemented on a processor, for performing the method according to claim 16.

18. A computer, comprising: The computer comprises a memory and a processor for storing and executing program code to perform the method according to claim 16. The computer comprises a memory and a processor for storing and executing program code to perform the method according to claim 16.

Images