Point cloud processing method, point cloud processing device and intelligent driving vehicle

Abstract

The invention provides a point cloud processing method, a point cloud processing device and an intelligent driving vehicle, and relates to the field of computers, in particular to the field of intelligent driving. The point cloud processing method comprises the following steps: performing feature extraction on point cloud data of a current frame of a laser radar to obtain a point cloud feature map of the current frame; performing time sequence fusion processing on the point cloud feature map of the current frame and the fusion feature map of the previous frame to obtain a fusion feature map of the current frame; and predicting one or more of the speed, the acceleration and the category of the target according to the fusion feature map of the current frame to obtain a speed prediction map, an acceleration prediction map and a category prediction map. The time sequence laser radar signals are used for sensing the speed, acceleration and category information of the target at the same time, the speed signals predicted by the two-dimensional speed prediction map have high accuracy in the radial direction and the transverse direction, the real motion state of the target can be reflected, and rich and high-quality upstream information is provided for active safety decision making of an automatic driving system.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, in particular to the field of intelligent driving, and in particular to a point cloud processing method, a point cloud processing device and an intelligent driving vehicle. Background Art

[0002] Intelligent driving vehicles are used to automatically transport people or objects from one location to another. They collect environmental information through sensors on the vehicle and complete automatic transportation. Intelligent driving vehicles controlled by autonomous driving technology for logistics transportation have greatly improved the convenience of production and life and saved labor costs.

[0003] The active safety module is an important component in the autonomous driving system. It senses and makes decisions about the driving environment. When a dangerous situation occurs, it sends emergency braking commands to the vehicle chassis to ensure vehicle safety.

[0004] The active safety modules of some intelligent driving vehicles use millimeter wave radar signals as input to sense the speed of the target. Millimeter wave radar is a radar that works in the millimeter wave band with a frequency range of 30-300GHz and a wavelength of 1-10mm. Millimeter wave radar transmits electromagnetic waves and receives echo signals reflected by the target, and obtains the target's position and speed information by comparing the transmitted signal with the recovered signal.

[0005] Active safety perception algorithms based on millimeter-wave radar can usually only perceive the target's speed information, lack acceleration and category information, and can only perceive radial velocity. The lateral velocity perception capability is weak, and it cannot capture the target's true motion state and cannot provide rich information for subsequent decision-making modules. Summary of the invention

[0006] The disclosed embodiment utilizes time-series lidar signals to simultaneously sense the speed, acceleration, and category information of a target, and the speed signal predicted by the two-dimensional speed prediction map has high accuracy in both radial and lateral directions, which can reflect the true motion state of the target and provide rich and high-quality upstream information for the active safety decision-making of the autonomous driving system.

[0007] Some embodiments of the present disclosure propose a point cloud processing method, including: performing feature extraction on point cloud data of a current frame of a laser radar to obtain a point cloud feature map of the current frame; performing time series fusion processing on the point cloud feature map of the current frame and a fused feature map of a previous frame to obtain a fused feature map of the current frame; predicting one or more of the speed, acceleration and category of the target based on the fused feature map of the current frame to obtain a speed prediction map, an acceleration prediction map and a category prediction map.

[0008] In some embodiments, the point cloud feature map of the current frame and the fused feature map of the previous frame are temporally fused to obtain the fused feature map of the current frame, including: splicing the point cloud feature map of the current frame and the fused feature map of the previous frame, and extracting temporal features to obtain the fused feature map of the current frame.

[0009] In some embodiments, the point cloud feature map of the current frame and the fused feature map of the previous frame are spliced, and the temporal features are extracted to obtain the fused feature map of the current frame, including: aligning the fused feature map of the previous frame to the coordinate system of the current frame according to the coordinate transformation matrix between the previous frame and the current frame, and then splicing it with the point cloud feature map of the current frame; using the temporal fusion neural network to extract the temporal features of the spliced ​​feature map to obtain the fused feature map of the current frame.

[0010] In some embodiments, the fused feature map of the current frame includes the timing information of each historical frame before the current frame, and the fused feature map of the previous frame includes the timing information of each historical frame before the previous frame.

[0011] In some embodiments, the fusion feature map of the initial frame is obtained by extracting the temporal features from the point cloud feature map of the initial frame using a temporal fusion neural network.

[0012] In some embodiments, predicting one or more of the speed, acceleration and category of the target based on the fused feature map of the current frame to obtain a speed prediction map, an acceleration prediction map and a category prediction map includes: using a speed prediction neural network, an acceleration prediction neural network and a category prediction neural network to process the fused feature map of the current frame, respectively, to obtain a speed prediction map, an acceleration prediction map and a category prediction map.

[0013] In some embodiments, the dimension of the velocity prediction map is 2, representing the radial component and lateral component of the target's velocity; or, the dimension of the acceleration prediction map is 1, representing the rate of change of the target's velocity; or, the dimension of the category prediction map is 1, representing the one-dimensional index of the target's category.

[0014] In some embodiments, feature extraction is performed on point cloud data of a current frame of the laser radar to obtain a point cloud feature map of the current frame, including: using a downsampling network to downsample the point cloud data of the current frame of the laser radar to obtain a first feature map of the current frame; using a cascade of multiple residual networks to downsample the first feature map to obtain a plurality of second feature maps with different downsampling rates output by each residual network; using a plurality of upsampling networks with different step sizes to upsample the plurality of second feature maps with different downsampling rates to obtain a plurality of third feature maps of the same size; and concatenating the plurality of third feature maps to obtain a point cloud feature map of the current frame.

[0015] In some embodiments, it also includes: before prediction, using an upsampling network to upsample the fused feature map of the current frame to obtain a fused feature map with the same size as the point cloud data of the current frame.

[0016] In some embodiments, the temporal fusion neural network includes a convolutional layer, a batch normalization layer, and an activation layer.

[0017] In some embodiments, the speed prediction neural network, the acceleration prediction neural network, and the category prediction neural network include a first convolutional layer, a batch normalization layer, an activation layer, and a second convolutional layer.

[0018] Some embodiments of the present disclosure propose a point cloud processing device, including: a feature extraction module, configured to extract features from point cloud data of a current frame of a laser radar to obtain a point cloud feature map of the current frame; a temporal fusion module, configured to perform temporal fusion processing on the point cloud feature map of the current frame and the fused feature map of the previous frame to obtain a fused feature map of the current frame; a prediction module, configured to predict one or more of the speed, acceleration and category of the target based on the fused feature map of the current frame to obtain a speed prediction map, an acceleration prediction map and a category prediction map.

[0019] Some embodiments of the present disclosure provide a point cloud processing device, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute a point cloud processing method based on instructions stored in the memory.

[0020] Some embodiments of the present disclosure propose an intelligent driving vehicle, including: a laser radar and a point cloud processing device, wherein the point cloud processing device is configured to execute a point cloud processing method on point cloud data from the laser radar.

[0021] Some embodiments of the present disclosure provide a computer-readable storage medium having computer instructions stored thereon, which implement the steps of a point cloud processing method when the computer instructions are executed by a processor.

[0022] Some embodiments of the present disclosure provide a computer program product, comprising computer instructions, which implement the steps of a point cloud processing method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The following is a brief introduction to the drawings required for use in the embodiments or related technical descriptions. The present disclosure can be more clearly understood according to the following detailed description with reference to the drawings.

[0024] Obviously, the drawings described below are only some embodiments of the present disclosure, and a person skilled in the art can obtain other drawings based on these drawings without creative work.

[0025] Figure 1 A schematic diagram showing the electrical architecture of an intelligent (autonomous) driving vehicle according to some embodiments of the present disclosure.

[0026] Figure 2 A schematic diagram showing the appearance and structure of an intelligent (automatic) driving vehicle according to some embodiments of the present disclosure.

[0027] Figure 3 A schematic diagram showing a neural network for processing point clouds according to some embodiments of the present disclosure.

[0028] Figure 4 A schematic diagram showing the structure of a downsampling network according to some embodiments of the present disclosure is shown.

[0029] Figure 5 A schematic diagram showing the structure of an upsampling network according to some embodiments of the present disclosure is shown.

[0030] Figure 6 A schematic diagram showing the structure of a temporal fusion neural network according to some embodiments of the present disclosure.

[0031] Figure 7 A schematic diagram showing the structure of a speed / acceleration / category prediction neural network according to some embodiments of the present disclosure.

[0032] Figure 8 A schematic diagram showing a point cloud processing method according to some embodiments of the present disclosure.

[0033] Fig. 9 A schematic diagram showing a point cloud processing device according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0034] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0035] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0036] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0037] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0038] In addition, the term "and / or" in the present disclosure is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0039] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0040] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0041] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0042] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0043] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0044] In addition, in order to avoid obscuring the present disclosure due to unnecessary details, only the processing steps and / or device structures closely related to at least the scheme according to the present disclosure are shown in the drawings, and other details that are not closely related to the present disclosure are omitted. It should also be noted that similar reference numerals and letters in the drawings indicate similar items, and therefore once an item is defined in one drawing, it does not need to be discussed again for subsequent drawings.

[0045] Figure 1 A schematic diagram showing the electrical architecture of an intelligent (autonomous) driving vehicle according to some embodiments of the present disclosure.

[0046] like Figure 1As shown, the intelligent (automatic) driving vehicle 100 of this embodiment includes, for example, an automatic driving module 110 and a chassis module 120, and may also include a remote monitoring streaming module 130 and a cargo box module 140 as needed. For example, a vehicle with a remote monitoring requirement is provided with a remote monitoring streaming module 130, and a vehicle without a remote monitoring requirement may not be provided with a remote monitoring streaming module 130. For another example, a vehicle with a cargo carrying requirement (such as a truck) is provided with a cargo box module 140, and a vehicle without a cargo carrying requirement (such as a manned car) may not be provided with a cargo box module 140. The intelligent (automatic) driving vehicle may be, for example, an unmanned vehicle, an unmanned delivery vehicle, an unmanned vending vehicle, etc.

[0047] The autonomous driving module 110 includes, as required, one or more of a central processor (Orin or Xavier module) 111, a traffic light recognition camera 112, a front camera 1131, a rear camera 1132, a left camera 1133, a right camera 1134, a laser radar 114, a front blind spot radar 1151, a rear blind spot radar 1152, a left blind spot radar 1153, a right blind spot radar 1154, a positioning module (such as Beidou, GPS, etc.) 116, an inertial navigation unit 117, a switch 118, etc. Each camera can communicate with the autonomous driving module. In order to increase the transmission speed and reduce the wiring harness, GMSL (Gigabit Multimedia Serial Links) link communication can be used. The central processor 111 may be implemented by a general-purpose central processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistors, or other discrete hardware components. The central processor 111 may be configured to perform autonomous driving control.

[0048] The chassis module 120 includes, for example, one or more of a battery 121, a power management device 122, a chassis controller 123, a motor driver 124, a power motor 125, and a communication module 126 as required. The battery 121 provides power for the entire autonomous driving vehicle system. The battery 121 includes a main battery 1211 and a standby battery 1212. When the autonomous driving vehicle is running, the main battery 1211 supplies power to each module of the autonomous driving vehicle. When the autonomous driving vehicle is on standby, the standby battery 1212 supplies power to the central processor 111 and the communication module 126. The power management device 122 converts the output of the battery 121 into different levels of voltage that can be used by each module, and controls power on and off. The chassis controller 123 receives motion instructions issued by the autonomous driving module 110, and controls the steering, forward, backward, braking, etc. of the autonomous driving vehicle. The communication module 126 communicates with the background server, and can realize the remote control of the autonomous driving vehicle by the background operator. The communication module 126 includes a cellular wireless communication device 1261 and a radio frequency communication device 1262. The cellular wireless communication device 1261 communicates using cellular wireless communication technology, such as 2G (second generation), 3G (third generation), 4G (fourth generation) or 5G (fifth generation) cellular wireless communication technology. The radio frequency communication device 1262 communicates using radio frequency communication technology.

[0049] The remote monitoring streaming module 130 includes, as required, one or more of the front monitoring camera 1311, the rear monitoring camera 1312, the left monitoring camera 1313, the right monitoring camera 1314 and the streaming module 132. The streaming module 132 transmits the video data collected by the monitoring cameras 1311-1314 to the backend server for viewing by the backend operator.

[0050] The cargo box module 140 includes, for example, a cargo box 141 as required, which is a cargo carrying device of the autonomous driving vehicle. The cargo box module 140 is also provided with a display interaction module 142 for the autonomous driving vehicle to interact with the user. The user can perform operations such as picking up, depositing, and purchasing goods through the display interaction module 142. The type of cargo box 141 can be changed according to actual needs. For example, in a logistics scenario, the cargo box can include multiple sub-boxes of different sizes, which can be used to load goods for delivery. In a retail scenario, the cargo box can be set as a transparent box so that users can see the products for sale.

[0051] Figure 2 A schematic diagram showing the appearance and structure of an intelligent (automatic) driving vehicle according to some embodiments of the present disclosure. Figure 2 The figure shows the appearance structure of an autonomous driving vehicle that can carry cargo. Autonomous driving vehicles with different functions can have different appearance structures. For example, the appearance structure of an autonomous driving vehicle that can carry people can refer to that of a car. Figure 2As shown, from the current perspective, one can see the chassis 210, cargo box 141, display interaction module 142, right camera 1134, lidar 114, rear blind spot radar 1152, left blind spot radar 1153, right blind spot radar 1154, etc. of the autonomous driving vehicle 200.

[0052] The active safety module of the intelligent driving vehicle of the disclosed embodiment uses the laser wave radar signal as input to sense the surrounding targets and their motion state. Laser radar is denser than millimeter wave radar, can sense small targets well, and has higher vertical resolution, which can separate static obstacles from the background. And based on the scheme proposed in the present disclosure, the speed, acceleration, and category information of the target can be sensed using the time-series laser radar signal, and the speed signal predicted by the two-dimensional speed prediction map has high accuracy in both radial and lateral directions, which can reflect the real motion state of the target and provide rich and high-quality upstream information for the active safety decision-making of the autonomous driving system.

[0053] Figure 3 A schematic diagram of a neural network for processing point clouds according to some embodiments of the present disclosure is shown, and the point cloud processing neural network may exist in the form of a product of a point cloud processing device. Figure 3 As shown, the point cloud processing neural network (point cloud processing device) includes: a feature extraction module 310, which is configured to extract features from the point cloud data of the current frame of the laser radar to obtain a point cloud feature map of the current frame; a time series fusion module 320, which is configured to perform time series fusion processing on the point cloud feature map of the current frame and the fused feature map of the previous frame, that is, to splice the point cloud feature map of the current frame and the fused feature map of the previous frame, and extract time series features to obtain a fused feature map of the current frame; a prediction module 330, which is configured to predict one or more of the speed, acceleration and category of the target according to the fused feature map of the current frame, and obtain a speed prediction map, an acceleration prediction map and a category prediction map.

[0054] The feature extraction module 310 includes, for example, a downsampling network, multiple residual networks, upsampling networks with different step sizes, and a splicing unit. The downsampling network downsamples the point cloud data of the current frame of the laser radar and outputs the first feature map of the current frame. The cascaded multiple residual networks downsample the first feature map and output multiple second feature maps with different downsampling rates. Multiple upsampling networks with different step sizes upsample the multiple second feature maps with different downsampling rates respectively and output multiple third feature maps of the same size. The splicing unit can splice multiple third feature maps and output the point cloud feature map of the current frame.

[0055] An example of feature extraction module 310 is listed below. Figure 3As shown in the figure, the point cloud information of the current frame (time T) is voxelized to form voxelized point cloud data as input data. The input data is recorded as [H, W, D], where H, W, and D represent height, width, and depth, respectively. The input data first passes through a downsampling network with a step size of 2 to reduce the feature size and reduce the amount of calculation for subsequent networks. The feature map output by the downsampling network is recorded as [H / 2, W / 2, C0], where the height and width are reduced to 1 / 2 of the original, and C0 represents the channel. Figure 4 As shown in the figure, the structure of the downsampling network is, for example, a convolutional layer with a stride of N (in this example, N=2) (the convolutional layer performs downsampling), with a batch normalization layer (BatchNorm) and a ReLU (Rectified LinearUnit) activation function layer. Then, the data output by the downsampling network will pass through four consecutive residual networks to extract deeper features. The convolution stride of these four residual networks is, for example, 2, and they output feature maps with downsampling rates of 4 / 8 / 16 / 32, respectively, denoted as [H / 4,W / 4,C1], [H / 8,W / 8,C2], [H / 16,W / 16,C3], [H / 32,W / 32,C4], with the height and width reduced to 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original, and C1C2C3C4 represent channels respectively. The residual network introduces cross-layer connections, and the input signal is directly added to the output of the residual network, making it easier for the network to pass gradients during back propagation, thereby solving the gradient vanishing problem in deep network training. Subsequently, the four feature maps of the four residual networks will pass through an upsampling network with a step size of 1 / 2 / 4 / 8, and output four feature maps of the same size with a downsampling rate of 4, all recorded as [H / 4, W / 4, C / 4], with a height and width of 1 / 4 of the original, H, W, C represent height, width and channel respectively. Figure 5 As shown in the figure, the structure of the upsampling network is, for example: a deconvolution layer (the deconvolution layer performs upsampling) with a stride of N (in this example, N=2), and a batch normalization layer and a ReLU activation function layer. These four upsampled feature maps of the same size represent the feature information at different network depths. Among them, the shallow features mainly focus on the local details and low-level features of the image. These features are usually closely related to the pixels of the image, including fine-grained information such as color, texture, edges, and corners. The deep features focus on global and high-level features and are more abstract and complex feature representations. These features are usually related to the semantic information of the image. Finally, through the splicing unit, the multiple feature maps obtained after upsampling are spliced ​​in the channel dimension to obtain the point cloud feature map of the current frame, which is recorded as [H / 4,W / 4,C].

[0056] The temporal fusion module 320 includes, for example, a coordinate system alignment unit, a splicing unit, and a temporal fusion neural network, and may also include an upsampling network. The coordinate system alignment unit aligns the fusion feature map of the previous frame to the coordinate system of the current frame according to the coordinate transformation matrix between the previous frame and the current frame. The splicing unit splices the fusion feature map of the aligned previous frame with the point cloud feature map of the current frame, and splices them in the channel dimension. The temporal fusion neural network extracts temporal features from the spliced ​​feature map to obtain a fusion feature map of the current frame. The fusion feature map of the current frame includes the temporal information of each historical frame before the current frame, and the fusion feature map of the previous frame includes the temporal information of each historical frame before the previous frame. Among them, the fusion feature map of the initial frame is obtained by extracting the temporal features from the point cloud feature map of the initial frame using the temporal fusion neural network. On this basis, the fusion feature maps of each frame can be obtained through iteration. By introducing the temporal features, the target motion state can be better perceived. The upsampling network performs upsampling processing on the fusion feature map of the current frame to obtain a fusion feature map with the same size as the point cloud data of the current frame.

[0057] An example of the time series fusion module 320 is listed below. Figure 3 As shown, the fused feature map (size [H / 4, W / 4, C]) of the point cloud of the previous frame (T-1 moment) is warped and aligned to the coordinate system of the current frame (T moment) according to the coordinate transformation matrix between the two frames, and then spliced ​​with the point cloud feature map (size [H / 4, W / 4, C]) of the current frame output by the feature extraction module 310, and then input into the temporal fusion neural network, which compares the point cloud features of the two frames, mines the temporal features, and outputs the fused feature map (size [H / 4, W / 4, C]) of the current frame. Figure 6 As shown in FIG. 1 , the temporal fusion neural network includes multiple cascaded network blocks (e.g., 3), each of which includes a convolution layer (with a step size of 1), a batch normalization layer, and an activation layer, such as a ReLU activation function layer. After the fused feature map of the current frame (with a size of [H / 4, W / 4, C]) is upsampled four times (with a step size of 4), a fused feature matrix (with a size of [H, W, C]) with the same size as the original image is obtained.

[0058] The prediction module 330, for example, includes a speed prediction neural network, an acceleration prediction neural network, and a category prediction neural network. The speed prediction neural network, the acceleration prediction neural network, and the category prediction neural network process the fused feature map of the current frame, respectively, and output a speed prediction map, an acceleration prediction map, and a category prediction map, respectively, whose sizes are the same as the original image (the original point cloud voxel grid). The dimension of the speed prediction map [H, W, 2] is 2, which represents the radial component and the lateral component of the target's speed. In the vehicle driving plane, the radial direction refers to the vehicle driving direction, that is, the direction of the front of the vehicle, and the lateral direction is perpendicular to the vehicle driving direction, that is, the vehicle body width direction. The dimension of the acceleration prediction map [H, W, 1] is 1, which represents the rate of change of the target's speed. The dimension of the category prediction map [H, W, 1] is 1, which represents the one-dimensional index of the target's category. Figure 7 As shown, the speed / acceleration / category prediction neural network includes multiple cascaded network blocks (such as 2), each network block includes a convolution layer (with a step size of 1), a batch normalization layer and an activation layer, such as a ReLU activation function layer, and the last network block cascades a convolution layer (with a step size of 1).

[0059] Point cloud processing neural networks are usually trained and then used for prediction. During training, the velocity prediction map and acceleration prediction map can use the L1 loss function and use the true value for supervision, and the category prediction map can use the Focal Loss loss function for supervised learning. The coefficients of the three loss functions can be set, for example, 1.0, 2.0, and 4.0 respectively. Under this coefficient configuration, the velocity, acceleration, and category information can be learned at the same time, which is conducive to the convergence of the point cloud processing neural network.

[0060] Based on the point cloud processing neural network, the laser radar point cloud can be processed to obtain information such as the speed, acceleration and category of the targets around the intelligent driving vehicle. Among them, the categories of the targets include, but are not limited to, vehicles, pedestrians, etc.

[0061] Figure 8 A schematic diagram showing a point cloud processing method according to some embodiments of the present disclosure is shown. Figure 8 As shown, the point cloud processing method of this embodiment includes the following steps.

[0062] In step 810, feature extraction is performed on the point cloud data of the current frame of the laser radar to obtain a point cloud feature map of the current frame.

[0063] This step can be implemented, for example, using the feature extraction module 310. Using a downsampling network, downsample the point cloud data of the current frame of the laser radar to obtain the first feature map of the current frame; using multiple cascaded residual networks, downsample the first feature map to obtain multiple second feature maps with different downsampling rates output by each residual network; using multiple upsampling networks with different step sizes, upsample the multiple second feature maps with different downsampling rates to obtain multiple third feature maps of the same size; splice the multiple third feature maps to obtain the point cloud feature map of the current frame. In addition, using an upsampling network, upsample the fused feature map of the current frame to obtain a fused feature map of the same size as the point cloud data of the current frame.

[0064] In step 820, the point cloud feature map of the current frame and the fused feature map of the previous frame are subjected to temporal fusion processing to obtain the fused feature map of the current frame.

[0065] In some embodiments, the point cloud feature map of the current frame and the fused feature map of the previous frame are spliced, and the temporal features are extracted to obtain the fused feature map of the current frame.

[0066] This step can be implemented, for example, using the temporal fusion module 320. According to the coordinate transformation matrix between the previous frame and the current frame, the fused feature map of the previous frame is aligned to the coordinate system of the current frame, and then spliced ​​with the point cloud feature map of the current frame; the temporal fusion neural network is used to extract the temporal features from the spliced ​​feature map to obtain the fused feature map of the current frame. Among them, the fused feature map of the initial frame is obtained by extracting the temporal features from the point cloud feature map of the initial frame using the temporal fusion neural network.

[0067] In step 830, one or more of the speed, acceleration and category of the target is predicted based on the fused feature map of the current frame to obtain a speed prediction map, an acceleration prediction map and a category prediction map.

[0068] This step can be implemented, for example, using the prediction module 330. The fused feature map of the current frame is processed using the speed prediction neural network, the acceleration prediction neural network, and the category prediction neural network, respectively, to obtain a speed prediction map, an acceleration prediction map, and a category prediction map. The dimension of the speed prediction map is 2, representing the radial component and the lateral component of the target's speed. The dimension of the acceleration prediction map is 1, representing the rate of change of the target's speed. The dimension of the category prediction map is 1, representing the one-dimensional index of the target's category.

[0069] The time-series LiDAR signal is used to simultaneously sense the speed, acceleration, and category information of the target. The speed signal predicted by the two-dimensional speed prediction map has high accuracy in both radial and lateral directions, which can reflect the real motion state of the target and provide rich and high-quality upstream information for the active safety decision-making of the autonomous driving system. In addition, LiDAR is denser than millimeter-wave radar, can well sense small targets, and has a higher vertical resolution, which can separate static obstacles from the background.

[0070] Fig. 9 A schematic diagram of a point cloud processing device according to some embodiments of the present disclosure is shown. The point cloud processing device 900 can be used in an intelligent (automatic) driving vehicle 100 as a part thereof.

[0071] like Fig. 9 As shown, the point cloud processing device 900 of this embodiment includes: a memory 910 and a processor 920 coupled to the memory 910 , and the processor 920 is configured to execute the point cloud processing method in any of the aforementioned embodiments based on instructions stored in the memory 910 .

[0072] The point cloud processing device 900 may further include an input / output interface 930 , a network interface 940 , a storage interface 950 , etc. These interfaces 930 , 940 , 950 , the memory 910 , and the processor 920 may be connected, for example, via a bus 960 .

[0073] The memory 910 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs.

[0074] The processor 920 may be implemented by a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistors or other discrete hardware components.

[0075] Among them, the input and output interface 930 provides a connection interface for input and output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 940 provides a connection interface for various networked devices. The storage interface 950 provides a connection interface for external storage devices such as SD cards and USB flash drives. The bus 960 can use any bus structure among a variety of bus structures. For example, the bus structure includes but is not limited to the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, and the Peripheral Component Interconnect (PCI) bus.

[0076] An intelligent driving vehicle includes: a laser radar and a point cloud processing device. The point cloud processing device is configured to execute a point cloud processing method on point cloud data from the laser radar.

[0077] It should be understood by those skilled in the art that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure may take the form of a computer program product implemented on one or more (non-transient) computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, cloud storage, etc.) containing computer program code. A computer program product should be understood as a software product that implements its solution mainly through a computer program.

[0078] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0079] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0080] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0081] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A point cloud processing method, characterized in that: include: Perform feature extraction on the point cloud data of the current frame of the laser radar to obtain a point cloud feature map of the current frame; Perform temporal fusion processing on the point cloud feature map of the current frame and the fused feature map of the previous frame to obtain the fused feature map of the current frame; According to the fusion feature map of the current frame, one or more of the speed, acceleration and category of the target is predicted to obtain a speed prediction map, an acceleration prediction map and a category prediction map.

2. The point cloud processing method according to claim 1, characterized in that: The point cloud feature map of the current frame and the fused feature map of the previous frame are subjected to temporal fusion processing, and the fused feature map of the current frame is obtained including: The point cloud feature map of the current frame and the fused feature map of the previous frame are concatenated, and the temporal features are extracted to obtain the fused feature map of the current frame.

3. The point cloud processing method according to claim 2, characterized in that: The point cloud feature map of the current frame is concatenated with the fused feature map of the previous frame, and the temporal features are extracted to obtain the fused feature map of the current frame, which includes: According to the coordinate transformation matrix between the previous frame and the current frame, the fused feature map of the previous frame is aligned to the coordinate system of the current frame, and then spliced ​​with the point cloud feature map of the current frame; The temporal features are extracted from the spliced ​​feature maps using the temporal fusion neural network to obtain the fused feature map of the current frame.

4. The point cloud processing method according to claim 1, characterized in that: The fused feature map of the current frame includes the timing information of each historical frame before the current frame, and the fused feature map of the previous frame includes the timing information of each historical frame before the previous frame.

5. The point cloud processing method according to claim 1, characterized in that: The fusion feature map of the initial frame is obtained by extracting the temporal features of the point cloud feature map of the initial frame using a temporal fusion neural network.

6. The point cloud processing method according to claim 1, characterized in that: According to the fusion feature map of the current frame, predicting one or more of the speed, acceleration and category of the target, obtaining the speed prediction map, acceleration prediction map and category prediction map includes: The speed prediction neural network, the acceleration prediction neural network and the category prediction neural network are used to process the fused feature map of the current frame respectively to obtain a speed prediction map, an acceleration prediction map and a category prediction map.

7. The point cloud processing method according to claim 6, characterized in that: The dimension of the velocity prediction map is 2, representing the radial component and lateral component of the target's velocity; or, the dimension of the acceleration prediction map is 1, representing the rate of change of the target's velocity; or, the dimension of the category prediction map is 1, representing the one-dimensional index of the target's category.

8. The point cloud processing method according to claim 1, characterized in that: The feature extraction is performed on the point cloud data of the current frame of the laser radar, and the point cloud feature map of the current frame is obtained, including: Using the downsampling network, downsample the point cloud data of the current frame of the lidar to obtain the first feature map of the current frame; Downsampling the first feature map using a plurality of cascaded residual networks to obtain a plurality of second feature maps output by the residual networks with different downsampling rates; Using multiple upsampling networks with different step sizes, upsampling the multiple second feature maps with different downsampling rates respectively to obtain multiple third feature maps with the same size; Multiple third feature maps are concatenated to obtain the point cloud feature map of the current frame.

9. The point cloud processing method according to claim 1, characterized in that: Also includes: Before prediction, the upsampling network is used to upsample the fused feature map of the current frame to obtain a fused feature map with the same size as the point cloud data of the current frame.

10. The point cloud processing method according to claim 3, characterized in that: The temporal fusion neural network includes a convolution layer, a batch normalization layer and an activation layer.

11. The point cloud processing method according to claim 6, characterized in that: The speed prediction neural network, the acceleration prediction neural network and the category prediction neural network include a first convolutional layer, a batch normalization layer, an activation layer and a second convolutional layer.

12. A point cloud processing device, characterized in that: include: The feature extraction module is configured to extract features from the point cloud data of the current frame of the laser radar to obtain a point cloud feature map of the current frame; The temporal fusion module is configured to perform temporal fusion processing on the point cloud feature map of the current frame and the fused feature map of the previous frame to obtain the fused feature map of the current frame; The prediction module is configured to predict one or more of the speed, acceleration and category of the target according to the fusion feature map of the current frame, and obtain a speed prediction map, an acceleration prediction map and a category prediction map.

13. A point cloud processing device, comprising: Memory; And a processor coupled to the memory, characterized in that the processor is configured to execute the point cloud processing method according to any one of claims 1-11 based on instructions stored in the memory.

14. An intelligent driving vehicle, characterized in that: include: A laser radar and a point cloud processing device, wherein the point cloud processing device is configured to execute the point cloud processing method described in any one of claims 1 to 11 on the point cloud data from the laser radar.

15. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the steps of the point cloud processing method described in any one of claims 1-11 are implemented.

16. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the point cloud processing method described in any one of claims 1-11 are implemented.

Images