Vehicle-mounted laser radar system with neural network-based double-density point cloud generator

Abstract

A vehicle system includes a lidar system that obtains an initial point cloud by implementing a first neural network and based on the initial point cloud and obtains a double-density point cloud. The double-density point cloud is produced by reducing a point density of the initial point cloud outside a region of interest (ROI). Processing the double-density point cloud produces detection results that indicate any objects in a field of view (FOV) of the lidar system. A controller obtains the detection results from the lidar system and controls operation of the vehicle based on the detection results.

Description

Technical Field

[0001] This subject matter discloses a vehicle lidar system with a neural network-based dual-density point cloud generator. Background Technology

[0002] Vehicles (e.g., cars, trucks, construction equipment, farm equipment) increasingly rely on sensors to provide information about the vehicle and its environment. Exemplary types of sensors that provide information about the vehicle's surroundings include radio detection and ranging (radio radar) systems, light detection and ranging (LiDAR) systems, and cameras. LiDAR systems provide a point cloud representation of features in the LiDAR system's field of view. As point cloud density increases, object detection in the field of view improves, but processing time and complexity also increase with density. Therefore, it is desirable to provide vehicle LiDAR systems with a neural network-based dual-density point cloud generator. Summary of the Invention

[0003] In one exemplary embodiment, the vehicle system includes a lidar system to obtain an initial point cloud and a dual-density point cloud by implementing a first neural network and based on the initial point cloud. The dual-density point cloud is generated by reducing the point density of the initial point cloud outside the region of interest (ROI). Processing the dual-density point cloud produces detection results indicating any object within the lidar system's field of view (FOV). The controller obtains the detection results from the lidar system and controls the operation of the vehicle based on the detection results.

[0004] In addition to one or more features described herein, the lidar system implements a first neural network to define the ROI within the FOV that generates the dual-density point cloud. The ROI is a region with a fixed area having a center selected from a set of potential centers based on the output of the first neural network.

[0005] In addition to one or more features described in this paper, the lidar system implements a second neural network to output detection results based on a dual-density point cloud.

[0006] In addition to one or more features described herein, the second neural network includes encoder and decoder levels that provide pointwise feature vectors such that each feature vector in the pointwise feature vectors is associated with each point of the dual-density point cloud.

[0007] In addition to one or more features described in this paper, the first neural network is a deep Q-network (DQN), which obtains pointwise feature vectors from the encoder and decoder levels of the second neural network.

[0008] In addition to one or more features described herein, training DQN involves comparing detection results obtained with a dual-density point cloud with ground-based detection results to generate a certain number of true positives, and comparing a second detection result obtained by reducing the point density of the initial point cloud across the entire FOV with ground-based detection results to generate a second number of true positives.

[0009] In addition to one or more features described herein, training DQN includes obtaining a reward by comparing the given number of true positives with the second number of true positives, and training DQN includes maximizing the reward.

[0010] In addition to one or more features described in this paper, DQN outputs a matrix that shows the predicted reward corresponding to each of the set of potential centers.

[0011] In addition to one or more features described herein, DQN outputs a matrix that indicates the probability of a positive reward corresponding to each of the said set of potential centers.

[0012] In addition to one or more features described in this paper, training a DQN involves obtaining a loss as the difference between the reward and the predicted reward provided by the DQN, and training a DQN involves minimizing the loss.

[0013] In another exemplary embodiment, a method includes obtaining an initial point cloud and implementing a first neural network to obtain a dual-density point cloud based on the initial point cloud. The dual-density point cloud is generated by reducing the point density of the initial point cloud outside the region of interest (ROI). The method also includes processing the dual-density point cloud to obtain detection results indicating any object within the field of view (FOV) of the lidar system.

[0014] In addition to one or more features described herein, implementing the first neural network results in defining the ROI within the FOV, which produces the dual-density point cloud, the ROI being a region of fixed area having a center selected from a set of potential centers based on the output of the first neural network.

[0015] In addition to one or more features described in this paper, the method also includes implementing a second neural network to output detection results based on a dual-density point cloud.

[0016] In addition to one or more features described herein, implementing a second neural network includes implementing encoder and decoder levels to provide pointwise feature vectors such that each feature vector in the pointwise feature vectors is associated with each point of the dual-density point cloud.

[0017] In addition to one or more features described in this paper, the first neural network is a deep Q-network (DQN), and implementing DQN involves obtaining pointwise feature vectors from the encoder and decoder levels of the second neural network.

[0018] In addition to one or more features described herein, the method further includes: training DQN by comparing detection results obtained using a dual-density point cloud with ground-based detection results to generate a certain number of true positives, and by comparing a second detection result obtained by reducing the point density of the initial point cloud of the entire FOV with ground-based detection results to generate a second number of true positives.

[0019] In addition to one or more features described herein, training a DQN includes obtaining a reward by comparing the given number of true positives with the second number of true positives, and training a DQN includes maximizing the reward, and training a DQN also includes obtaining a loss as the difference between the reward and the predicted reward provided by the DQN, and training a DQN includes minimizing the loss.

[0020] In addition to one or more features described herein, implementing DQN involves outputting a matrix that shows the predicted reward corresponding to each of the set of potential centers.

[0021] In addition to one or more features described herein, implementing DQN involves outputting a matrix that indicates the probability of a positive reward corresponding to each of the said set of potential centers.

[0022] In addition to one or more features described herein, the method also includes a vehicle controller obtaining detection results from a lidar system and controlling the operation of the vehicle based on the detection results.

[0023] The above-described features and advantages, as well as other features and advantages of this disclosure, will be readily understood when taken in conjunction with the accompanying drawings and the following detailed description. Attached Figure Description

[0024] Other features, advantages, and details appear only by way of example in the following detailed description, which refers to the accompanying drawings, wherein:

[0025] Figure 1 It is a block diagram of a vehicle having a neural network-based dual-density point cloud generator according to one or more embodiments;

[0026] Figure 2 An exemplary field of view with a neural network-based dual-density point cloud is shown according to one or more embodiments;

[0027] Figure 3This is a processing flow of a method for generating dual-density point clouds based on neural networks in a vehicle's LiDAR system, according to one or more embodiments; and

[0028] Figure 4 Is Figure 3 The process flow shown illustrates the process flow of various aspects of the method for training a neural network. Detailed Implementation

[0029] The following description is exemplary in nature only and is not intended to limit this disclosure, its application, or use. It should be understood that throughout the drawings, corresponding reference numerals denote the same or corresponding parts and features.

[0030] As previously mentioned, a lidar system is one of the sensors used to obtain information about the environment surrounding a vehicle. It should also be noted that the density of the point cloud obtained from the lidar system affects performance and processing time. Specifically, performance increases with increasing point cloud density, but so do processing time and bandwidth costs. Embodiments of the systems and methods detailed herein relate to a vehicle lidar system with a neural network-based dual-density point cloud generator. The neural network is part of the lidar system, not a neural network that performs post-processing on the output point cloud from the lidar system.

[0031] Dual-density point clouds refer to point clouds with regions of lower density and regions of higher density. Specifically, the initial output point density is retained only for a portion of the field of view (FOV) (i.e., the region of interest (ROI)). In the rest of the FOV, the density is reduced to a predetermined percentage. The predetermined percentage of the original point density in the non-ROI region of the FOV is based on the understanding that even if processing time does increase, detection performance will not increase by retaining more than the predetermined percentage of points in the non-ROI region. The dual-density approach helps to achieve higher processing time and bandwidth only in the ROI, corresponding to higher performance, thereby reducing the overall processing time and bandwidth requirements of the LiDAR system. For example, to simulate human vision, a lower point cloud density (i.e., lower resolution) can be provided in the peripheral regions of the field of view. As detailed, the size of the ROI region (where the point cloud density is relatively higher than in other regions of the FOV) can be fixed. A neural network is used to determine where the ROI should be centered within the FOV.

[0032] According to an exemplary embodiment, Figure 1 This is a block diagram of a vehicle 100 with a neural network-based dual-density point cloud generator. Figure 1The exemplary vehicle 100 shown is an automobile 101. Vehicle 100 includes a lidar system 110 and may also include other sensors 130 (e.g., radio radar systems, cameras). The number and location of the lidar system 110 and the number and location of the other sensors 130 are not limited. Figure 1 The exemplary illustrations in the figure are limited. The lidar system 110 includes a controller 120 implementing a neural network-based dual-density point cloud generator according to one or more embodiments.

[0033] Specifically, the controller 120 of the lidar system 110 determines the location of the ROI 220 within the FOV 210 by implementing a neural network, such as... Figure 2 As shown. As previously described, the higher density point cloud is maintained only within ROI 220, and a dual-density point cloud is generated by reducing the point density in the region of FOV 210 outside ROI 220. According to different exemplary embodiments, the neural network implemented by controller 120 outputs different indicators of the location of ROI 220 within FOV 210.

[0034] Vehicle 100 includes a vehicle controller 140 that can obtain information from lidar system 110 and other sensors 130 to control aspects of autonomous or semi-autonomous operation of vehicle 100. For example, semi-autonomous operation such as adaptive cruise control or automatic braking can be implemented by vehicle controller 140 based on information from lidar system 110 and / or other sensors 130. Both lidar system 110 controller 120 and vehicle controller 140 may include processing circuitry, which may include application-specific integrated circuits (ASICs), electronic circuitry, processors (shared, dedicated, or grouped) executing one or more software or firmware programs, memory, combinational logic circuitry, and / or other suitable components providing the described functionality.

[0035] Figure 2 An exemplary FOV 210 with a neural network-based dual-density point cloud is shown according to one or more embodiments. The FOV 210 of the lidar system 110 is... Figure 2 The data is shown as discretized into a grid. As indicated, the horizontal span (HFOV) of FOV 210 is 90 degrees, and the vertical span (VFOV) of FOV 210 is 50 degrees. Figure 2An exemplary ROI 220 is shown. Based on the exemplary grid, the horizontal grid span (Rh) of ROI 220 is 9 cells, and the vertical grid span (Rv) of ROI 220 is 5 cells. The center 235 of the exemplary ROI 220 is labeled. Center 235 is part of a set of potential centers 230, which is a subgroup of all cells representing FOV 210, which can serve as the center 235 of ROI 220 based on the fixed dimensions (i.e., RH and RV) of ROI 220.

[0036] In other words, selecting a different center 235 from this set of potential centers 230 will cause the ROI 220 to shift within the FOV 210. The cell representation of the complete FOV 210 constituting this set of potential centers 230 is constrained by a fixed area of ​​the ROI 220. That is, this set of potential centers 230 is selected such that the ROI 220 centered at any one of these potential centers 230 will not fall outside the FOV 210. According to one or more embodiments and as detailed herein, a neural network is implemented by the controller 120 of the LiDAR system 110 to select a center 235 from the set of potential centers 230, thereby defining the position of the ROI 220 within the FOV 210.

[0037] Figure 3 This is a processing flow of a method 300 for generating a neural network-based dual-density point cloud in a lidar system 110 of a vehicle 100, according to one or more embodiments. At block 310, the process includes obtaining an initial point cloud P generated within the lidar system 110. t Initial point cloud P t The point density is uniform throughout FOV 210. At box 320, the initial point cloud P in the region outside ROI 220 is reduced. t The density can produce dual-density point clouds. The percentage reduction in point density in the FOV210 region outside ROI 220 can be fixed (e.g., 20%). This dual-density point cloud is further processed using a neural network according to a known process (at boxes 330-360). To obtain the detection result D, which indicates the detected objects and lanes around vehicle 100. For example, the neural network could be a region-based convolutional neural network (R-CNN).

[0038] At box 330, implementing the encoder / decoder stage of the neural network results in a dual-density point cloud. The points are mapped to a lower-level representation. Then, the decoder layer performs upsampling and generates a point-wise feature vector X.t Point-by-point vectors refer to vectors used in dual-density point clouds. Each point is used to generate a vector. For example, for a dual-density point cloud. For each of the N points in the matrix, an N×M matrix can be generated, or in other words, an M-length vector (e.g., M = 128) can be generated as a point-by-point feature vector X for each point. t Part of the process. At box 340, generating a 3D proposal means classifying each point as either a foreground or background point. At box 340, 3D regions are generated as proposals for objects associated with each foreground point. At box 350, the process performed by the neural network includes pooling the point cloud regions. Region pooling refers to combining 3D region proposals corresponding to the same object. At box 360, refining the 3D bounding boxes results in the detection of objects and lanes within the FOV 210. The detection results D from box 360 can be provided to the vehicle controller 140 to influence the operation of the vehicle 100.

[0039] At box 370, implementing another neural network refers to implementing a Deep Q-Network (DQN). The pointwise feature vector X from the encoder / decoder... t (At box 330) is also provided to DQN, such as Figure 3 As shown. DQN estimates the Q-value of each possible action a based on the weights θ. The result is matrix A. t Its size is the same as the unit size of the set of potential centers 230 (e.g., according to...). Figure 2 (Exemplary case shown in 19×11). That is, each action a is to select one of the potential centers 230 in this set as the center 235 of ROI 220. At box 380, determining ROI 220 means determining the next point cloud P output by the LiDAR system 110 at box 310 for the next frame. t+1 The ROI is 220 at the center of 235.

[0040] According to an exemplary embodiment, matrix A is output from block 370. t This includes the predicted reward associated with each location within this set of potential centers 230. In this case, determining the ROI 220 at box 380 involves using matrix A. t Determine which of the potential centers 230 in this set is associated with the highest predicted reward. According to another exemplary embodiment, matrix A is output from box 370. t This includes the probability associated with each location within this set of potential centers 230. In this case, the next point cloud P is determined at box 380. t+1 The ROI 220 involves the calculation based on matrix A. tThe system determines which of the 230 potential centers is associated with the highest probability of generating a positive reward. According to this embodiment, known as policy gradient, DQN implements an additional softmax layer to obtain the probability of generating a positive reward. (See reference...) Figure 4 Further discussion on rewards.

[0041] According to an exemplary embodiment, the DQN implemented at box 370 can be simplified by splitting the x and y dimensions. That is, instead of a single Q value for each grid point in FOV 210 that can serve as the center 235 of ROI 220, Qx and Qy can be determined separately at box 370 via two branches of the DQN. The DQN can then output Ax. t and Ay t .

[0042] Figure 4 Is Figure 3 The training process of DQN implemented at box 370 involves various aspects of the processing flow. Training DQN at box 370 involves comparing the detection result D (output from box 360) obtained using ROI 220 (at box 320) with the ground truth, and also considering the results if dual-density point clouds are not used. (That is, the density of the initial point cloud Pt decreases uniformly across the entire FOV210) The obtained detection result D' will be compared with the ground reality. In other words, using a dual-density point cloud The improvement of the obtained detection result D compared to the detection result D' obtained without maintaining the higher point density in ROI 220 was used to train DQN.

[0043] At box 410, refer to Figure 3 The detection result D is obtained at the output of box 360 in detail. At box 420, obtaining the detection result D' involves using a reference. Figure 3 The same R-CNN is being discussed. However, the input to the R-CNN (implemented at box 330-360) is a dual-density point cloud. Instead, the input is an initial point cloud P with uniformly decreasing density across the entire FOV 210. t At box 430, the test result D is compared with the ground condition to generate multiple true positives (TPs). A true positive is the number of test objects in test result D that match the ground condition. At box 430, the test result D' is compared with the ground condition to generate multiple true positives (TPs) in test result D'.

[0044] At frame 440, a dual-density point cloud will be used. The true positive TP obtained is compared with the true positive TP' obtained using a uniformly reduced point density to provide a reward for DQN. For example, if the true positive TP exceeds the true positive TP' (i.e., the dual-density point cloud), then DQN provides a reward. If a more accurate detection result (D) is produced, the reward can be positive. If true positive TP equals true positive TP' (i.e., double-density point cloud), the reward is positive. If the accuracy is the same as that produced by uniformly decreasing point density, then the reward can be zero. If the true positive TP is less than the true positive TP' (i.e., double-density point cloud), the reward can be zero. If a detection result (D) is less accurate than that produced using a uniformly decreasing point density, then the reward can be negative. Training a DQN to maximize the reward is called reinforcement learning.

[0045] This is the reward discussed in the output of reference box 370. As previously described, according to one exemplary embodiment, the output of box 370 may be a reward predicted based on selecting each of the set of potential centers 230 as center 235 of ROI 220. According to another exemplary (policy gradient) embodiment, the output of box 370 may be based on the probability that the reward for each of the set of potential centers 230 selected as center 235 of ROI 220 is positive.

[0046] In addition to rewards, loss can also be used when training DQN. Loss is derived from the comparison between the predicted reward and the actual reward. Therefore, instead of using detection results D and D' (as discussed for reward determination), the predicted reward at the output of DQN is compared with the actual reward. The greater the difference between the predicted and actual rewards, the greater the loss attributed to DQN during training. Therefore, in addition to maximizing the reward, the training process also seeks to minimize the loss.

[0047] While the foregoing disclosure has been described with reference to exemplary embodiments, those skilled in the art will understand that various changes can be made and elements can be substituted with equivalents without departing from its scope. Furthermore, many modifications can be made to adapt particular situations or materials to the teachings of this disclosure without departing from the basic scope of this disclosure. Therefore, this disclosure is not intended to be limited to the specific embodiments disclosed, but will include all embodiments falling within its scope.

Claims

1. A vehicle comprising: A lidar system is configured to acquire an initial point cloud, to acquire a dual-density point cloud by implementing a first neural network and based on the initial point cloud, wherein the dual-density point cloud is generated by reducing the point density of the initial point cloud outside the region of interest (ROI), and the lidar system is configured to process the dual-density point cloud to obtain detection results indicating any object in the field of view (FOV) of the lidar system. as well as The controller is configured to obtain the detection results from the lidar system and control the operation of the vehicle based on the detection results; Wherein, the first neural network is a deep Q-network, or DQN, which obtains pointwise feature vectors from the encoder and decoder levels of the second neural network. Training the DQN includes comparing the detection results obtained with the dual-density point cloud with ground-based detection results to generate a certain number of true positives, and comparing the second detection results obtained by reducing the point density of the initial point cloud of the entire FOV with the ground-based detection results to generate a second number of true positives. Training the DQN includes obtaining a reward by comparing the certain number of true positives with the second number of true positives, and training the DQN includes maximizing the reward. The DQN is configured to output a matrix that indicates the predicted reward corresponding to each potential center in a set of potential centers; or the DQN is configured to output a matrix that indicates the probability of a positive reward corresponding to each potential center in the set of potential centers. Training the DQN includes obtaining a loss as the difference between the reward and the predicted reward provided by the DQN, and training the DQN includes minimizing the loss.

2. The vehicle of claim 1, wherein, The lidar system is configured to implement the first neural network to define the region of interest (ROI) within the field of view (FOV) that generates the dual-density point cloud. The ROI is a region with a fixed area and a center selected from a set of potential centers based on the output of the first neural network.

3. The vehicle of claim 2, wherein, The lidar system is configured to implement a second neural network to output the detection results based on the dual-density point cloud.

4. The vehicle of claim 3, wherein, The second neural network includes an encoder and a decoder stage configured to provide pointwise feature vectors such that each feature vector in the pointwise feature vectors is associated with each point of the dual-density point cloud.

5. A method comprising: Initial point cloud is obtained using a lidar system; The lidar system is used to implement a first neural network to obtain a dual-density point cloud based on the initial point cloud, wherein the dual-density point cloud is generated by reducing the point density of the initial point cloud outside the region of interest (ROI); and The dual-density point cloud is processed to obtain detection results indicating any object in the field of view (FOV) of the lidar system; Wherein, the first neural network is a deep Q-network, i.e., DQN, and implementing the DQN includes obtaining pointwise feature vectors from the encoder and decoder levels of the second neural network; training the DQN is based on comparing the detection results obtained using the dual-density point cloud with ground-based detection results to generate a certain number of true positives, and comparing the second detection results obtained by reducing the point density of the initial point cloud of the entire FOV with the ground-based detection results to generate a second number of true positives; training the DQN includes obtaining a reward by comparing the certain number of true positives with the second number of true positives; training the DQN includes maximizing the reward; training the DQN also includes obtaining a loss as the difference between the reward and the predicted reward provided by the DQN; training the DQN includes minimizing the loss; implementing the DQN includes outputting a matrix indicating the predicted reward corresponding to each potential center in a set of potential centers; or implementing the DQN includes outputting a matrix indicating the probability of a positive reward corresponding to each potential center in a set of potential centers.

6. The method of claim 5, wherein, Implementing the first neural network results in defining the ROI within the FOV, which produces the dual-density point cloud, where the ROI is a region with a fixed area having a center selected from a set of potential centers based on the output of the first neural network.

7. The method of claim 6, further comprising implementing a second neural network to output the detection result based on the dual-density point cloud, wherein implementing the second neural network includes implementing an encoder and decoder level to provide point-by-point feature vectors such that each feature vector in the point-by-point feature vectors is associated with each point of the dual-density point cloud.

8. The method according to claim 5 further includes a vehicle controller obtaining the detection result from the lidar system and controlling the operation of the vehicle based on the detection result.

Images