Vehicle sensor point cloud probability density function estimation based on visual sensor data

Abstract

Techniques for using machine learning to generate vehicle position sensor data from vision sensor data are disclosed. By using a limited amount of vehicle position sensor data and vision sensor data, a deep learning network may be trained to produce an estimated vehicle position sensor point cloud distribution individually from, for example, vision sensor data. Using a deep learning network trained in this way, a vehicle with limited sensor functionality or without sensor functionality may be equipped with a camera to produce an estimated vehicle position sensor point cloud distribution. These estimated vehicle position sensor point cloud distributions may then be compared to universal sensor point cloud distributions to improve detection of vehicles, environmental objects, and ghosting objects, then used to improve vehicle safety and / or produce enhanced sensor data through vehicle control or driver notification.

Description

Technical Field

[0001] The present invention relates to a method for estimating a probability density function of a vehicle sensor point cloud based on visual sensor data. Background Art

[0002] Radar data is often used to improve vehicle safety. For example, radar-based object avoidance and collision avoidance are often utilized in vehicles to, for example, notify the driver of nearby objects when the driver is parking, or even automatically apply the vehicle's brakes when an impending collision is detected. However, radar data can be inaccurate and performance often degrades in cluttered environments, which can lead to false positives due to ghost object detection, such as reflections from actual objects caused by static guardrails on the street. Therefore, it is necessary to use machine learning techniques to improve radar data derived from these sensors. One difficulty with using machine learning to improve radar data is that machine learning techniques typically require a large amount of training data that may be difficult to obtain in order to train a deep learning network for improving radar data or related systems. Therefore, it is necessary to find practical ways to fully utilize machine learning to improve radar data or related systems. Summary of the invention

[0003] According to one embodiment, a method includes:

[0004] Receive visual sensor data;

[0005] processing the visual sensor data to produce an estimated object location sensor point cloud distribution;

[0006] receiving a general sensor point cloud distribution corresponding to the vision sensor data;

[0007] comparing the general sensor point cloud distribution to the estimated object location sensor point cloud distribution; and

[0008] A vehicle, an environmental object, or a ghost object is identified in the generic sensor point cloud distribution based on the comparison.

[0009] According to one embodiment, the visual sensor data includes one or more camera images.

[0010] According to one embodiment, the general sensor point cloud distribution is a radar point cloud distribution including one or more of the following: distance, azimuth, elevation and Doppler information.

[0011] According to one embodiment, the generic sensor point cloud distribution includes radar points with range and azimuth information, and the estimated object position sensor point cloud distribution includes estimated radar points with range, azimuth, elevation and Doppler information.

[0012] According to an embodiment, the method further comprises modifying the estimated object position sensor point cloud distribution or the generic sensor point cloud distribution based on the comparison.

[0013] According to one embodiment, the modification comprises increasing or decreasing a confidence level associated with the region where the generic sensor point cloud is distributed.

[0014] According to one embodiment, the method further comprises identifying a vehicle in the generic sensor point cloud distribution when the estimated object location sensor point cloud distribution includes a high likelihood region that coincides with a high likelihood region of the generic sensor point cloud distribution.

[0015] According to one embodiment, the method further comprises identifying an environmental object in the generic sensor point cloud distribution when the estimated object location sensor point cloud distribution includes a low likelihood region that coincides with a high likelihood region of the generic sensor point cloud distribution.

[0016] According to one embodiment, the method further comprises identifying a ghost object in the generic sensor point cloud distribution when the estimated object location sensor point cloud distribution includes a low likelihood region that coincides with a low likelihood region of the generic sensor point cloud distribution.

[0017] According to one specific embodiment, the method further comprises controlling the vehicle based on the comparison.

[0018] According to one embodiment, the method comprises providing a notification to an occupant of the vehicle based on the comparison.

[0019] According to one embodiment, the method further comprises:

[0020] processing the visual sensor data to estimate a vehicle position sensor point cloud distribution, wherein the processing is performed using a deep learning network that can be trained using only visual sensor training images and other training data corresponding to the visual sensor training images as input training data; and

[0021] The vehicle position sensor point cloud distribution is generated based on the estimated vehicle position sensor point cloud distribution.

[0022] According to one embodiment, receiving the visual sensor data and receiving the generic sensor point cloud distribution corresponding to the visual sensor data includes generating the visual sensor data and the generic sensor point cloud distribution using a sensor.

[0023] According to one embodiment, a non-transitory computer-readable medium embodies a set of executable instructions for operating at least one processor to:

[0024] Receive visual sensor data;

[0025] processing the visual sensor data to produce an estimated object location sensor point cloud distribution;

[0026] receiving a general sensor point cloud distribution corresponding to the vision sensor data;

[0027] comparing the general sensor point cloud distribution to the estimated object location sensor point cloud distribution; and

[0028] A vehicle, an environmental object, or a ghost object is identified in the generic sensor point cloud distribution based on the comparison.

[0029] According to one embodiment, the set of executable instructions further operates the at least one processor to:

[0030] The estimated object location sensor point cloud distribution or the generic sensor point cloud distribution is modified based on the comparison.

[0031] According to one embodiment, the modification comprises increasing or decreasing a confidence level associated with the region where the generic sensor point cloud is distributed.

[0032] According to one embodiment, the set of executable instructions additionally operates the at least one processor to:

[0033] When the estimated object position sensor point cloud distribution includes a high likelihood region that coincides with a high likelihood region of the general sensor point cloud distribution, a vehicle in the general sensor point cloud distribution is identified.

[0034] According to one embodiment, the set of executable instructions additionally operates the at least one processor to:

[0035] When the estimated object location sensor point cloud distribution includes a low likelihood region that coincides with a high likelihood region of the general sensor point cloud distribution, an environmental object in the general sensor point cloud distribution is identified.

[0036] According to one embodiment, a method includes:

[0037] Receive visual sensor data;

[0038] processing the visual sensor data to estimate a vehicle position sensor point cloud distribution, wherein the processing is performed using a deep learning network that can be trained using only the visual sensor training data and other training data corresponding to the visual sensor training data as input training data; and

[0039] An estimated vehicle position sensor point cloud is generated based on the estimated vehicle position sensor point cloud distribution.

[0040] According to one embodiment, the visual sensor data is the visual sensor training data, and the method further comprises:

[0041] comparing the estimated vehicle position sensor point cloud distribution with the other training data corresponding to the vision sensor training data to obtain a loss function; and

[0042] The deep learning network is updated based on the loss function. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] By referring to the accompanying drawings, the present disclosure may be better understood and its numerous features and advantages made apparent to those skilled in the art. The use of the same reference symbols in different drawings indicates similar or identical items.

[0044] Figure 1 is a block diagram of a method for training a deep learning network to generate an estimated sensor point cloud distribution based on visual sensor data, according to some embodiments.

[0045] Figure 2 is a representation of a photographic image distributed along a sensor point cloud with corresponding estimates of superimposed training data.

[0046] Figure 3 is a block diagram of a method for generating an estimated sensor point cloud distribution based on visual sensor data, according to some embodiments.

[0047] Figure 4 is a block diagram of a method for generating an estimated sensor point cloud distribution based on vision sensor data and other sensor data according to some embodiments.

[0048] Figure 5 is a block diagram of a method for generating and using an estimated sensor point cloud distribution based on visual sensor data, according to some embodiments.

[0049] Figure 6 is a representation of a photographic image distributed along a corresponding estimated sensor point cloud with superimposed training data, including vehicle position training data.

[0050] Figure 7 is a block diagram of a method for generating an estimated vehicle position sensor point cloud distribution based on visual sensor data, according to some embodiments.

[0051] Figure 8 is a block diagram of a method for generating and using an estimated vehicle position sensor point cloud distribution based on visual sensor data, according to some embodiments.

[0052] Fig. 9 is a block diagram of a method for generating and using an estimated vehicle position sensor point cloud distribution based on visual sensor data, according to some embodiments. DETAILED DESCRIPTION

[0053] Figures 1 to 9 A technique for using machine learning to generate vehicle position sensor data from visual sensor data is shown. As described above, machine learning techniques typically require a large amount of well-labeled training data that may be difficult to obtain in order to train a deep learning network for improving sensor data or related systems. Although a large amount of well-labeled training data is typically not easily available for radar applications, for example, a large amount of well-labeled training data can be used for visual sensor data such as roads. By using a limited amount of vehicle position sensor data, such as radar and / or light detection and ranging (LiDAR) data and such visual sensor data, a deep learning network can be trained to generate an estimated sensor point cloud distribution (i.e., from the visual sensor data alone). Typically, a sensor point cloud distribution refers to a collection of data points typically generated using a sensor, such as a radar or laser radar sensor, where each point represents the position of an object detected in three-dimensional space, typically together with additional attributes such as speed or cross-section. Using a deep learning network trained in this manner, a vehicle with limited or no radar or other sensor functionality can be equipped with a visual sensor to generate an estimated vehicle position sensor point cloud distribution (i.e., an estimated distribution of points of interest that should be detectable using sensors such as radar or LiDAR sensors). These estimated vehicle position sensor point cloud distributions may then be used to improve vehicle safety and / or generate enhanced sensor data through vehicle control or driver notification.

[0054] Figure 1 A method 100 of training a deep learning network to generate an estimated sensor point cloud distribution based on visual sensor data is shown in accordance with some embodiments. In order to provide a clear explanation of the various features of the method 100, in conjunction with Figure 2 discuss Figure 1 , Figure 2 A representation of one or more photographic images 200, which may include one or more camera images, along with a corresponding estimated sensor point cloud distribution 250 with superimposed training data 254 is provided. To generate an estimated sensor point cloud distribution based on visual sensor data using a deep learning network, in some embodiments, labeled training images 102 (e.g., see Figure 2 The deep learning network is trained with photographic images 200) and other training data 103, wherein the other training data 103 may include corresponding to the training image 102 (e.g., see Figure 2Sensor data such as a ground truth point cloud (e.g., a low resolution known distribution of points of interest detected using a sensor such as a radar or LiDAR sensor) of the training data 254 (e.g., co-centered with the training image 102).

[0055] exist Figure 1 In the example of FIG. 1 , a training image 102 is provided as an input to a deep learning network (in this case, a convolutional neural network (CNN) model 104), but it should be noted that other types of deep learning networks may be used instead of or in addition to a CNN model, such as a multilayer perceptron model or a recurrent neural network model. The CNN model 104 processes the training image 102 (see, for example, FIG. 1 ). Figure 2 The method further comprises: using the one or more probability density function (PDF) parameters 106 (e.g., Gaussian mixture model parameters or other statistical modeling parameters, such as mean, variance, and weight parameters) to estimate the PDF 108 (e.g., in a Gaussian mixture model or other statistical model), thereby generating an estimated PDF 109 corresponding to the training image 102 (e.g., see Figure 2 In some embodiments, the other training data 103 includes a point cloud at a first resolution, and the estimated sensor point cloud distribution 250 has a second resolution higher than the first resolution.

[0056] In general, a PDF is a function whose value at any given sample (or point) in sample space (the set of possible values ​​that a random variable can take) can be interpreted as providing a relative likelihood that the value of the random variable will be equal to that sample. Thus, the values ​​of the PDF at two different samples can be used to infer how likely the random variable will be to be close to one sample than to another in any particular draw of the random variable. The estimated PDF 109 should (after sufficient training of the deep learning network) include higher likelihood values ​​in regions where sensor points are likely to be found, and lower likelihood values ​​otherwise. Notably, in this example, the estimated PDF 109 provides estimated regions of high and low likelihood for sensor points derived from the training image 102, but excludes regions of high and low likelihood for sensor points derived from the training image 102. Figure 2 The training data 254 is shown in FIG. For clarity, Figure 3 Another example of an estimated sensor point cloud distribution without superimposed training data is shown in estimated sensor point cloud distribution 350 .

[0057] like Figure 2As can be seen in FIG, by comparing high likelihood regions in the estimated sensor point cloud distribution 250 (corresponding to the estimated PDF 109) with the locations of the training data 254, after training is complete, the PDF 109 can be used to approximately predict the locations of the training data 254 using only the visual sensor data. However, before this level of accuracy can be achieved, in some embodiments, the estimated PDF 109 is first compared to other training data 103 and the CNN model 104 is modified to make the estimated PDF 109 (e.g., Figure 2 Other training data 103 (e.g., Figure 2 The deep learning network is trained by maximizing the likelihood of the training data 254).

[0058] exist Figure 1 In the example of , the estimated PDF 109 is compared 110 with other training data 103 to calculate a conditional probability (a type of probability generated by updating a prior probability with information summarized by the likelihood via application of Bayes' rule), after which a loss function 114 is calculated using a loss function 112, which measures the difference between the PDF 109 and the other training data 103 and maximizes the posterior probability of the other training data relative to the estimated sensor point cloud distribution (e.g., PDF 109). After calculating the loss function 114, the loss function 114 is compared with a threshold 116, which can be predetermined or dynamically controlled. In other embodiments, instead of comparing the loss function 114 with the threshold 116, the threshold 116 controls the number of iterations, thereby allowing training to be limited to a predetermined number of iterations. In some embodiments, a sensor point cloud is generated from the estimated sensor point cloud distribution as a realization with a likelihood value exceeding a threshold. Such embodiments may include: comparing the estimated sensor point cloud distribution with other training data corresponding to vision sensor training data to obtain a loss function; generating a sensor point cloud as an implementation from the estimated sensor point cloud distribution; and comparing a likelihood value in the sensor point cloud with a predetermined threshold to determine whether training of the deep learning network is complete.

[0059] The CNN model 104 is updated based on the loss function, and when the threshold 116 is met, the method 100 of training the CNN model 104 is completed. However, if the threshold 116 is not met or further training is otherwise required, the method 100 provides the same or different training images 102 to the CNN model 104 and repeats the training process, where the goal of each iteration is to make the estimated PDF 109 (e.g., Figure 2 The estimated sensor point cloud distribution 250) and other training data 103 (e.g., Figure 2Any differences between high likelihood regions in the training data 254, or different training data corresponding to different training images, are further minimized.

[0060] It is worth noting that the training image 102 may be a pre-recorded image or a live image recorded by an onboard vehicle system that may include one or more sensors (e.g., photography, radar, etc.). Figure 2 The other training data 103 of the training data 254 shown in FIG. 1 may be pre-recorded sensor data, or live sensor data recorded by an onboard vehicle system. In some embodiments, the other training data 103 and / or the estimated PDF 109 include 2 to 4 dimensions, such as range, azimuth, elevation, and Doppler (i.e., relative velocity) information. In some embodiments, for example, Figure 2 The estimated sensor point cloud distribution 250 is a radar point cloud distribution including estimated radar points with range, azimuth, elevation, and Doppler information, and the other training data 103 includes radar points with range and azimuth information.

[0061] In some embodiments, the estimated PDF 109 is generated using a parametric Gaussian mixture model (GMM), as shown in Equation 1, where K represents the number of components in the GMM, represents each Gaussian, and is the PDF of the resulting point cloud. are the parameters of the vector and matrix respectively and Σ i can include 2 to 4 dimensions, and φ i Each Gaussian The weighting parameter of .

[0062]

[0063] Due to the definition of the Gaussian mixture model, in some embodiments, Equation 2 must be satisfied, forcing each Gaussian The weighting parameter φ i The total is 1.

[0064]

[0065] Gauss Defined by Equation 3.

[0066]

[0067] In some embodiments, the loss function 114 is calculated using Equation 4, where L is the loss and N represents the point cloud. , θ represents the GMM parameters as determined by the deep learning network (e.g., CNN model 104), and α is a tuning parameter that can be used to tune the characteristic p of the PDF (e.g., to provide a sharpening effect). By using a loss function similar to the loss function presented in Equation 4, the deep learning network can be trained to provide maximum likelihood in an estimated PDF or an estimated region of the sensor point cloud distribution, regardless of where the training sensor points are located.

[0068]

[0069] Figure 3 A method 300 for generating an estimated sensor point cloud distribution based on visual sensor data according to some embodiments is shown. Figure 1 training examples, but omitting the training steps, such as Figure 3 As can be seen in FIG. 1 , a live image 302 is provided instead of a training image 102 as a training deep learning network, in this case using Figure 1 The CNN model 104 is trained by the training method 100. The CNN model 104 processes the real image 302 (similar to Figure 2 The photographic image 200, but the live image 302 may include a static image and / or a video or a collection of images) is used to generate one or more PDF parameters 306, which are then used (e.g., in a Gaussian mixture model or other statistical model) to estimate the PDF 108, which can be used to generate an estimated sensor point cloud distribution 350 corresponding to the live image 302 by inference (e.g., at high resolution or super resolution) without providing any sensor data corresponding to the live image 302.

[0070] Although Figure 1 Method 100 may be performed by a training system separate from any vehicle or in an onboard vehicle system, but method 300 and other methods described below for generating an estimated sensor point cloud distribution may be performed partially or completely by an onboard vehicle system. However, in some embodiments, some portions of the method may be performed by an external (e.g., cloud or server-based) computing system. For example, in some embodiments, a live image 302 is obtained using an onboard vehicle system, and the CNN model 104 analyzes the live image 302, generates PDF model parameters 306, estimates PDF 108, and / or generates an estimated sensor point cloud distribution 350 by communicating with the onboard vehicle system (e.g., WiFi or cellular) via an external computing system.

[0071] Figure 4 A method 400 for generating an estimated sensor point cloud distribution based on visual sensor data and other sensor data is shown in accordance with some embodiments. Figure 4 , method 400 is the same as method 300 except for live sensor data 402. In the example of method 400, when live sensor data 402 corresponding to live image 302 is available, estimating PDF 108 may include taking into account live sensor data 402. Thus, rather than generating a PDF that produces estimated sensor point cloud distribution 450 based solely on analysis of live image 302 as in method 300, in method 400, an estimated PDF and / or estimated sensor point cloud distribution 450 is created or modified based on live sensor data 402 in addition to live image 302.

[0072] For example, in some embodiments, a correlation between the live sensor data 402 and the estimated PDF or the estimated sensor point cloud distribution 450 is determined, and the estimated PDF or the estimated sensor point cloud distribution 450 is created or modified based on the correlation. For example, when a sensor point in the live sensor data 402 and a high likelihood region of the estimated sensor point cloud distribution 450 coincide, the estimated PDF or the estimated sensor point cloud distribution 450 can be modified based on the identified correlation, for example, by further increasing the likelihood of the high likelihood region corresponding to the sensor point in the live sensor data 402 and / or increasing the confidence level associated with the high likelihood region corresponding to the sensor point in the live sensor data 402. Similarly, when a sensor point in the live sensor data 402 and a low likelihood region of the estimated sensor point cloud distribution 450 coincide, the estimated PDF or the estimated sensor point cloud distribution 450 may be modified based on the identified lack of correlation, for example, by increasing the likelihood of the low likelihood region corresponding to the sensor point in the live sensor data 402 and / or decreasing the confidence level associated with the low likelihood region corresponding to the sensor point in the live sensor data 402. Additionally, when no sensor point in the live sensor data 402 and a high likelihood region of the estimated sensor point cloud distribution 450 coincide, the estimated PDF or the estimated sensor point cloud distribution 450 may be modified based on the identified lack of correlation, for example, by decreasing the likelihood that the high likelihood region lacks any coincident sensor point in the live sensor data 402 and / or decreasing the confidence level associated with the high likelihood region lacks any coincident sensor point in the live sensor data 402. It is worth noting that although "live" images and "live" sensor data are referred to herein for clarity and convenience, previously recorded images and, optionally, previously recorded sensor data corresponding to the recorded images may be used in place of "live" images and data in the various methods disclosed herein to generate an estimated PDF and / or an estimated sensor point cloud distribution of the recorded images and data.

[0073] Figure 5A method 500 for generating and using an estimated sensor point cloud distribution based on visual sensor data is shown in accordance with some embodiments. Figure 5 As shown, method 500 includes receiving visual sensor data at block 502, such as Figure 2 At block 504, method 500 includes processing visual sensor data to estimate a sensor point cloud distribution, such as a point cloud PDF or other parameters, using a deep learning network (e.g., using Figure 1 The processing is performed by a deep learning network trained by method 100, wherein the deep learning network may be trained using only the visual sensor training data and other training data corresponding to the visual sensor training data as input training data. At block 506, method 500 includes: Figure 4 The estimated sensor point cloud distribution 450 is similar to the estimated sensor point cloud distribution to generate an estimated sensor point cloud distribution. At block 508 , the method 500 includes utilizing the estimated sensor point cloud distribution.

[0074] The estimated sensor point cloud distribution can be used as the basis for any of a number of applications, ranging from notifying the occupants of the vehicle of an approaching object or a turn identified in the estimated sensor point cloud distribution, to directly controlling the vehicle, such as applying the brakes or acceleration to avoid a collision with an approaching object or automatically turning a turn when an impending turn is identified, such as via the steering wheel. Figure 4 As discussed, in some embodiments, the estimated sensor point cloud distribution is used to improve the live sensor data. In other embodiments, the estimated PDF or the estimated sensor point cloud distribution is modified based on the live sensor data to produce an improved estimated sensor point cloud distribution. In some embodiments, the estimated PDF or the estimated sensor point cloud distribution is used to increase the granularity and / or dimension of the live sensor data, for example, by adding sensor points and / or converting 2D (or 3D) live sensor data into 3D (or 4D) live sensor data. In some embodiments, the estimated PDF or the estimated sensor point cloud distribution is used together with the corresponding visual sensor data as training data for other machine learning applications. Therefore, the method disclosed herein can be used not only to generate the sensor data from the visual sensor data for the purpose of utilizing the sensor data in live or "real-time" applications, but also to generate a large amount of sensor data for the visual sensor data, so as to enable other deep learning applications that may require such a number of corresponding sensor data and visual sensor data.

[0075] In some embodiments, in order to increase the usefulness of the estimated sensor point cloud distribution, it is necessary to generate an estimated vehicle position sensor point cloud distribution in which only vehicles are identified as high likelihood areas. To generate the vehicle position sensor point cloud distribution, the same method described above (e.g., methods 100, 300, 400, 500) can be used, but with different additional training data 103 specific to vehicle positions and different loss functions. For example, Figure 6 A photographic image is shown along with a corresponding estimated sensor point cloud distribution with superimposed training data, the superimposed training data including vehicle position training data. By limiting other training data 103 to vehicle position training data 654 and excluding other training data 656, a deep learning network can be trained to generate an estimated vehicle position sensor point cloud distribution based only on visual sensor data.

[0076] In some embodiments, to generate the vehicle position sensor point cloud distribution, a loss function different from the loss function specified in Equation 4 above is used, such as the loss function specified below in Equation 5. By using a loss function similar to the loss function presented in Equation 5, a deep learning network can be trained to provide maximum likelihood in regions of the estimated PDF or estimated sensor point cloud distribution regardless of where the training vehicle position sensor points are located, while forcing other regions to reach a minimum.

[0077]

[0078] Figure 7 A method 700 for generating an estimated vehicle position sensor point cloud distribution based on visual sensor data is shown in accordance with some embodiments. Figure 1 , but omits the training steps, such as Figure 7 As can be seen in FIG. 1 , instead of using the training image 102, a live image 302 is provided as a training deep learning network, in this case using Figure 1 100 training methods for Figure 6 The CNN model 104 is trained with the vehicle position training data 654 similar to the vehicle position training data. The CNN model 104 processes the real image 302 (similar to Figure 6 The vehicle position PDF 108 can be used to generate an estimated vehicle position sensor point cloud distribution 750 corresponding to the live image 302 without providing any sensor data corresponding to the live image 302. Figure 7As shown, after sufficient training, method 700 can generate an estimated vehicle position sensor point cloud distribution 750 , in which only the vehicle or vehicle neighboring areas are identified as high likelihood areas.

[0079] In some embodiments, by using the Figure 4 As discussed above, the vehicle position PDF 108 and / or the estimated vehicle position sensor point cloud distribution 750 may be modified based on the live sensor data 402 in a manner similar to the manner in which the PDF 108 and / or the estimated sensor point cloud distribution 450 may be updated based on the live sensor data 402. Additionally, in some embodiments, a generic (e.g., non-vehicle specific) estimated sensor point cloud distribution, such as Figure 3 and 4 The estimated sensor point cloud distribution 350 or 450 (or the corresponding general PDF) and the estimated vehicle position sensor point cloud distribution, such as Figure 7 The vehicle position sensor point cloud distribution 750 (or corresponding vehicle PDF) is compared to identify vehicles, environmental objects, and ghost objects (e.g., false positives) with higher confidence.

[0080] For example, when the estimated vehicle position sensor point cloud distribution includes a high likelihood region that overlaps with a high likelihood region of the general estimated sensor point cloud distribution, it can be inferred that this overlap region corresponds to a vehicle with a reasonably high confidence. On the other hand, when the estimated vehicle position sensor point cloud distribution includes a low likelihood region that overlaps with a high likelihood region of the general estimated sensor point cloud distribution, it can be inferred that this overlap region corresponds to an environmental object with a reasonably high confidence. When the estimated vehicle position point cloud distribution includes a low likelihood region that overlaps with a low likelihood region of the general estimated sensor point cloud distribution, it can be inferred that this overlap region corresponds to a ghost object with a reasonably high confidence. Therefore, by comparing the estimated vehicle position sensor point cloud distribution with the general estimated sensor point cloud distribution, ghost objects can be eliminated or assigned low confidence values, while vehicles and / or environmental objects can be identified and / or high confidence values ​​can be assigned to vehicles and / or environmental objects.

[0081] For example, in some embodiments, after obtaining the estimated vehicle position sensor point cloud distribution and the general estimated sensor point cloud distribution, the posterior probability of each point in the co-centered sensor point cloud distribution is evaluated on the two PDFs. If p1(x|θ) and p22(x|θ) are the posterior probabilities calculated for the sensor point x based on these probabilities using the general estimated sensor point cloud distribution PDF and the estimated vehicle position sensor point cloud distribution PDF, respectively, then in some embodiments, the point x is classified as shown in the following relations 6-8 (i.e., example fusion rules), where τ1 and τ2 are configurable thresholds. Notably, in some embodiments, a series of polar coordinates z (where p2(z|θ)>τ2) that have no points in the nearby estimated sensor point cloud distribution is considered a missed detection (or the confidence of those coordinates corresponding to the missed detection is increased).

[0082] If p1(x|θ)>τ1 and p2(x|θ)>τ2, then x is likely to be a vehicle (6)

[0083] If p1(x|θ)>τ1 and p2(x|θ)<τ2, then x may be an environment object object (7)

[0084] If p1(x|θ)<τ1 and p2(x|θ)<τ2, then x may be a ghost object (8)

[0085] In some embodiments, the estimated vehicle position sensor point cloud distribution or the general sensor point cloud distribution is modified based on the comparison, for example to eliminate ghost objects and / or increase confidence values ​​of vehicles and / or environmental objects. Notably, when both the general estimated sensor point cloud distribution and the estimated vehicle position sensor point cloud distribution are produced using the same system or a related system, one or more parameters of the CNN model 104 can be shared between the CNN model 104 used to produce the general estimated sensor point cloud distribution and the CNN model 104 used to produce the estimated vehicle position sensor point cloud distribution.

[0086] Figure 8 A method 800 for generating and using an estimated vehicle position sensor point cloud distribution based on visual sensor data is shown in accordance with some embodiments. Figure 8 As shown, method 800 includes receiving visual sensor data at block 802, such as Figure 6 At block 804, method 800 includes processing the visual sensor data to estimate the vehicle position sensor point cloud distribution, such as the point cloud PDF or other parameters, using a deep learning network (e.g., using Figure 1The processing is performed by using a deep learning network trained by method 100, wherein the deep learning network may be trained using only visual sensor training images and vehicle position training data corresponding to the visual sensor training images as input training data. At block 806, method 800 includes: Figure 7 The estimated vehicle position sensor point cloud distribution 750 is similar to the estimated vehicle position sensor point cloud distribution to produce an estimated vehicle position sensor point cloud distribution. At block 808, the method 800 includes utilizing the estimated sensor point cloud distribution.

[0087] Similar to the general estimated sensor point cloud distribution, the estimated vehicle position sensor point cloud distribution can be used as the basis for any of a number of applications (possibly in combination with the general estimated sensor point cloud distribution, as discussed above), ranging from notifying the occupants of the vehicle of the approaching vehicle, to directly controlling the vehicle, such as applying brakes to avoid collision with the approaching vehicle. As mentioned above, in some embodiments, the estimated vehicle position sensor point cloud distribution is used to improve the live sensor data, or to modify the estimated vehicle position PDF or the estimated vehicle position sensor point cloud distribution based on the live sensor data to produce an improved estimated vehicle position sensor point cloud distribution. In some embodiments, the estimated vehicle position PDF or the estimated vehicle position sensor point cloud distribution is used to increase the granularity and / or dimensionality of the live sensor data, such as by adding sensor points and / or converting 2D (or 3D) live sensor data into 3D (or 4D) live sensor data. In some embodiments, the estimated vehicle position PDF or the estimated vehicle position sensor point cloud distribution is used together with the corresponding visual sensor data as training data for other machine learning applications. Thus, the methods disclosed herein may be used not only to generate estimated vehicle position sensor data from visual sensor data for the purpose of utilizing the estimated vehicle position sensor data in live or "real-time" applications, but may also be used to generate large amounts of estimated vehicle position sensor data for visual sensor data in order to enable additional deep learning applications that may require such amounts of corresponding sensor data and visual sensor data.

[0088] Fig. 9 A method 900 for generating and using an estimated vehicle position sensor point cloud distribution based on visual sensor data is shown in accordance with some embodiments. Fig. 9 As shown, method 900 includes receiving visual sensor data at block 902, such as Figure 6of a photographic image 200. At box 904, method 900 includes processing the visual sensor data to produce an estimated vehicle or object position sensor point cloud distribution, which may be produced using the techniques described above, such as those of method 800. At box 906, method 900 includes receiving a generic sensor point cloud distribution corresponding to the visual sensor data, which may be generated based on sensor data located in or on a vehicle or received from another source (e.g., an automated road or highway sensor broadcast). At box 908, method 900 includes comparing the generic sensor point cloud distribution with the estimated object position sensor point cloud distribution. At box 910, method 900 includes identifying a vehicle, environmental object, or ghost object in the generic sensor point cloud distribution based on the comparison, which may include comparing the confidence and / or likelihood of the estimated object or vehicle position sensor point cloud distribution and the generic sensor point cloud distribution as discussed above. Thus, method 900 provides Figure 3 Method 300 and Figure 7 The fusion of method 700 enables a CNN such as CNN104 to train a parameter estimation model such as GMM to obtain a probability distribution of a first point cloud distribution (e.g., estimated sensor point cloud distribution 450) and a second detection / vehicle point cloud distribution (e.g., estimated vehicle position sensor point cloud distribution 750). Then, based on the trained machine learning model such as CNN 104, a posterior point cloud distribution is obtained, including, for example, vehicles, environmental objects, and / or ghost objects, so that points in the posterior point cloud distribution can be classified or weighted (e.g., confidence weighting) based on whether the points therein are more likely to be vehicles, environmental objects, and / or ghost objects.

[0089] In some embodiments, certain aspects of the technology described above can be implemented by one or more processors of a processing system that executes software. The software includes one or more sets of executable instructions stored or otherwise tangibly embodied on a non-transitory computer-readable storage medium. The software may include instructions and certain data, which, when executed by one or more processors, manipulate one or more processors to perform one or more aspects of the technology described above. The non-transitory computer-readable storage medium may include, for example, a disk or optical disk storage device, a solid-state storage device such as a flash memory, a cache memory, a random access memory (RAM), or other one or more non-volatile memory devices, etc. The executable instructions stored on the non-transitory computer-readable storage medium may be in the form of source code, assembly language code, object code, or in other instruction formats that can be interpreted or otherwise executed by one or more processors.

[0090] Computer-readable storage media may include any storage media or combination of storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, tapes, or magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache memory), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or micro-electromechanical system (MEMS)-based storage media. Computer-readable storage media may be embedded in a computing system (e.g., system RAM or ROM), fixedly attached to a computing system (e.g., a magnetic hard drive), removably attached to a computing system (e.g., an optical disc or flash memory based on a universal serial bus (USB)), or coupled to a computer system via a wired or wireless network (e.g., a network accessible storage device (NAS)).

[0091] It should be noted that not all of the activities or elements described above in the general description are required, a part of a particular activity or device may not be required, and one or more additional activities or elements may be performed or included in addition to those described. Furthermore, the order of activities listed is not necessarily the order in which they are performed. In addition, concepts have been described with reference to specific embodiments. However, it will be appreciated by those of ordinary skill in the art that various modifications and changes may be made without departing from the scope of the present disclosure as set forth in the following claims. Therefore, the description and drawings should be viewed in an illustrative rather than a restrictive sense, and it is expected that all such modifications are included within the scope of the present disclosure.

[0092] The benefits, other advantages, and solutions to problems are described above with respect to specific embodiments. However, these benefits, advantages, solutions to problems, and any features that may make any benefit, advantage, or solution appear or become more prominent should not be understood as key, required, or essential features of any or all claims. In addition, the specific embodiments disclosed above are merely illustrative, as the disclosed subject matter may be modified and practiced in different but equivalent ways that are obvious to those skilled in the art who benefit from the teachings herein. Except as described in the appended claims, it is not intended to limit the details of the construction or design shown herein. Therefore, it is apparent that the specific embodiments disclosed above may be changed or modified, and all such changes are deemed to be within the scope of the disclosed subject matter. Therefore, the protection sought herein is as set forth in the appended claims.

Claims

1. A method, characterized in that The method comprises: Receive visual sensor data; processing the visual sensor data to produce an estimated object location sensor point cloud distribution; receiving a general sensor point cloud distribution corresponding to the vision sensor data; comparing the general sensor point cloud distribution to the estimated object location sensor point cloud distribution; and A vehicle, an environmental object, or a ghost object is identified in the generic sensor point cloud distribution based on the comparison.

2. The method according to claim 1, characterized in that Additionally including identifying a vehicle in the general sensor point cloud distribution when the estimated object location sensor point cloud distribution includes a high likelihood region that coincides with a high likelihood region of the general sensor point cloud distribution.

3. The method according to claim 1, characterized in that Additionally including identifying an environmental object in the generic sensor point cloud distribution when the estimated object location sensor point cloud distribution includes a low likelihood region that coincides with a high likelihood region of the generic sensor point cloud distribution.

4. The method according to claim 1, characterized in that: Additionally including identifying a ghost object in the generic sensor point cloud distribution when the estimated object location sensor point cloud distribution includes a low likelihood region that coincides with a low likelihood region of the generic sensor point cloud distribution.

5. The method according to claim 1, characterized in that Also includes: processing the visual sensor data to estimate a vehicle position sensor point cloud distribution, wherein the processing is performed using a deep learning network that can be trained using only visual sensor training images and other training data corresponding to the visual sensor training images as input training data; as well as The vehicle position sensor point cloud distribution is generated based on the estimated vehicle position sensor point cloud distribution.

6. A non-transitory computer-readable medium, characterized in that The non-transitory computer readable medium embodies a set of executable instructions for manipulating at least one processor to: Receive visual sensor data; processing the visual sensor data to produce an estimated object location sensor point cloud distribution; receiving a general sensor point cloud distribution corresponding to the vision sensor data; comparing the general sensor point cloud distribution to the estimated object location sensor point cloud distribution; as well as A vehicle, an environmental object, or a ghost object is identified in the generic sensor point cloud distribution based on the comparison.

7. The non-transitory computer-readable medium of claim 6, wherein: The set of executable instructions may additionally operate the at least one processor to: When the estimated object position sensor point cloud distribution includes a high likelihood region that coincides with a high likelihood region of the general sensor point cloud distribution, a vehicle in the general sensor point cloud distribution is identified.

8. The non-transitory computer-readable medium of claim 6, wherein: The set of executable instructions may additionally operate the at least one processor to: When the estimated object location sensor point cloud distribution includes a low likelihood region that coincides with a high likelihood region of the generic sensor point cloud distribution, an environmental object in the generic sensor point cloud distribution is identified.

9. A method, characterized in that The method comprises: Receive visual sensor data; processing the visual sensor data to estimate a vehicle position sensor point cloud distribution, wherein the processing is performed using a deep learning network that can be trained using only the visual sensor training data and other training data corresponding to the visual sensor training data as input training data; and An estimated vehicle position sensor point cloud is generated based on the estimated vehicle position sensor point cloud distribution.

10. The method according to claim 9, characterized in that The visual sensor data is the visual sensor training data, and the method further comprises: comparing the estimated vehicle position sensor point cloud distribution with the other training data corresponding to the vision sensor training data to obtain a loss function; and The deep learning network is updated based on the loss function.

Images