Sensor point cloud probability density function estimation based on visual sensor data
Through deep learning network processing of visual sensor data and estimating sensor point cloud distribution, the dependence problem on a large number of radar training data in the prior art is solved, and high-quality sensor point cloud distribution generation and vehicle safety improvement under visual sensor data conditions are achieved.
Patent Information
- Application Number
- CN202411129183.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-08-25
- Filing Date
- 2024-08-16
- Publication Date
- 2025-02-25
AI Technical Summary
The prior art requires a large amount of training data when using machine learning to improve radar data, but this data may be difficult to obtain, making it difficult to fully utilize machine learning to improve radar data or related systems.
By processing vision sensor data using a deep learning network, the sensor point cloud distribution is estimated and based on this, an estimated sensor point cloud is generated. This method only uses vision sensor training data and other related training data for training, avoiding dependence on a large amount of radar training data.
In the absence of large amounts of radar training data, deep learning networks are trained through visual sensor data to generate high-quality sensor point cloud distributions, thereby improving vehicle safety and generating enhanced sensor data.
Smart Images
Figure CN119516219A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method and device for estimating a probability density function of a sensor point cloud based on visual sensor data. Background Art
[0002] Radar data is often used to improve vehicle safety. For example, radar-based object avoidance and collision avoidance are often utilized in vehicles to, for example, notify the driver of nearby objects when the driver is parking the vehicle, or even automatically apply vehicle braking when an impending collision is detected. However, radar sensors are often expensive and radar data may be inaccurate, and therefore it is necessary to use machine learning techniques to, for example, minimize the number of sensors required for a given application and / or improve the radar data derived from those sensors. One difficulty in using machine learning to improve radar data is that machine learning techniques typically require a large amount of training data in order to train a deep learning network so that it can be used to improve radar data or related systems, and a large amount of training data may be difficult to obtain. Accordingly, it is necessary to find practical ways to fully utilize machine learning to improve radar data or related systems. Summary of the invention
[0003] According to one embodiment, a method includes:
[0004] Receive visual sensor data;
[0005] processing the visual sensor data to estimate a sensor point cloud distribution, wherein the processing is performed using a deep learning network capable of being trained using only the visual sensor training data and other training data corresponding to the visual sensor training data as input training data; and
[0006] An estimated sensor point cloud is generated based on the estimated sensor point cloud distribution.
[0007] According to one embodiment, the visual sensor data includes one or more camera images.
[0008] According to one embodiment, the sensor point cloud distribution is a radar point cloud distribution including one or more of the following: distance, azimuth, elevation and Doppler information.
[0009] According to one embodiment, the other training data includes radar points with range and azimuth information, and the estimated sensor point cloud distribution includes estimated radar points with range, azimuth, elevation and Doppler information.
[0010] According to one embodiment, the visual sensor data is the visual sensor training data, and the method further comprises:
[0011] comparing the estimated sensor point cloud distribution with the other training data corresponding to the vision sensor training data to obtain a loss function; and
[0012] The deep learning network is updated based on the loss function.
[0013] According to one embodiment, the loss function maximizes the posterior probability of the further training data to the estimated sensor point cloud distribution.
[0014] According to one embodiment, the further training data comprises a point cloud at a first resolution, and the estimated sensor point cloud distribution has a second resolution higher than the first resolution.
[0015] According to one embodiment, the method further comprises generating a sensor point cloud from the estimated sensor point cloud distribution as a realization having a likelihood value exceeding a threshold value.
[0016] According to one embodiment, the visual sensor data is the visual sensor training data, and the method further comprises:
[0017] comparing the estimated sensor point cloud distribution with the other training data corresponding to the vision sensor training data to obtain a loss function;
[0018] As one implementation, generating a sensor point cloud from the estimated sensor point cloud distribution; and
[0019] The likelihood value in the sensor point cloud is compared with a predetermined threshold to determine whether the training of the deep learning network is complete.
[0020] According to one embodiment, the estimated sensor point cloud distribution comprises Gaussian mixture model parameters.
[0021] According to one embodiment, the method further comprises:
[0022] receiving sensor data corresponding to the visual sensor data; and
[0023] The estimated sensor point cloud distribution is modified based on the sensor data.
[0024] According to one embodiment, the modification further comprises:
[0025] determining a correlation between the sensor data and the estimated sensor point cloud distribution;
[0026] as well as
[0027] The estimated sensor point cloud distribution is modified based on the correlation.
[0028] According to an embodiment, the method further comprises controlling a vehicle based on the estimated sensor point cloud distribution.
[0029] According to an embodiment, the method further comprises providing a notification to an occupant of the vehicle based on the estimated sensor point cloud distribution.
[0030] According to one embodiment, a non-transitory computer-readable medium embodies a set of executable instructions for operating at least one processor to perform the following operations:
[0031] Receive visual sensor data;
[0032] processing the visual sensor data to estimate a sensor point cloud distribution, wherein the processing is performed using a deep learning network capable of being trained using only the visual sensor training data and other training data corresponding to the visual sensor training data as input training data; and
[0033] An estimated sensor point cloud is generated based on the estimated sensor point cloud distribution.
[0034] According to one embodiment, the vision sensor data is the vision sensor training data, and the set of executable instructions additionally operates the at least one processor to:
[0035] comparing the estimated sensor point cloud distribution with the other training data corresponding to the vision sensor training data to obtain a loss function; and
[0036] The deep learning network is updated based on the loss function.
[0037] According to one embodiment, the vision sensor data is the vision sensor training data, and the set of executable instructions additionally operates the at least one processor to:
[0038] comparing the estimated sensor point cloud distribution with the other training data corresponding to the vision sensor training data to obtain a loss function;
[0039] As one implementation, generating a sensor point cloud from the estimated sensor point cloud distribution; and
[0040] The likelihood value in the sensor point cloud is compared with a predetermined threshold to determine whether the training of the deep learning network is complete.
[0041] According to one embodiment, a device comprising a processor is configured to:
[0042] Receive visual sensor data;
[0043] processing the visual sensor data to estimate a sensor point cloud distribution, wherein the processing is performed using a deep learning network capable of being trained using only the visual sensor training data and other training data corresponding to the visual sensor training data as input training data; and
[0044] An estimated sensor point cloud is generated based on the estimated sensor point cloud distribution.
[0045] According to one embodiment, the visual sensor data is the visual sensor training data, and the processor is further configured to:
[0046] comparing the estimated sensor point cloud distribution with the other training data corresponding to the vision sensor training data to obtain a loss function; and
[0047] The deep learning network is updated based on the loss function.
[0048] According to one embodiment, the visual sensor data is the visual sensor training data, and the processor is further configured to:
[0049] comparing the estimated sensor point cloud distribution with the other training data corresponding to the vision sensor training data to obtain a loss function;
[0050] As one implementation, generating a sensor point cloud from the estimated sensor point cloud distribution; and
[0051] The likelihood value in the sensor point cloud is compared with a predetermined threshold to determine whether the training of the deep learning network is complete. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] By referencing the accompanying drawings, the present disclosure may be better understood, and its numerous features and advantages made apparent to those skilled in the art. The use of the same reference numerals in different drawings indicates similar or identical items.
[0053] Figure 1 is a block diagram of a method of training a deep learning network to generate an estimated sensor point cloud distribution based on visual sensor data, according to some embodiments.
[0054] Figure 2 is a representation of a photographic image distributed along a corresponding estimated sensor point cloud with superimposed training data.
[0055] Figure 3 is a block diagram of a method for generating an estimated sensor point cloud distribution based on visual sensor data, according to some embodiments.
[0056] Figure 4 is a block diagram of a method for generating an estimated sensor point cloud distribution based on vision sensor data and other sensor data, according to some embodiments.
[0057] Figure 5 is a block diagram of a method for generating and using an estimated sensor point cloud distribution based on visual sensor data, according to some embodiments. DETAILED DESCRIPTION
[0058] Figure 1-5 A technique for generating sensor data from visual sensor data using machine learning is shown. As described above, machine learning techniques typically require a large amount of well-labeled training data that may be difficult to obtain in order to train a deep learning network for improving sensor data or related systems. Although a large amount of well-labeled training data is typically not easily available for radar applications, for example, a large amount of well-labeled training data can be used for visual sensor data such as roads. By using a limited amount of sensor data, such as radar and / or light detection and ranging (LiDAR) data and such visual sensor data, a deep learning network can be trained to generate an estimated sensor point cloud distribution (i.e., from the visual sensor data alone). Typically, a sensor point cloud distribution refers to a collection of data points typically generated using a sensor such as a radar or LiDAR sensor, where each point represents the position of an object detected in three-dimensional space, often along with additional attributes such as speed or cross-section. Using a deep learning network trained in this manner, a vehicle with limited or no radar or other sensor functionality can be equipped with a visual sensor to generate an estimated sensor point cloud distribution (i.e., an estimated distribution of points of interest that should be detectable using a sensor such as a radar or LiDAR sensor). These estimated sensor point cloud distributions may then be used to improve vehicle safety and / or generate enhanced sensor data through vehicle control or driver notification.
[0059] Figure 1 A method 100 of training a deep learning network to generate an estimated sensor point cloud distribution based on visual sensor data according to some embodiments is shown. In order to provide a clear explanation of the various features of the method 100, in conjunction with Figure 2 Discussion Figure 1 , Figure 2 A representation of one or more photographic images 200, which may include one or more camera images, along with a corresponding estimated sensor point cloud distribution 250 with superimposed training data 254 is provided. To generate an estimated sensor point cloud distribution based on visual sensor data using a deep learning network, in some embodiments, labeled training images 102 (e.g., see Figure 2The deep learning network is trained with photographic images 200) and other training data 103, wherein the other training data 103 may include corresponding to the training image 102 (e.g., see Figure 2 Sensor data such as a ground truth point cloud (e.g., a low-resolution known distribution of points of interest detected using a sensor such as a radar or LiDAR sensor) (e.g., co-centered with the training data 254).
[0060] exist Figure 1 In the example of FIG. 1 , a training image 102 is provided as an input to a deep learning network (in this case, a convolutional neural network (CNN) model 104), but it should be noted that other types of deep learning networks may be used instead of or in addition to a CNN model, such as a multilayer perceptron model or a recurrent neural network model, among others. The CNN model 104 processes the training image 102 (e.g., see Figure 2 The photographic image 200 of the training image 102 is then used to generate one or more probability density function (PDF) parameters 106 (e.g., Gaussian mixture model parameters or other statistical modeling parameters, such as mean, variance, and weight parameters), and then these parameters 106 are used to estimate the PDF 108 (e.g., in a Gaussian mixture model or other statistical model), thereby generating an estimated PDF 109 corresponding to the training image 102 (e.g., see Figure 2 In some embodiments, the other training data 103 includes a point cloud at a first resolution, and the estimated sensor point cloud distribution 250 has a second resolution higher than the first resolution.
[0061] In general, a PDF is a function whose value at any given sample (or point) in sample space (the set of possible values that a random variable can take) can be interpreted as providing a relative likelihood that the value of the random variable will be equal to the sample. Accordingly, the values of the PDF at two different samples can be used to infer how likely it is that the random variable will be closer to one sample than to another sample in any particular drawing of the random variable. The estimated PDF 109 should (after sufficient training of the deep learning network) include higher likelihood values in regions where sensor points are likely to be found, and lower likelihood values otherwise. Notably, in this example, the estimated PDF 109 provides estimated high and low likelihood regions for sensor points derived from the training image 102, but does not include Figure 2 The training data 254 is shown in FIG. Figure 3 Another example of an estimated sensor point cloud distribution without superimposed training data is shown in estimated sensor point cloud distribution 350 of FIG.
[0062] like Figure 2As can be seen in FIG. 1 , after training is complete, PDF 109 can be used to approximately predict the location of training data 254 using only visual sensor data by comparing high likelihood regions in estimated sensor point cloud distribution 250 (corresponding to estimated PDF 109) with the locations of training data 254. However, in some embodiments, before this level of accuracy can be achieved, the estimated PDF 109 is first compared to other training data 103 and the CNN model 104 is modified to make the estimated PDF 109 (e.g., Figure 2 Other training data 103 (e.g., Figure 2 The likelihood of the training data 254) is maximized to train the deep learning network.
[0063] exist Figure 1 In the example of , the estimated PDF 109 is compared 110 with other training data 103 to calculate a conditional probability (a type of probability generated by updating the prior probability with information summarized by the likelihood via application of Bayes' rule), after which a loss function 114 is calculated using a loss function 112, which measures the difference between the PDF 109 and the other training data 103 and maximizes the posterior probability of the other training data to the estimated sensor point cloud distribution (e.g., PDF 109). After calculating the loss function 114, the loss function 114 is compared with a threshold 116, which can be predetermined or dynamically controlled. In other embodiments, instead of comparing the loss function 114 with the threshold 116, the threshold 116 controls the number of iterations, thereby allowing training to be limited to a predetermined number of iterations. In some embodiments, a sensor point cloud is generated from the estimated sensor point cloud distribution as an implementation having a likelihood value that exceeds a threshold. Such embodiments may include: comparing the estimated sensor point cloud distribution with other training data corresponding to vision sensor training data to obtain a loss function; generating a sensor point cloud from the estimated sensor point cloud distribution as an implementation; and comparing a likelihood value in the sensor point cloud with a predetermined threshold to determine whether training of the deep learning network is complete.
[0064] The CNN model 104 is updated based on the loss function, and when the threshold 116 is met, the method 100 of training the CNN model 104 is completed. However, if the threshold 116 is not met or further training is otherwise required, the method 100 provides the same or different training images 102 to the CNN model 104 and repeats the training process, where the goal of each iteration is to make the estimated PDF 109 (e.g., Figure 2 of estimated sensor point cloud distribution 250) and other training data 103 (e.g., Figure 2Any differences between high likelihood regions in the training data 254, or different training data corresponding to different training images, are further minimized.
[0065] It is worth noting that the training images 102 may be pre-recorded images, or live images recorded by an onboard vehicle system that may include one or more sensors (e.g., photography, radar, etc.). Figure 2 The other training data 103 of the training data 254 shown in FIG. 104 may be pre-recorded sensor data, or live sensor data recorded by an onboard vehicle system. In some embodiments, the other training data 103 and / or the estimated PDF 109 include 2-4 dimensions, such as distance, azimuth, elevation, and Doppler (i.e., relative velocity) information. In some embodiments, for example Figure 2 The estimated sensor point cloud distribution 250 is a radar point cloud distribution including estimated radar points with range, azimuth, elevation, and Doppler information, and the other training data 103 includes radar points with range and azimuth information.
[0066] In some embodiments, the estimated PDF 109 is generated using a parametric Gaussian mixture model (GMM), as shown in Equation 1, where K represents the number of components in the GMM, represents each Gaussian, and is the generated point cloud PDF. and Σ i (which are vectors and matrices, respectively) can include 2-4 dimensions, and φ i For each Gauss The weighting parameter of .
[0067]
[0068] Due to the definition of the Gaussian mixture model, in some embodiments, Equation 2 must be satisfied, forcing each Gaussian The weighting parameter φ i The total is 1.
[0069]
[0070] Gauss Defined by Equation 3.
[0071]
[0072] In some embodiments, the loss function 114 is calculated using Equation 4, where L is the loss and N represents the point cloud. , θ represents the GMM parameters determined by the deep learning network (e.g., CNN model 104), and α is a tuning parameter that can be used to tune the properties of PDF p (e.g., to provide a sharpening effect). By using a loss function similar to the loss function presented in Equation 4, the deep learning network can be trained to provide maximum likelihood in a region of the estimated PDF or estimated sensor point cloud distribution, regardless of where the training sensor points are located.
[0073]
[0074] Figure 3 A method 300 of generating an estimated sensor point cloud distribution based on visual sensor data according to some embodiments is shown. Figure 1 but omits the training steps, such as Figure 3 As can be seen in FIG. 1 , instead of the training image 102, a live image 302 is provided as input to a trained deep learning network, in this case using Figure 1 The training method 100 trains the CNN model 104. The CNN model 104 processes the live image 302 (similar to Figure 2 The sensor point cloud distribution 350 corresponding to the live image 302 may be generated by inferring (e.g., at a high resolution or super resolution) one or more PDF parameters 306, which are then used to estimate the PDF 108 (e.g., in a Gaussian mixture model or other statistical model). The PDF 108 can be used to generate an estimated sensor point cloud distribution 350 corresponding to the live image 302 by inference (e.g., at a high resolution or super resolution) without providing any sensor data corresponding to the live image 302.
[0075] Although Figure 1 The method 100 may be performed by a training system separate from any vehicle or in an onboard vehicle system, but the method 300 and other methods described below for generating an estimated sensor point cloud distribution may be performed partially or completely by an onboard vehicle system. However, in some embodiments, some portions of the method may be performed by an external (e.g., cloud or server-based) computing system. For example, in some embodiments, a live image 302 is obtained using an onboard vehicle system, and the analysis of the live image 302 by the CNN model 104, the generation of the PDF model parameters 306, the estimation of the PDF 108, and / or the generation of the estimated sensor point cloud distribution 350 are performed by an external computing system communicating with the onboard vehicle system (e.g., WiFi or cellular).
[0076] Figure 4 A method 400 for generating an estimated sensor point cloud distribution based on visual sensor data and other sensor data is shown in accordance with some embodiments. Figure 4, method 400 is the same as method 300 except for live sensor data 402. In the example of method 400, when live sensor data 402 corresponding to live image 302 is available, estimating PDF 108 may include taking into account live sensor data 402. Thus, instead of generating a PDF that produces estimated sensor point cloud distribution 450 based solely on analysis of live image 302 as in method 300, in method 400, an estimated PDF and / or estimated sensor point cloud distribution 450 is created or modified based on live sensor data 402 in addition to live image 302.
[0077] For example, in some embodiments, a correlation between the live sensor data 402 and the estimated PDF or the estimated sensor point cloud distribution 450 is determined, and the estimated PDF or the estimated sensor point cloud distribution 450 is created or modified based on the correlation. For example, when a sensor point in the live sensor data 402 and a high likelihood region of the estimated sensor point cloud distribution 450 coincide, the estimated PDF or the estimated sensor point cloud distribution 450 may be modified based on the identified correlation, for example, by further increasing the likelihood of the high likelihood region corresponding to the sensor point in the live sensor data 402 and / or increasing the confidence associated with the high likelihood region corresponding to the sensor point in the live sensor data 402. Similarly, when a sensor point in the live sensor data 402 and a low likelihood region of the estimated sensor point cloud distribution 450 coincide, the estimated PDF or the estimated sensor point cloud distribution 450 may be modified based on the identified lack of correlation, for example, by increasing the likelihood of the low likelihood region corresponding to the sensor point in the live sensor data 402 and / or decreasing the confidence associated with the low likelihood region corresponding to the sensor point in the live sensor data 402. Furthermore, when no sensor point in the live sensor data 402 coincides with a high likelihood region of the estimated sensor point cloud distribution 450, the estimated PDF or the estimated sensor point cloud distribution 450 may be modified based on the identified lack of correlation, for example, by decreasing the likelihood of the high likelihood region in the live sensor data 402 lacking any coincident sensor points and / or decreasing the confidence associated with the high likelihood region in the live sensor data 402 lacking any coincident sensor points. It is worth noting that although "live" images and "live" sensor data are referred to herein for the sake of clarity and convenience, previously recorded images and, optionally, previously recorded sensor data corresponding to the recorded images may be used in place of "live" images and data in the various methods disclosed herein to generate an estimated PDF and / or an estimated sensor point cloud distribution of the recorded images and data.
[0078] Figure 5 A method 500 for generating and using an estimated sensor point cloud distribution based on visual sensor data is shown in accordance with some embodiments. Figure 5 As shown, method 500 includes receiving visual sensor data at block 502, such as Figure 2 At block 504, method 500 includes processing the visual sensor data to estimate a sensor point cloud distribution, such as a point cloud PDF or other parameters, wherein the processing is performed using a deep learning network that may be trained using only the visual sensor training data and other training data corresponding to the visual sensor training data as input training data, such as using Figure 1 At block 506, method 500 includes training a deep learning network based on a method similar to Figure 4 The estimated sensor point cloud distribution of the estimated sensor point cloud distribution 450 produces an estimated sensor point cloud distribution. At block 508, the method 500 includes utilizing the estimated sensor point cloud distribution.
[0079] The estimated sensor point cloud distribution can be used as the basis for any of a number of applications, ranging from notifying vehicle occupants of an approaching object or a turn identified in the estimated sensor point cloud distribution, to directly controlling the vehicle, such as applying brakes or acceleration to avoid a collision with an approaching object or automatically turning a turn when an impending turn is identified, such as via a steering wheel. Figure 4 As discussed, in some embodiments, the estimated sensor point cloud distribution is used to improve the live sensor data. In other embodiments, the estimated PDF or the estimated sensor point cloud distribution is modified based on the live sensor data to produce an improved estimated sensor point cloud distribution. In some embodiments, the estimated PDF or the estimated sensor point cloud distribution is used to increase the granularity and / or dimension of the live sensor data, for example, by adding sensor points and / or converting 2D (or 3D) live sensor data into 3D (or 4D) live sensor data. In some embodiments, the estimated PDF or the estimated sensor point cloud distribution is used together with the corresponding visual sensor data as training data for other machine learning applications. Therefore, the method disclosed herein can be used not only to generate the sensor data from the visual sensor data for the purpose of utilizing the sensor data in live or "real-time" applications, but also to generate a large amount of sensor data for the visual sensor data in order to enable other deep learning applications that may require such a number of corresponding sensor data and visual sensor data.
[0080] In some embodiments, certain aspects of the techniques described above may be implemented by one or more processors of a processing system that executes software. The software includes one or more sets of executable instructions stored or otherwise tangibly embodied on a non-transitory computer-readable storage medium. The software may include instructions and certain data that, when executed by the one or more processors, manipulate the one or more processors to perform one or more aspects of the techniques described above. The non-transitory computer-readable storage medium may include, for example, a magnetic or optical disk storage device, a solid-state storage device such as a flash memory, a cache memory, a random access memory (RAM), or one or more other non-volatile memory devices, etc. The executable instructions stored on the non-transitory computer-readable storage medium may be in the form of source code, assembly language code, object code, or in the form of other instruction formats that can be interpreted or otherwise executed by one or more processors.
[0081] Computer-readable storage media may include any storage media or combination of storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, tapes, or magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache memory), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or micro-electromechanical system (MEMS)-based storage media. Computer-readable storage media may be embedded in a computing system (e.g., system RAM or ROM), fixedly attached to a computing system (e.g., a magnetic hard drive), removably attached to a computing system (e.g., an optical disc or flash memory based on a universal serial bus (USB)), or coupled to a computer system via a wired or wireless network (e.g., a network accessible storage device (NAS)).
[0082] It should be noted that not all activities described in the general description above are required, a particular activity or part of a device may not be required, and one or more additional activities in addition to the described activities may be performed, or one or more additional elements in addition to the described elements may be included. Furthermore, the order in which the activities are listed is not necessarily the order in which they are performed. Moreover, the concepts have been described with reference to specific embodiments. However, it is understood by those of ordinary skill in the art that various modifications and changes can be made without departing from the scope of the present disclosure as set forth in the appended claims. Therefore, the description and drawings should be viewed in an illustrative rather than a restrictive sense, and it is expected that all such modifications are included within the scope of the present disclosure.
[0083] The benefits, other advantages and solutions to the problems are described above with respect to specific embodiments. However, the benefits, advantages, solutions to the problems and any features that may cause any benefit, advantage or solution to appear or become more obvious should not be interpreted as essential, necessary or basic features of any or all claims. In addition, the specific embodiments disclosed above are merely illustrative, because the disclosed subject matter can be modified and practiced in different but equivalent ways, which is obvious to those skilled in the art who benefit from the teachings of this article. Except for the limitations described in the appended claims, it is not desired to limit the details of the construction or design shown in this article. It is therefore obvious that the specific embodiments disclosed above can be changed or modified, and all such changes are considered to be within the scope of the disclosed subject matter. Therefore, the protection sought herein is as set forth in the appended claims.
Claims
1. A method, characterized in that include: Receive visual sensor data; processing the visual sensor data to estimate a sensor point cloud distribution, wherein the processing is performed using a deep learning network capable of being trained using only the visual sensor training data and other training data corresponding to the visual sensor training data as input training data; as well as An estimated sensor point cloud is generated based on the estimated sensor point cloud distribution.
2. The method according to claim 1, characterized in that The visual sensor data is the visual sensor training data, and the method further comprises: comparing the estimated sensor point cloud distribution with the other training data corresponding to the vision sensor training data to obtain a loss function; and The deep learning network is updated based on the loss function.
3. The method according to claim 1, characterized in that Also included is an implementation of generating a sensor point cloud from the estimated sensor point cloud distribution as having a likelihood value exceeding a threshold.
4. The method according to claim 1, characterized in that The visual sensor data is the visual sensor training data, and the method further comprises: comparing the estimated sensor point cloud distribution with the other training data corresponding to the vision sensor training data to obtain a loss function; As one implementation, generating a sensor point cloud from the estimated sensor point cloud distribution; and The likelihood value in the sensor point cloud is compared with a predetermined threshold to determine whether the training of the deep learning network is complete.
5. A non-transitory computer-readable medium embodying a set of executable instructions, characterized in that: The set of executable instructions is used to operate at least one processor to perform the following operations: Receive visual sensor data; processing the visual sensor data to estimate a sensor point cloud distribution, wherein the processing is performed using a deep learning network capable of being trained using only the visual sensor training data and other training data corresponding to the visual sensor training data as input training data; as well as An estimated sensor point cloud is generated based on the estimated sensor point cloud distribution.
6. The non-transitory computer-readable medium of claim 5, wherein: The vision sensor data is the vision sensor training data, and the set of executable instructions additionally operates the at least one processor to: comparing the estimated sensor point cloud distribution with the other training data corresponding to the vision sensor training data to obtain a loss function; and The deep learning network is updated based on the loss function.
7. The non-transitory computer-readable medium of claim 5, wherein: The vision sensor data is the vision sensor training data, and the set of executable instructions additionally operates the at least one processor to: comparing the estimated sensor point cloud distribution with the other training data corresponding to the vision sensor training data to obtain a loss function; As one implementation, generating a sensor point cloud from the estimated sensor point cloud distribution; and The likelihood value in the sensor point cloud is compared with a predetermined threshold to determine whether the training of the deep learning network is complete.
8. A device comprising a processor, characterized in that: The processor is configured to: Receive visual sensor data; processing the visual sensor data to estimate a sensor point cloud distribution, wherein the processing is performed using a deep learning network capable of being trained using only the visual sensor training data and other training data corresponding to the visual sensor training data as input training data; as well as An estimated sensor point cloud is generated based on the estimated sensor point cloud distribution.
9. The device according to claim 8, characterized in that The vision sensor data is the vision sensor training data, and the processor is further configured to: comparing the estimated sensor point cloud distribution with the other training data corresponding to the vision sensor training data to obtain a loss function; and The deep learning network is updated based on the loss function.
10. The device according to claim 8, characterized in that The vision sensor data is the vision sensor training data, and the processor is further configured to: comparing the estimated sensor point cloud distribution with the other training data corresponding to the vision sensor training data to obtain a loss function; As one implementation, generating a sensor point cloud from the estimated sensor point cloud distribution; and The likelihood value in the sensor point cloud is compared with a predetermined threshold to determine whether the training of the deep learning network is complete.