Method for processing multiple pieces of echo data, device and computer program product

Abstract

A method for processing multiple pieces of echo data, a device, and a computer program product, for processing multiple pieces of echo data generated by a light detection and ranging device. The method (300) comprises: for each pixel point in a depth map of an environment generated by a light detection and ranging device, obtaining a plurality of pieces of echo data (310); determining the probability that particular echo data among the plurality of pieces of echo data corresponds to an effective echo or a noise echo, thereby forming a confidence map corresponding to the depth map (320); and on the basis of the confidence map, generating a saliency map corresponding to the depth map (330), wherein the saliency map is used for indicating the amount of effective information in a corresponding part of the depth map of the environment.

Description

Method, apparatus and computer program product for processing multi-echo data Technical Field

[0001] The present disclosure generally relates to multi-echo data processing, including techniques for processing multi-echo data generated by light detection and ranging devices. Background Art

[0002] Light detection and ranging equipment is used to detect and locate objects. Its working process involves emitting light through a transmitter and receiving light reflected from a target object through a receiver. The distance to the target object is measured based on the time difference between light emission and reception (i.e., time of flight, ToF). The receiver records the number of photons received at different times, which is called echo data. After the light is reflected by the object, an echo is formed on the echo data. The distance to the target object can be determined by the position of this echo. Light detection and ranging equipment can also present the three-dimensional structural information of the target object by analyzing the amount of reflected energy on the surface of the target object, the amplitude, frequency, and phase of the reflected spectrum, and other information.

[0003] When the light source is laser, the light detection and ranging device is called LiDAR (Light Detection and Ranging).

[0004] Light detection and ranging devices such as lidar (LiDAR) can generate multi-echo data corresponding to individual pixels. This data is expected to be used effectively and efficiently in applications such as distance measurement of target objects and construction of three-dimensional structures. Summary of the Invention

[0005] A first aspect of the present disclosure relates to a method for processing multiple echo data. According to one embodiment, the method includes: obtaining multiple echo data for each pixel in a depth map of an environment generated by a light detection and ranging device; determining the probability that corresponding echo data in the multiple echo data corresponds to a valid echo or a noise echo, thereby forming a confidence map corresponding to the depth map of the environment; and generating a saliency map corresponding to the depth map of the environment based on the confidence map, wherein the saliency map is used to indicate the amount of valid information in a corresponding portion of the depth map of the environment. The first aspect of the present disclosure also relates to a light detection and ranging device for performing the method.

[0006] A second aspect of the present disclosure relates to a method for training a neural network model. According to one embodiment, the method includes: obtaining a data sample set, where the data sample set includes multiple echo data for each pixel in a depth map generated by multiple light detection and ranging devices; for each pixel, labeling the corresponding data sample as a valid echo or a noise echo using a pixel-specific noise probability model; and training the neural network model using the data sample set and the corresponding labels as training data. The second aspect of the present disclosure also relates to an electronic device for performing the method.

[0007] A third aspect of the present disclosure relates to a method for establishing a noise probability model. According to one embodiment, the method includes: obtaining a data sample set, where the data sample set includes multiple echo data for a first pixel in a depth map generated by multiple light detection and ranging devices; and determining parameters of a noise probability model specific to the first pixel based on the multiple echo data for the first pixel. The third aspect of the present disclosure also relates to an electronic device for performing the method.

[0008] The present disclosure also relates to a computer program product comprising one or more instructions. When executed by a processor, the one or more instructions implement various methods according to embodiments of the present disclosure. For example, the methods include a method for processing echo data, a method for training a neural network model, and a method for establishing a noise probability model.

[0009] The above summary is provided to summarize some exemplary embodiments in order to provide a basic understanding of various aspects of the subject matter described herein. Therefore, the above features are merely examples and should not be construed as narrowing the scope or spirit of the subject matter described herein in any way. Other features, aspects, and advantages of the subject matter described herein will become apparent from the detailed description described below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] A better understanding of the present disclosure may be obtained when the following detailed description of the embodiments is considered in conjunction with the accompanying drawings. The same or similar reference numerals are used in the various drawings to represent the same or similar components. The accompanying drawings, together with the following detailed description, are incorporated into and form a part of this specification and are used to illustrate the embodiments of the present disclosure and to explain the principles and advantages of the present disclosure. In particular:

[0011] FIG. 1 shows an example of a light detection and ranging device according to an embodiment of the present disclosure.

[0012] FIG2A shows a schematic diagram of an application scenario for light detection and ranging according to an embodiment of the present disclosure.

[0013] FIG2B is a schematic diagram showing echo data obtained in the application scenario of FIG2A according to an embodiment of the present disclosure.

[0014] FIG3 illustrates an example method for processing multi-echo data according to an embodiment of the present disclosure.

[0015] 4A and 4B illustrate examples of confidence maps corresponding to a depth map of an environment according to an embodiment of the present disclosure.

[0016] FIG. 4C illustrates an example of a saliency map corresponding to a depth map of an environment, according to an embodiment of the present disclosure.

[0017] FIG5 illustrates an example method for training a neural network model according to an embodiment of the present disclosure.

[0018] FIG6 shows an example of a neural network model according to an embodiment of the present disclosure.

[0019] FIG7 shows an example of a convolutional neural network model according to an embodiment of the present disclosure.

[0020] FIG8 shows an example of a simulation model for light detection and ranging according to an embodiment of the present disclosure.

[0021] FIG9 illustrates an example method for establishing a noise probability model according to an embodiment of the present disclosure.

[0022] FIG10 shows an example of a noise probability model according to an embodiment of the present disclosure.

[0023] FIG. 11 shows an example block diagram of an electronic device for implementing various methods according to an embodiment of the present disclosure.

[0024] While the embodiments described in this disclosure may be susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and are herein described in detail. However, it should be understood that the drawings and detailed description thereof are not intended to limit the embodiments to the particular forms disclosed, but on the contrary, the intent is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the claims. DETAILED DESCRIPTION

[0025] The following describes representative applications of various aspects of the apparatus and method of the present disclosure. The description of these examples is only to add context and help understand the described embodiments. Therefore, it is clear to those skilled in the art that the embodiments described below can be implemented without some or all of the specific details. In other cases, well-known process steps are not described in detail to avoid unnecessarily obscuring the described embodiments. Other applications are also possible, and the solutions of the present disclosure are not limited to these examples.

[0026] Light Detection and Ranging Device Examples

[0027] FIG. 1 illustrates an example of a light detection and ranging device and its operation according to an embodiment of the present disclosure.

[0028] In the example of FIG1 , the light detection and ranging device 100 may include a transmitter 110, a receiver 120, a processor 130, and a memory 140. The light detection and ranging device 100 may be configured to detect and measure the distance to objects in the surrounding environment by emitting and receiving light signals, and may even perform three-dimensional modeling. For example, the transmitter 110 may be configured to transmit a light signal 115 via a light source. The transmitted light signal 115 propagates through a medium to a target object 160 in the environment and reflects from the object 160. The reflected light signal 115′ propagates back through the medium to the light detection and ranging device 100 and is received by the receiver 120. In one embodiment, the transmitter 110 and the receiver 120 may each include an optical lens (not shown). The receiver 120 may include a detector to detect the received light signal 115′. In one embodiment, the light source is configured to emit a laser signal having a specific sequence of pulses. In this manner, the light detection and ranging device 100 functions as a laser radar (LiDAR).

[0029] In an example, the processor 130 may be configured to be coupled to the transmitter 110, the receiver 120, and the memory 140. The processor 130 may execute one or more modules and / or processes to enable the light detection and ranging device 100 to perform various functions. The functions include controlling the transmission and reception of light signals (e.g., laser signals) and various functions described below. For example, the processor 130 may be configured to perform various functions by reading and executing computer programs, codes, or executable instructions stored in the memory 140. In some embodiments, the processor 130 may include a microprocessor, a microcontroller, a digital signal processor, a central processing unit (CPU), a graphics processing unit (GPU), etc.

[0030] The memory 140 may be a non-transitory computer-readable storage medium, including but not limited to an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, and any suitable combination thereof.

[0031] FIG1 shows only one target object 160. The target object can be any type of detection object, including but not limited to trees, furniture, roadblocks, vehicles, pedestrians, artifacts, etc. In real-world situations, there are often multiple objects of various types in the environment, complicating the detection, ranging, and 3D modeling of the target object.

[0032] Figure 2A is a schematic diagram illustrating an application scenario for light detection and ranging according to an embodiment of the present disclosure. Those skilled in the art will appreciate that the application scenario illustrated in Figure 2A is merely an example, and the application scenarios for light detection and ranging disclosed herein are not limited thereto. Figure 2B is a schematic diagram illustrating echo data obtained in the application scenario of Figure 2A according to an embodiment of the present disclosure. This example application scenario will be described below in conjunction with the light detection and ranging device 100 in Figure 1.

[0033] As shown in FIG2A , in a light detection and ranging (LDR) scheme, transmitter 110 is configured to transmit light signals 210 and 220. Light signal 210 propagates through a medium, reaches the surface of a leading object, and is reflected from the object. Reflected light signal 210′ ​​propagates through the medium and returns to LDR device 100, where it is received by receiver 120. Light signal 220 propagates through the medium and reaches the edge of the leading object. Subsequently, a portion of light signal 220 is reflected from the edge of the leading object, and reflected light signal 220′ returns to LDR device 100 and is received by receiver 120. Another portion of light signal 220 continues to propagate through the medium, reaches a trailing object, and is reflected from the surface of the trailing object. Reflected light signal 220″ returns to LDR device 100 and is received by receiver 120. Receiver 120 can be configured to record the number of photons received at different times, thereby generating echo data.

[0034] The light detection and ranging device 100 can have multiple pixels corresponding to objects in the field of view. Here, the meaning of a pixel is similar to that of a pixel in image sensing, but the data recorded for the pixel is not the RGB color components, but the number of received photons (i.e., echo data). Each pixel can correspond to one or more echo data. In the example of Figure 2A, the light signal 210 is received by the receiver 120 after hitting point A1 on the front object, thereby generating one echo data of the corresponding pixel. In the example, the pixel corresponding to point A1 on the front object can be represented as pixel A1. The light signal 220 is received by the receiver 120 after hitting point A2 on the front object and point B2 on the rear object, thereby generating two echo data of the corresponding pixel. In the example, the pixel corresponding to point A2 on the front object and point B2 on the rear object can be represented as pixel A2 / B2.

[0035] Pixels can correspond one-to-one with the field of view of the light detection and ranging device 100, and thus can be represented and distinguished using the field of view. Furthermore, the density of pixels can reflect the angular resolution of the light detection and ranging device 100. For simplicity, FIG2A only shows two pixel points A1 and A2 / B2 generated by the light detection and ranging device 100. In one or more embodiments, a larger number of pixels may be generated depending on the configuration of the light detection and ranging device 100.

[0036] FIG2A shows only the optical signal emitted by transmitter 110 and the real object echo. In this disclosure, the real object echo is referred to as the valid echo. However, receiver 120 generally also receives optical signals originating from ambient light (e.g., sunlight). These optical signals constitute noise echoes relative to the real object echo. Noise echoes can interfere with the detection of valid echoes, thereby affecting the accuracy of valid echo detection and even preventing the detection of valid echoes.

[0037] FIG2B shows multiple echo data corresponding to two pixel points A1 and A2 / B2 through examples (1) and (2), respectively. In the example of FIG2B, the horizontal axis represents time, and thus can indicate the flight time of the echo. The vertical axis represents the number of received photons, and thus can indicate the echo intensity. In the example, there may also be some lower peaks. Since these lower peaks do not meet the echo judgment criteria (such as intensity and duration conditions), they can be ignored.

[0038] Referring to example (1), echo data a corresponds to the light signal reflected from point A1 of the front object and is therefore a valid echo. Echo data b and c both correspond to noise echoes. Referring to example (2), echo data d corresponds to the light signal reflected from point A2 of the front object and echo data e corresponds to the light signal reflected from point B2 of the rear object and are therefore both valid echoes. Echo data f corresponds to a noise echo.

[0039] It should be understood that when the peak value is high, the noise echo will be mistakenly identified as a valid echo, thereby interfering with the detection of the valid echo. In examples (1) and (2), the noise echo b, c or f may affect the detection accuracy of the corresponding valid echo. This is obviously disadvantageous for the detection and ranging of the target object and the three-dimensional modeling. It should also be understood that multiple valid echoes of a single pixel can convey more useful information. For example, these multiple valid echoes may correspond to multiple target objects. Therefore, detecting and excluding noise echoes for a single pixel, and / or detecting and identifying multiple valid echoes, will be beneficial for the detection and ranging of the target object and the surrounding environment and the three-dimensional modeling.

[0040] In the embodiments of the present disclosure, a confidence score is generated for the echo data of each pixel in the depth map to assess whether the echo data is a noise echo. Based on this assessment, the noise echo at that pixel can be excluded, retaining only the valid echo. On the one hand, this can reduce the interference of noise echoes on valid echoes, improving detection performance. On the other hand, it can reduce the computing and storage resource requirements that would otherwise be caused by processing and storing noise echoes at the pixel level.

[0041] In an embodiment of the present disclosure, a significance score is formed for each pixel or pixel set in the depth map to evaluate the effective information content (e.g., the number of effective echoes) of the pixel or pixel set. Based on this evaluation, the echo data of the pixel or pixel set with high effective information content can be processed preferentially. By reducing the processing of the echo data of the pixel with low effective information content, echo data processing resources and even storage resources can be saved at the level of the entire depth map.

[0042] Still referring to FIG. 1 , according to an embodiment of the present disclosure, the memory 140 of the light detection and ranging device 100 may include modules, such as an echo data acquisition module 142, a confidence analysis module 144, and a significance analysis module 146, in order to improve the performance of the light detection and ranging device 100 in the above-mentioned and other aspects.

[0043] As an example, the echo data acquisition module 142 can be configured to obtain a plurality of echo data for each pixel in the depth map of the environment generated by the light detection and ranging device 100. The confidence analysis module 144 can be configured to determine the probability that corresponding echo data among the plurality of echo data corresponds to a valid echo or a noise echo, thereby forming a confidence map corresponding to the depth map of the environment. The saliency analysis module 146 can be configured to generate a saliency map corresponding to the depth map of the environment based on the confidence map, wherein the saliency map is used to indicate the amount of valid information in the corresponding portion of the depth map of the environment. The detailed operation of each module can be understood in conjunction with the further description of the embodiments of the present disclosure.

[0044] In the embodiments, the term module is used to represent an example division of executable instructions to facilitate discussion. It should be noted that module division can be performed in different ways, and one or more functions can be arranged in different ways (for example, combined into a smaller number of modules, divided into a larger number of modules, etc.). In addition, the functions and modules described herein can be implemented in whole or in part by software and / or firmware executable on a processor, or can be implemented in whole or in part by hardware (for example, dedicated processing circuitry, etc.).

[0045] Multi-echo data processing example

[0046] 3 shows an example method for processing multi-echo data according to an embodiment of the present disclosure. The example method 300 can be performed by various light detection and ranging devices. The example operations of the method 300 are described below in conjunction with the light detection and ranging device 100.

[0047] As shown in Figure 3, method 300 may include obtaining multiple echo data (310) for each pixel in the depth map of the environment generated by the light detection and ranging device 100. The pixel is, for example, pixel A2 / B2 in Figure 2A, and the multiple echo data include, for example, echo data d, e, and f. Generally speaking, the echo data obtained are echo data that meet specific judgment criteria. For example, the judgment criteria may require that the intensity of the echo is higher than a first predetermined threshold, and / or the duration of the echo is longer than a second predetermined threshold. It should be understood that any number of echo data can be obtained for a single pixel, such as 4, 5, 6 or more. In one or more embodiments, the echo data may include information such as the echo start position, echo peak position, echo end position, and light intensity of the corresponding echo data.

[0048] Accordingly, the method 300 may include determining the probability that the corresponding echo data in the plurality of echo data corresponds to a valid echo or a noise echo, thereby forming a confidence map (320) corresponding to the plurality of pixels in the depth map. Further, the echo data may include the probability that the corresponding echo data corresponds to a valid echo or a noise echo, or include a mark for indicating whether the corresponding echo data is a valid echo or a noise echo. As described above, for light detection and ranging equipment, the light signal caused by the propagation of ambient light (such as sunlight) will constitute a noise echo. The noise echo will interfere with the detection of the valid echo, thereby affecting the detection accuracy of the valid echo, and even causing the inability to detect the valid echo. In the example of FIG2B , the peak difference of the noise echo b relative to the valid echo a or the noise echo f relative to the valid echo e may not be sufficient to always pick out the noise echo. Accordingly, it is desirable to effectively distinguish between valid echoes and noise echoes. In one or more embodiments, a neural network model may be used to distinguish between valid echoes and noise echoes. For example, using a neural network model to distinguish valid echoes from noise echoes may include using a unified neural network model for multiple pixels in the depth map to distinguish valid echoes from noise echoes. Alternatively, in one or more embodiments, a pixel-specific noise probability model may be used to distinguish valid echoes from noise echoes. For example, using a noise probability model to distinguish valid echoes from noise echoes may include using multiple pixel-specific noise probability models for multiple pixels in the depth map to distinguish valid echoes from noise echoes.

[0049] In an embodiment, a confidence map can be used to represent the probability or label of multiple echo data of each pixel corresponding to a valid echo or a noise echo. Figures 4A and 4B show examples of confidence maps corresponding to a depth map of an environment according to an embodiment of the present disclosure. In this example, it is assumed that the light detection and ranging device 100 has a pixel configuration of 3 rows by 3 columns. Accordingly, the depth map can have 9 pixels in 3 rows by 3 columns. In this example, each row corresponds to a different pixel. Pixel (1, 1) corresponds to the pixel in the 1st row and 1st column, and so on. The column corresponds to multiple (for example, 6) echo data. The cell where the rows and columns intersect corresponds to a single echo data detected at the corresponding pixel. It is easy to understand that the above pixel configuration and the number of echo data are only examples. In one or more embodiments, there may be a higher or lower pixel configuration. The number of echo data for each pixel may also be more or less, and the number of echo data between multiple pixels may not necessarily be the same.

[0050] In the example of FIG4A , the probability in the cells where rows and columns intersect represents the possibility that the single echo data at the corresponding pixel point corresponds to a valid echo or a noise echo. As shown in FIG4A , the probability that the echo data 1 at pixel point (1, 1) corresponds to a valid echo is 0.95 (or the probability that it corresponds to a noise echo is 0.05). The probability that the echo data 6 detected at pixel point (2, 3) corresponds to a valid echo is 0.13 (or the probability that it corresponds to a noise echo is 0.87). In one or more embodiments, one or more confidence thresholds can be pre-set so that echo data with a probability corresponding to a valid echo higher than the corresponding confidence threshold is determined as a valid echo.

[0051] In the example of FIG4B , the marks in the cells where the rows and columns intersect indicate whether the single echo data at the corresponding pixel corresponds to a valid echo or a noise echo. The example of FIG4B is based on the probability information in FIG4A and a confidence threshold of 0.6. Accordingly, the echo data 1 detected at pixel (1, 1) has the mark of a valid echo (i.e., 1). The echo data 6 detected at pixel (2, 3) has the mark of a noise echo (i.e., 0).

[0052] It should be understood that generating multiple echo data based on a single pixel in a depth map will multiply the amount of data, requiring more resources for storage and processing. The increased resource requirements will be more significant when there are many pixels. In one or more embodiments, the storage and processing resource requirements for multiple echo data can be controlled to an appropriate level by excluding noise echoes from a single pixel. For example, echo data 3, 5, and 6 for pixel (1, 1) can be excluded.

[0053] As shown in FIG3 , method 300 may further include generating a saliency map (330) corresponding to the depth map based on the confidence map. The saliency map is used to indicate the saliency score of a corresponding pixel or set of pixels in the depth map of the environment. The saliency score is used to indicate the amount of effective information of the corresponding pixel or set of pixels. In one embodiment, the amount of effective information may refer to the number of effective echoes or the proportion of effective echoes to all the obtained echoes.

[0054] FIG4C shows an example of a saliency map corresponding to a depth map of an environment according to an embodiment of the present disclosure. This example is based on the same depth map pixels as FIG4A and FIG4B , i.e., having 3 rows by 3 columns, totaling 9 pixels. Each pixel corresponds to 6 echo data. In this example, there are 3 valid echoes among the 6 echo data of pixel (1, 1), and its saliency score is 3 or 1 / 2. There are 5 valid echoes among the 6 echo data of pixel (2, 3), and its saliency score is 5 or 5 / 6.

[0055] It should be understood that the higher the significance score, the more effective information it corresponds to. Therefore, the echo data of pixels or pixel sets with high effective information can be processed first. For example, in terms of allocating storage and processing resources, higher priority can be given to pixels or pixel sets with high effective information, and the priority of pixels or pixel sets with low significance scores can be reduced. In this way, the processing resource requirements and even storage resource requirements of the echo data can be controlled at an appropriate level at the level of the entire depth map. For example, assuming that a threshold is pre-set to 3, pixels above or not below the threshold can be processed or stored first. In Figure 4C, there are 6 such pixels, accounting for 2 / 3 of the total. The processing and even storage resources of the remaining 1 / 3 of the pixels can be saved.

[0056] In one embodiment, the echo data obtained by the light detection and ranging device 100 is used for data fusion with data obtained by other sensors. For example, in autonomous driving applications, it is generally necessary to fuse lidar data and camera data to compensate for the shortcomings of both sensors and improve the accuracy and quality of the application. By using the saliency score of a single pixel or the entire saliency map, only valid echoes from pixels with saliency scores above a certain threshold can be included in the data fusion. This can be beneficial for reducing the computing resource requirements of data fusion.

[0057] In one embodiment, a saliency map can be output in real time based on the user's region of interest. For example, a user can determine their region of interest based on a specific application scenario. For example, for an indoor scene, the region of interest might be a cube with a side length of 3 meters. For an outdoor scene, the region of interest might be a cube with a side length of 10 meters. After selecting the region of interest, the light detection and ranging device 100 outputs a saliency map for the region of interest in real time based on the global confidence map and multiple echo data. Subsequently, the multi-echo data can be stored and processed based on the real-time saliency map for the region of interest.

[0058] Neural Network Model Training Example

[0059] In embodiments of the present disclosure, a neural network model can be used to distinguish valid echoes from noise echoes. By ensuring the performance of the neural network model through training, valid echoes from noise echoes can be quickly and accurately distinguished, achieving real-time and accurate operation. Figure 5 illustrates an example method for training a neural network model according to an embodiment of the present disclosure. Example method 500 can be performed by any computer or electronic device.

[0060] As shown in FIG5 , method 500 may include obtaining a first set of data samples (510). The first set of data samples includes a plurality of echo data for each pixel in a depth map generated by a plurality of light detection and ranging devices. In one or more embodiments, the first set of data samples is obtained during a plurality of frames. Continuous dynamic data from a plurality of frames may enhance the diversity of the data samples.

[0061] As shown in FIG5 , method 500 may include, for each pixel, using a noise probability model to label the corresponding data sample as either a valid echo or a noise echo (520). Because the propagation environment of the light signal at each pixel and the characteristics of the surface of the object it impacts are independent of each other, in one or more embodiments, the noise probability model used is pixel-specific. This can improve the accuracy of the labels added to the data samples.

[0062] As shown in FIG5 , method 500 may further include training a neural network model using the first set of data samples and corresponding labels as training data ( 530 ). Specifically, the multi-echo data of each pixel may be used as input to the neural network model, and the output of the neural network model may be compared with the labels of the data samples. Based on the comparison results, the parameters and structure of the neural network model may be adjusted to achieve training of the neural network model.

[0063] Figure 6 shows an example of a neural network model according to an embodiment of the present disclosure. As shown in Figure 6, the neural network model 600 includes multiple layers, including an input layer 601, an output layer 606, and intermediate layers (or hidden layers) 602 to 605. Each layer has a certain number of neurons, each neuron has a specific weight, and there are connections between neurons in different layers. When the input value 620 is input into the neural network model 600, the corresponding numerical value is first received by the neurons at the input layer 601, and the corresponding numerical value is propagated to the neurons in the intermediate layer 602 through the connection with the neurons in the next layer. The neurons in the intermediate layer 602 calculate the weighted sum of the output values ​​of the neurons in the previous layer, and output the weighted sum to the neurons in the next intermediate layer 603 through the connection with the neurons in the next layer. And so on, until the neurons in the output layer 606 calculate the weighted sum of the output values ​​of the neurons in the previous layer, and output the inference result 640 for the input value 620.

[0064] In the example of Figure 6, the neural network model 600 has four intermediate layers 602 to 605. Depending on the application requirements, the intermediate layers can be any number, and the present disclosure does not need to limit this. When there are more than a certain number of intermediate layers, the neural network model is also called a deep neural network model. The neural network model 600 is composed of a series of fully connected layers (i.e., all outputs are connected to all inputs) and is called a multi-layer perceptron (MLP) model. As a further example, the neural network model also includes convolutional neural networks (CNNs) and recurrent neural networks (RNNs) models, etc. Figure 7 shows an example of a convolutional neural network (CNN) model according to an embodiment of the present disclosure. As shown in Figure 7, the CNN model 700 is divided into four modules, namely, an input module, a preprocessing module, a convolution module, and an output module.

[0065] In one or more embodiments, a unified neural network model can be trained based on data samples from a variety of environments. This allows for the use of a unified neural network model to distinguish valid echoes from noise echoes for pixels in different depth maps, or for different pixels in the same depth map. This simplifies the process of distinguishing valid echoes from noise echoes.

[0066] In one or more embodiments, a first set of data samples can be obtained using a real light detection and ranging device and a simulation model. That is, the first set of data samples can include real data and simulated data obtained by the real device and the simulation model for one or more environments. For example, a real device or simulation model can be used to continuously generate multi-echo data for each pixel at the same location over multiple frames. During the simulation, device and object positions or environmental settings can be adjusted to obtain more diverse data samples.

[0067] Figure 8 shows an example of a simulation model for light detection and ranging according to an embodiment of the present disclosure. This example simulation model can be established using tools such as Matlab, Python, Unreal Engine, and Autoware.

[0068] As shown in Figure 8, the following physical components of the component need to be simulated: the emitter, the transmitting optical device, the photon detector, and the receiving optical device. The transmitting optical device includes a lens and cover glass. The photon detector includes sensing, filtering, and processing units. The receiving optical device includes a lens, filter, and glass. The background environment, target objects, and other objects also need to be simulated. The background environment includes optical noise. The intensity of the optical noise can be controlled to simulate different ambient light conditions.

[0069] By simulating the optical signal transmission, propagation, and reception phases of the light detection and ranging scheme, a sufficient number and diversity of data samples can be obtained. This helps address issues such as insufficient real data samples or the time-consuming process of obtaining sufficient and diverse data samples. Furthermore, the first set of data samples still includes real data samples, ensuring that the neural network model training still reflects the real environment.

[0070] In one embodiment, the neural network model may be trained offline and the trained neural network model may be stored locally in the light detection and ranging device 100 for use.

[0071] Example of building a noise probability model

[0072] As described above, in an embodiment of the present disclosure, a pixel-specific noise probability model can be used to distinguish between valid echoes and noise echoes at corresponding pixel points. During neural network model training, a label indicating valid echoes or noise echoes can be added to corresponding data samples based on the differentiation result. FIG9 shows an example method for establishing a noise probability model according to an embodiment of the present disclosure. Because the propagation environment of the light signal at each pixel point and the characteristics of the surface of the object it impacts are independent of each other, a noise probability model can be established for a single pixel point. Example method 900 can be executed by any computer or electronic device.

[0073] As shown in FIG9 , method 900 may include obtaining a second set of data samples for a first pixel ( 910 ). The second set of data samples includes a plurality of echo data of the first pixel in a depth map generated by a plurality of light detection and ranging devices. In one or more embodiments, the second set of data samples is obtained during a plurality of frames. Continuous dynamic data of a plurality of frames may enhance the diversity of the data samples. Method 900 may also include determining parameters of a noise probability model specific to the first pixel based on the plurality of echo data of the first pixel ( 920 ).

[0074] In one or more embodiments, a real light detection and ranging device and a simulation model can be used to obtain the second set of data samples. That is, the second set of data samples can include real data and simulated data obtained by the real device and the simulation model for one or more environments. For example, the real device or the simulation model can be used to continuously generate multi-echo data for the first pixel at the same location over multiple frames. During the simulation, the device and object positions or environmental settings can be adjusted to obtain more diverse data samples.

[0075] For a specific pixel, multiple echo data corresponding to that pixel can be obtained from the raw data through histogram decomposition. The waveform of the echo data can be approximated as a Gaussian waveform with a tail. Figure 10 shows an example of a noise probability model according to an embodiment of the present disclosure. For the Gaussian distribution shown in Figure 10, the parameters to be determined include the mean μ and the standard deviation σ.

[0076] In the scheme for establishing a noise probability model based on data samples obtained from a real device, it is assumed that five echo data are obtained from the raw data of a single pixel, and based on empirical assumptions, the three echo data at the end are noise echoes, or the three echo data with lower peak values ​​are noise echoes. Based on this assumption and based on the data samples [144, 120, 114, 113, 112], it can be determined that the mean of the Gaussian distribution obeyed by the noise echo is 113 (i.e., (114 + 113 + 112) / 3). Furthermore, the standard deviation can be determined using the following empirical algorithm-based formula (1).

[0077] In formula (1), Min std represents the minimum standard deviation, Max peak represents the maximum peak value among multiple echo data, Mean represents the peak mean of multiple echo data, and First echo represents the peak value of the first echo data. In one or more embodiments, the mean and standard deviation of the Gaussian distribution can be estimated based on multiple frames of echo data to fit a noise probability model for a specific pixel.

[0078] In one or more embodiments, using the noise probability model includes determining whether the obtained plurality of echo data are valid echoes or noise echoes based on the noise probability model. In the above Gaussian distribution example, echo data with a peak value greater than or equal to the sum of the mean and standard deviation of the Gaussian distribution can be determined as valid echoes, and echo data with a peak value less than the sum of the mean and standard deviation can be determined as noise echoes.

[0079] It should be noted that due to limitations such as the bandwidth of the light detection and ranging equipment itself, real devices may not be able to output complete raw data, but can only output multiple echo data obtained through histogram decomposition. The noise obtained in this way may be an extreme value and may not represent the general situation. Moreover, the process of establishing the noise probability model mentioned above makes assumptions about the noise echo (i.e., the last three echoes), and the determination of the standard deviation is also based more on empirical algorithms. These factors will reduce the accuracy of the established noise probability model, and the corresponding parameters will be biased parameters. As a supplement or alternative, simulation can obtain raw data in the form of a histogram of a single pixel, thereby compensating for the limitation that real devices cannot output complete raw data.

[0080] Therefore, in one or more embodiments, a light detection and ranging device can be simulated, and based on the physical model of the simulated light detection and ranging device, raw data in the form of a histogram of individual pixels can be obtained. Subsequently, a set of data samples can be repeatedly sampled from the histogram to obtain a more accurate noise probability model.

[0081] The simulation model shown in FIG8 can be used to obtain a data sample set. As described above, the simulation can involve simulating the optical signal transmission, propagation, and reception stages. The following specifically describes examples of simulations for the optical signal transmission and propagation stages. Those skilled in the art will appreciate that the optical signal reception stage can be simulated based on the specific principles and parameters of different types of receiving devices (e.g., PD, APD, SAPD).

[0082] In the example simulation, the illumination caused by the signal light can be calculated by formula (2): in, and They represent the object illumination caused by the signal light, the object reflection parameters, and the lens parameters respectively.

[0083] The illumination caused by ambient light is calculated by formula (3):

[0084] Among them, E obj,amb Indicates the object illumination caused by ambient light. The meanings of other parameters are as follows: P t : optical power; Ω TX : light projection angle; ρ: object reflectivity; F: F value of the lens; η RX : optical system power; R: distance to the object.

[0085] The counting rate caused by the signal light is calculated by formula (4), that is, the number of photons caused by the signal light detected by the detector per unit time.

[0086] The counting rate caused by the signal ambient light is calculated by formula (5), that is, the number of photons caused by the ambient light detected by the detector per unit time.

[0087] The parameters have the following meanings: A pix : pixel area; h: Planck constant; c: speed of light; λ: wavelength of light; PDE: photon detection efficiency.

[0088] The probability of detecting a photon at least once in a time bin is calculated using Equation (6).

[0089] The parameters have the following meanings: CR sig : ideal counting rate of signal light; CR amb : ideal counting rate of ambient light; T bin : A bin time slot of the histogram.

[0090] p high The probability of a single detector diode being triggered within a time interval for each illumination is expressed as a Poisson distribution. Assuming the illumination is repeated N times, the probability of the detector diode being triggered follows a binomial distribution. Equations (7) and (8) represent the number of times the detector diode is triggered by the combined signal light and ambient light, and the number of times it is triggered by ambient light (i.e., noise), respectively.

[0091] The above process yields raw echo data containing noise for a single pixel. Next, the mean and variance of the Gaussian distribution can be determined based on the echo data. The following are examples of parameter values ​​used in the simulation.

[0092] In one embodiment, the noise probability model may be established offline, and the established noise probability model may be stored locally in the light detection and ranging device 100 for use.

[0093] Embodiments of the present disclosure also provide electronic devices. The electronic devices include one or more processors and one or more memories storing one or more instructions. When executed by the one or more processors, the one or more instructions cause the one or more processors to perform the method for processing multi-echo data, the method for training a neural network model, or the method for establishing a noise probability model according to embodiments of the present disclosure.

[0094] The present disclosure also provides a computer-readable storage medium having one or more instructions stored thereon. When executed by a processor, the one or more instructions cause the processor to perform the method for processing multi-echo data, the method for training a neural network model, or the method for establishing a noise probability model according to the present disclosure.

[0095] The present disclosure also provides a computer program product comprising one or more instructions. When executed by a processor, the one or more instructions cause the processor to perform the method for processing multi-echo data, the method for training a neural network model, or the method for establishing a noise probability model according to the present disclosure.

[0096] It should be understood that the instructions in the computer-readable storage medium according to the embodiments of the present disclosure can be configured to perform operations corresponding to the above-mentioned device and method embodiments. When referring to the above-mentioned device and method embodiments, the embodiments of the computer-readable storage medium are clear to those skilled in the art and are therefore not described again. Computer-readable storage media for carrying or including the above-mentioned instructions also fall within the scope of the present disclosure. Such computer-readable storage media may include, but are not limited to, floppy disks, optical disks, magneto-optical disks, memory cards, memory sticks, and the like.

[0097] The embodiments of the present disclosure also provide various devices including components or units for executing the steps of the method for processing multi-echo data, the method for training a neural network model, or the method for establishing a noise probability model in the above embodiments.

[0098] It should be noted that the above-mentioned various components or units are only logical modules divided according to the specific functions implemented by them, rather than being used to limit specific implementation methods, such as can be implemented in the form of software, hardware or a combination of software and hardware. In actual implementation, the above-mentioned various components or units can be implemented as independent physical entities, or can also be implemented by a single entity (for example, a processor (CPU or DSP, etc.), an integrated circuit, etc.). For example, a plurality of functions included in a unit in the above embodiments can be implemented by separate devices. Alternatively, a plurality of functions implemented by a plurality of units in the above embodiments can be implemented by separate devices respectively. In addition, one of the above functions can be implemented by a plurality of units.

[0099] In addition, it should be understood that the above series of processes and devices can also be implemented through software and / or firmware. In the case of implementation through software and / or firmware, the program constituting the software is installed from a storage medium or network to a computer with a dedicated hardware structure, such as the electronic device 1300 shown in Figure 11. When the various programs are installed, the electronic device can perform various functions, etc. Figure 11 shows an example block diagram of an electronic device for implementing various methods according to embodiments of the present disclosure.

[0100] 11 , a central processing unit (CPU) 1301 executes various processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage section 1308 to a random access memory (RAM) 13803. In the RAM 1303, data required when the CPU 1301 executes various processes and the like is also stored as needed.

[0101] The CPU 1301, the ROM 1302, and the RAM 1303 are connected to one another via a bus 1304. An input / output interface 1305 is also connected to the bus 1304.

[0102] The following components are connected to the input / output interface 1305: an input section 1306 including a keyboard, a mouse, etc.; an output section 1307 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1308 including a hard disk, etc.; and a communication section 1309 including a network interface card such as a LAN card, a modem, etc. The communication section 1309 performs communication processing via a network such as the Internet.

[0103] A drive 1310 is also connected to the input / output interface 1305 as needed. A removable medium 1311 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 1310 as needed so that a computer program read therefrom is installed in the storage section 1308 as needed.

[0104] In the case of realizing the above-described series of processing by software, a program constituting the software is installed from a network such as the Internet or a storage medium such as the removable medium 1311 .

[0105] Those skilled in the art will appreciate that such storage media are not limited to the removable medium 1311 shown in FIG11 , which stores the program therein and is distributed separately from the device to provide the program to the user. Examples of the removable medium 1311 include magnetic disks (including floppy disks (registered trademark)), optical disks (including compact disk read-only memories (CD-ROMs) and digital versatile disks (DVDs)), magneto-optical disks (including minidiscs (MDs) (registered trademark)), and semiconductor memories. Alternatively, the storage medium may be the ROM 1302, a hard disk included in the storage section 1308, or the like, in which the program is stored and distributed to the user together with the device containing the program.

[0106] It should be understood that the technical solution of the present disclosure can be implemented through the following example embodiments. 1. A method for processing multiple echo data, comprising: obtaining multiple echo data for each pixel in a depth map of an environment generated by a light detection and ranging device; determining the probability that corresponding echo data in the multiple echo data corresponds to a valid echo or a noise echo, thereby forming a confidence map corresponding to the depth map of the environment; and generating a saliency map corresponding to the depth map of the environment based on the confidence map, wherein the saliency map is used to indicate the amount of valid information in the corresponding part of the depth map of the environment. 2. The method according to clause 1, wherein determining the probability that the corresponding echo data corresponds to a valid echo or a noise echo includes using a neural network model to distinguish between valid echoes and noise echoes. 3. The method according to clause 2, wherein using a neural network model to distinguish between valid echoes and noise echoes includes: using a unified neural network model to distinguish between valid echoes and noise echoes for different pixels in the depth map. 4. The method of clause 1, wherein the neural network model is trained by: obtaining a first set of data samples, wherein the first set of data samples comprises a plurality of echo data for each pixel in a depth map generated by a plurality of light detection and ranging devices; for each pixel, labeling the corresponding data sample as a valid echo or a noise echo using a pixel-specific noise probability model; and training the neural network model using the first set of data samples and the corresponding labels as training data. 5. The method of clause 4, wherein the plurality of light detection and ranging devices comprises real devices and simulated models, the first set of data samples comprises real data and simulated data obtained by the real devices and the simulated models for one or more environments, and wherein the first set of data samples is obtained during a first plurality of frames. 6. The method of clause 4, wherein parameters of the pixel-specific noise probability model are determined by: obtaining a second set of data samples, wherein the second set of data samples comprises a plurality of echo data for a first pixel in the depth map generated by the plurality of light detection and ranging devices; and determining parameters of the first pixel-specific noise probability model based on the plurality of echo data for the first pixel. 7. The method of clause 6, wherein the plurality of light detection and ranging devices include real devices and simulated models, the second set of data samples includes real data and simulated data acquired by the real devices and the simulated models for one or more environments, and wherein the second set of data samples is acquired during a second plurality of frames. 8. The method of clause 1, wherein determining the probability that the corresponding echo data corresponds to a valid echo or a noise echo comprises using a pixel-specific noise probability model to distinguish between valid echoes and noise echoes.9. The method according to clause 1, further comprising: prioritizing, based on the saliency map, processing echo data from a portion of the depth map of the environment with high effective information content; and / or outputting the saliency map in real time based on a user's region of interest. 10. The method according to clause 1, wherein the neural network model is a convolutional neural network model, and / or the light detection and ranging device comprises a lidar. 11. A method for training a neural network model, comprising: obtaining a data sample set, wherein the data sample set comprises multiple echo data for each pixel in a depth map generated by multiple light detection and ranging devices; for each pixel, labeling the corresponding data sample as a valid echo or a noise echo using a pixel-specific noise probability model; and training the neural network model using the data sample set and the corresponding label as training data. 12. The method according to clause 11, wherein the multiple light detection and ranging devices comprise real devices and simulated models, the data sample set comprises real data and simulated data obtained by the real devices and simulated models for one or more environments, and wherein the data sample set is obtained over multiple frames. 13. The method of clause 11, wherein the neural network model is a convolutional neural network model, and / or the light detection and ranging device comprises a lidar. 14. A method for establishing a noise probability model, comprising: obtaining a set of data samples, wherein the set of data samples comprises a plurality of echo data for a first pixel in a depth map generated by a plurality of light detection and ranging devices; and determining parameters of a noise probability model specific to the first pixel based on the plurality of echo data for the first pixel. 15. The method of clause 14, wherein the plurality of light detection and ranging devices comprise real devices and simulated models, the set of data samples comprises real data and simulated data obtained by the real devices and the simulated models for one or more environments, and wherein the set of data samples is obtained during a plurality of frames. 16. The method of clause 14, wherein the noise probability model satisfies a Gaussian distribution, and the parameters comprise the mean and variance of the Gaussian distribution. 17. A light detection and ranging device comprising: at least one processor; and at least one memory including computer program code, wherein the at least one memory and the computer program code are configured to, through the at least one processor, cause the light detection and ranging device to perform the method according to any one of clauses 1 to 10. 18. An electronic device comprising: at least one processor; and at least one memory including computer program code, wherein the at least one memory and the computer program code are configured to, through the at least one processor, cause the electronic device to perform the method according to any one of clauses 11 to 13 or 14 to 16.19. A computer program product comprising one or more instructions which, when executed by a processor, cause the method according to any one of clauses 1-10, 11-13 or 14-16 to be implemented.

[0107] The exemplary embodiments of the present disclosure are described above with reference to the accompanying drawings, but the present disclosure is certainly not limited to the above examples. Those skilled in the art may obtain various changes and modifications within the scope of the appended claims, and it should be understood that these changes and modifications will naturally fall within the technical scope of the present disclosure.

[0108] For example, a plurality of functions included in one unit in the above embodiments may be implemented by separate devices. Alternatively, a plurality of functions implemented by a plurality of units in the above embodiments may be implemented by separate devices, respectively. In addition, one of the above functions may be implemented by a plurality of units. Needless to say, such a configuration is included in the technical scope of the present disclosure.

[0109] In this specification, the steps described in the flowchart include not only processing executed in time series in the order described, but also processing executed in parallel or individually rather than necessarily in time series. In addition, even in the steps processed in time series, it goes without saying that the order can be changed as appropriate.

[0110] Although the present disclosure and its advantages have been described in detail, it should be understood that various changes, substitutions and transformations can be made without departing from the spirit and scope of the present disclosure as defined by the appended claims. Moreover, the terms "comprises," "comprising," or any other variations thereof in the embodiments of the present disclosure are intended to cover non-exclusive inclusions, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article, or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

Claims

1. A method for processing multi-echo data, comprising: For each pixel point in the depth map of the environment generated by the light detection and ranging device, a plurality of echo data are obtained; determining a probability that corresponding echo data among the plurality of echo data corresponds to a valid echo or a noise echo, thereby forming a confidence map corresponding to the depth map of the environment; and A saliency map corresponding to the depth map of the environment is generated based on the confidence map, wherein the saliency map is used to indicate an amount of valid information in a corresponding portion of the depth map of the environment.

2. The method according to claim 1, wherein Determining a probability that the corresponding echo data corresponds to a valid echo or a noise echo includes using a neural network model to distinguish between valid echoes and noise echoes.

3. The method according to claim 2, wherein: Using a neural network model to distinguish valid echoes from noise echoes includes: using a unified neural network model to distinguish valid echoes from noise echoes for different pixel points in the depth map.

4. The method according to claim 1, wherein The neural network model is trained as follows: Obtaining a first set of data samples, wherein the first set of data samples includes a plurality of echo data of each pixel in a depth map generated by a plurality of light detection and ranging devices; For each pixel, a pixel-specific noise probability model is used to add a valid echo or noise echo label to the corresponding data sample; as well as The neural network model is trained using the data samples of the first set and the corresponding labels as training data.

5. The method according to claim 4, in, The plurality of light detection and ranging devices include real devices and simulation models, the data samples of the first set include real data and simulation data obtained by the real devices and the simulation models for one or more environments, and The first set of data samples is obtained during a first plurality of frames.

6. The method according to claim 4, wherein: The parameters of the pixel-specific noise probability model are determined as follows: Obtaining a second set of data samples, wherein the second set of data samples includes a plurality of echo data of a first pixel point in a depth map generated by a plurality of light detection and ranging devices; and Based on the plurality of echo data of the first pixel point, parameters of a noise probability model specific to the first pixel point are determined.

7. The method according to claim 6, in, The plurality of light detection and ranging devices include real devices and simulation models, the data samples of the second set include real data and simulation data obtained by the real devices and the simulation models for one or more environments, and The second set of data samples is obtained during a second plurality of frames.

8. The method according to claim 1, wherein Determining the probability that the corresponding echo data corresponds to a valid echo or a noise echo includes using a pixel-specific noise probability model to distinguish between valid echoes and noise echoes.

9. The method according to claim 1, further comprising: Based on the saliency map, preferentially processing echo data of a portion of the depth map of the environment having a high amount of effective information; and / or The saliency map is output in real time based on the user's region of interest.

10. The method according to claim 1, wherein The neural network model is a convolutional neural network model, and / or the light detection and ranging device includes a lidar.

11. A method for training a neural network model, comprising: Obtaining a data sample set, wherein the data sample set includes a plurality of echo data of each pixel point in a depth map generated by a plurality of light detection and ranging devices; For each pixel, a pixel-specific noise probability model is used to add a valid echo or noise echo label to the corresponding data sample; as well as The data sample set and corresponding labels are used as training data to train the neural network model.

12. The method according to claim 11, in, The plurality of light detection and ranging devices include real devices and simulation models, the data sample set includes real data and simulation data obtained by the real devices and the simulation model for one or more environments, and The data sample set is obtained during a plurality of frames.

13. The method according to claim 11, wherein The neural network model is a convolutional neural network model, and / or the light detection and ranging device includes a lidar.

14. A method for establishing a noise probability model, comprising: Obtaining a data sample set, wherein the data sample set includes a plurality of echo data of a first pixel point in a depth map generated by a plurality of light detection and ranging devices; as well as Based on the plurality of echo data of the first pixel point, parameters of a noise probability model specific to the first pixel point are determined.

15. The method according to claim 14, in, The plurality of light detection and ranging devices include real devices and simulation models, the data sample set includes real data and simulation data obtained by the real devices and the simulation model for one or more environments, and The data sample set is obtained during a plurality of frames.

16. The method according to claim 14, wherein The noise probability model satisfies a Gaussian distribution, and the parameters include a mean and a variance of the Gaussian distribution.

17. A light detection and ranging device comprising: at least one processor; and At least one memory comprises computer program code, wherein the at least one memory and the computer program code are configured to, through the at least one processor, enable the light detection and ranging device to perform the method according to any one of claims 1 to 10.

18. An electronic device comprising: at least one processor; and At least one memory comprises computer program code, wherein the at least one memory and the computer program code are configured to, through the at least one processor, enable the electronic device to perform the method according to any one of claims 11-13 or 14-16.

19. A computer program product comprising one or more instructions which, when executed by a processor, cause the method according to any one of claims 1-10, 11-13 or 14-16 to be implemented.

Images