Point cloud solving method, electronic device and storage medium

Abstract

The application discloses a point cloud solving method, an electronic device and a storage medium. The point cloud solving method comprises the following steps: acquiring a first echo sequence signal captured by a laser radar; acquiring visual information of a heterogenous sensor; and determining a second echo sequence signal used for point cloud generation by combining the visual information, the first echo sequence signal and echo information of each echo in the first echo sequence signal.

Description

Technical Field

[0001] This application relates to the field of data acquisition technology, and in particular to a point cloud solution method, electronic device, and storage medium. Background Technology

[0002] In related technologies, the point cloud generation process in a LiDAR system works as follows: an optical transmitter emits a beam of light with a certain scattering angle. This beam strikes the surface of an object in the environment, generating reflected light. An optical receiver captures the time-intensity waveform of the reflected light and sends it to the point cloud generation unit for echo-point cloud processing. In the point cloud generation step, the point cloud processing unit analyzes a series of echo peak-valley values, peak-valley times, half-peak values, and half-peak times in its computing devices (including but not limited to FPGAs / ASICs / DSPs) based on the echoes from the optical receiver provided by the underlying signal processing unit. It calculates the distance to one or more echoes, converts them into point clouds, and outputs them to the downstream point cloud processing algorithm.

[0003] The inventors discovered that the limitation of the related technology is that it can only use the echo information from one or more optical devices at the same or different times to infer the situation of the object reflecting the light path, without any other information input.

[0004] Due to the aforementioned limitations in input information, the point cloud processing unit cannot accurately infer which echoes are optical path interference (including but not limited to rain, fog, vehicle exhaust, and dust) and which echoes are actual objects based solely on echo time, echo intensity, and echo slope. This is especially true when the echo intensity of optical path interference exceeds the reflection intensity of objects in the actual environment; in such cases, all information from the echo sequence must generally be discarded. Because of these input limitations, the point cloud processing unit cannot accurately infer the actual situation of objects in the environment corresponding to a series of echoes, thus failing to generate accurate point clouds. Summary of the Invention

[0005] This invention provides a point cloud calculation method, an electronic device, and a storage medium to at least solve one of the above-mentioned technical problems.

[0006] In a first aspect, embodiments of the present invention provide a point cloud processing method, comprising: acquiring a first echo sequence signal captured by a lidar; acquiring visual information from a heterogeneous sensor; and using the visual information in combination with the first echo sequence signal and the echo information of each echo in the first echo sequence signal to determine a second echo sequence signal for point cloud generation.

[0007] Secondly, embodiments of the present invention provide an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the point cloud solving methods described above.

[0008] Thirdly, embodiments of the present invention provide a storage medium storing one or more programs including execution instructions, the execution instructions being readable and executed by electronic devices (including but not limited to computers, servers, or network devices, etc.) to perform any of the point cloud solving methods described above.

[0009] Fourthly, embodiments of the present invention also provide a computer program product, the computer program product including a computer program stored on a storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, causing the computer to execute any of the above-described point cloud solving methods.

[0010] Fifthly, embodiments of the present invention also provide a portable platform, including the electronic device as described in the second aspect.

[0011] The method of this application utilizes visual information from a heterogeneous sensor, combined with one or more pieces of information such as echo time, echo intensity, and echo slope, to determine the nature of the echo and decide which one or more echoes from a series of echoes to use for point cloud generation, thereby providing a higher density and higher precision point cloud output. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart of a point cloud solving method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] Please refer to Figure 1 The diagram illustrates a flowchart of a point cloud calculation method according to an embodiment of the present invention. This point cloud calculation method may exist in the form of an algorithm within a computer; in subsequent embodiments, it may be referred to as a point cloud calculation device, such as a piece of program code or software, etc., and this application is not limited thereto. This point cloud calculation method can be used in a lidar system or a sensor containing lidar, or in a computing device connected to a sensor, and this application is not limited thereto.

[0016] like Figure 1 As shown, in step 101, the first echo sequence signal captured by the lidar is acquired; In step 102, visual information from the heterogeneous sensor is acquired; In step 103, the visual information is used in combination with the first echo sequence signal and the echo information of each echo in the first echo sequence signal to determine the second echo sequence signal for point cloud generation.

[0017] In this embodiment, for step 101, the point cloud processing device acquires the first echo sequence signal captured by the lidar. For example, the first echo sequence signal can be an echo sequence signal provided by the lidar optical receiving device, including but not limited to the peak-valley value, peak-valley time, half-peak value, and half-peak time of the echo. Each measurement produces multiple echoes, referred to as an echo sequence. Each individual echo in the echo sequence carries echo time information, intensity information, slope information, etc., collectively referred to as echo information.

[0018] Then, in step 102, the point cloud solving device acquires visual information from a different sensor within the same or adjacent time window as the first echo sequence signal. This visual information includes semantics, texture, and depth, and the different sensor can be a visual sensor, millimeter-wave radar, infrared sensor, or any other sensor capable of acquiring visual information.

[0019] Finally, for step 103, the point cloud solving device uses visual information combined with the first echo sequence signal and the echo information of each echo in the first echo sequence signal to remove interference echoes in the first echo sequence signal and determine the second echo sequence signal used for point cloud generation.

[0020] The method in this embodiment determines the second echo sequence signal for point cloud generation by using visual information combined with the first echo sequence signal and the echo information of each echo in the first echo sequence signal, thereby achieving higher precision point cloud output.

[0021] In some optional embodiments, the visual information includes semantic information and / or depth information. The visual information, combined with the first echo sequence signal and its echo information, is used to determine a second echo sequence signal for point cloud generation. This includes: the point cloud processing device selecting one or more echoes from the first echo sequence signal, wherein the first echo sequence signal includes multiple echoes; using the semantic information and / or depth information of a heterogeneous sensor within the same time period as the selected current echo, determining whether the current echo is an optical path interference echo; if the current echo is an optical path interference echo, using the semantic information and / or depth information combined with the echo information of the current echo to remove the optical path interference echo; and determining the signal after removing all optical path interference echoes from the first echo sequence signal as the second echo sequence signal for point cloud generation.

[0022] In a specific example, the point cloud processing device selects one or more echoes based on the first echo sequence signal to represent potential distance measurements along the optical path. A characteristic of lidar sensors is that each potential ranging result along the optical path is reflected as an echo in the data; if there are multiple objects along the optical path, there will be multiple echoes. Then, based on semantic information provided by heterogeneous sensors within the same time window, such as a monocular sensor, it determines whether rain, fog, dust, or vehicle exhaust fumes are present. If optical path interference exists, it combines the echo information to eliminate interfering echoes from the first echo sequence signal, selecting the echo of the actual measured object in the environment. If a binocular sensor is used, it combines the semantic and depth information provided by heterogeneous sensors within the same time window with the echo information to eliminate interfering echoes from the first echo sequence signal, selecting the echo of the actual measured object in the environment, thereby generating a more accurate point cloud. While lidar is detecting targets, in addition to the target's echo, some electromagnetic waves of the same frequency from non-target objects are sometimes scattered into the radar receiving system, forming clutter, known as radial interference echoes. Radial interference echoes are a significant factor contaminating radar echo data, severely impacting the quantitative application of radar data. Therefore, it is necessary to filter out optical path interference echoes. In the presence of optical path interference, the detection capability of lidar against interference echoes can be improved through one or more techniques, such as multi-echo detection, wavelength optimization, signal processing, multi-sensor fusion, dynamic scanning mode adjustment, environmental modeling, hardware improvements, and machine learning. These methods can be used individually or in combination, depending on the application scenario and requirements.

[0023] In some optional embodiments, the visual information includes semantic information and texture information. The visual information, combined with the first echo sequence signal and its echo information, is used to determine a second echo sequence signal for point cloud generation. This includes: selecting one or more echoes from the first echo sequence signal; using the semantic and texture information of a heterogeneous sensor within the same time period as the selected current echo, determining whether only a portion of the potential objects in the optical path of the current echo are within the light emission angle; specifically, if multiple semantic targets are encompassed within the spot size of a certain beam, it can be considered that only a portion of the objects are within the light emission angle; if only a portion of the potential objects in the optical path of the current echo are within the light emission angle, using the semantic information and texture information combined with the echo information of the current echo, multiple actual measured distances are provided in the current echo; and the multiple actual measured distances of the first echo sequence signal and at least one echo are determined as the second echo sequence signal for point cloud generation.

[0024] In a specific example, the point cloud processing device selects one or more echoes from a first echo sequence signal, which includes multiple echoes. Then, using semantic and texture information from heterogeneous sensors within the same time period as the selected current echo, it determines whether only a portion of the potential object in the optical path of the current echo is within the light emission angle. The laser radar spot has a size; for example, it might be a circle, where the left half hits one target and the right half hits another, thus providing distance measurements for two targets within a single spot. Therefore, cases where only a portion is within the light emission angle occur. If only a portion of the potential object in the optical path of the current echo is within the light emission angle, the semantic and texture information, combined with the echo information of the current echo, provides multiple actual distance measurements within the current echo. For example, if the spot hits multiple objects, multiple echoes are provided. Following the validity of these multiple echoes, target points can be generated for multiple echoes with different time sequences (echo time - emission time = photon flight time = distance x 2), thus providing multiple actual distance measurements. Finally, the first echo sequence signal and multiple actual measured distances of at least one echo are determined as the second echo sequence signal for point cloud generation, thereby enabling more sophisticated point cloud generation.

[0025] In some optional embodiments, the visual information includes semantic information and texture information. The visual information, combined with a first echo sequence signal and its echo information, is used to determine a second echo sequence signal for point cloud generation. This includes: selecting one or more echoes from the first echo sequence signal, wherein the first echo sequence signal includes multiple echoes; using semantic and texture information from a heterogeneous sensor within the same time period as the selected current echo, determining whether only a portion of the potential object in the optical path of the current echo is at the light emission angle; if only a portion of the potential object in the optical path of the current echo is at the light emission angle, using the semantic and texture information combined with the echo information of the current echo to provide a single actual measured distance in the current echo, and offsetting the emission angle of the optical path of the current echo accordingly based on the semantic and texture information; and determining the first echo sequence signal, the single actual measured distance of at least one echo, and the offset corresponding to the emission angle of the at least one echo as the second echo sequence signal for point cloud generation. The determination of whether to provide single or multiple actual measured distances depends primarily on the objects within the light spot area. If the semantic information within the light spot area indicates an actual object, including people, vehicles, signs, etc., then a single measured distance is provided. If the semantic information indicates multiple objects, such as a person in the foreground and a building in the background, then multiple measured distances can be provided. Regarding the specific offsetting method, the following example is provided for reference by those skilled in the art: When the light spot is relatively large, for example, if the semantic information indicates that the left half of the light spot represents a pedestrian and the right half a building, the light spot can be divided into two parts. The angle of the pedestrian is adjusted to the left half, and the angle of the building is adjusted to the right half. For example, if the light spot has a resolution of 0.2 degrees, it can be split into left and right 0.1-degree segments and assigned to different objects. Thus, the original resolution of only 0.2 degrees can be reduced to 0.1 degrees after subdivision.

[0026] In a specific example, the point cloud processing device selects one or more echoes from a first echo sequence signal, wherein the first echo sequence signal includes multiple echoes. Then, using semantic and texture information from heterogeneous sensors within the same time period as the selected current echo, such as visual sensors, millimeter-wave radar, and infrared sensors, it determines whether only a portion of the potential objects in the optical path of the current echo are within the light emission angle, i.e., providing distance measurements for multiple targets within a single light spot. If only a portion of the potential objects in the optical path of the current echo are within the light emission angle, a single actual measured distance is provided in the current echo using semantic and texture information combined with the echo information of the current echo. The emission angle of the optical path of the current echo is then offset according to the semantic and texture information, allowing selection of a single echo for point cloud generation, thus obtaining a single distance measurement. Finally, the first echo sequence signal, the single actual measured distance of at least one echo, and the offset corresponding to the emission angle of at least one echo are determined as the second echo sequence signal for point cloud generation, thereby performing more accurate point cloud generation.

[0027] In some optional embodiments, the visual information includes semantic information and texture information. The visual information, combined with the first echo sequence signal and its echo information, is used to determine a second echo sequence signal for point cloud generation. This includes: selecting one or more echoes from the first echo sequence signal, wherein the first echo sequence signal includes multiple echoes; then, using the semantic and texture information of a heterogeneous sensor within the same time period as the selected current echo, determining whether a potential object in the optical path of the current echo contains a low-reflectivity material; if the potential object in the optical path of the current echo contains a low-reflectivity material, using the semantic and texture information combined with the echo information of the current echo, providing multiple distances corresponding to low-reflectivity and normal reflectivity materials respectively in the echo information of the current echo; finally, determining the multiple distances corresponding to low-reflectivity and normal reflectivity materials between the first echo sequence signal and at least one echo as the second echo sequence signal for point cloud generation, thereby performing higher-precision point cloud generation. Here, a low-reflectivity material refers to a material whose surface has a weak ability to reflect light. Low-reflectivity materials, through surface roughness, high light absorption, or special coatings to reduce light reflection, are widely used in optics, architecture, automotive, and electronics. When using lidar to detect low-reflectivity materials, detection performance can be improved through techniques such as increasing laser power, optimizing receiver sensitivity, using multi-echo detection, selecting appropriate wavelengths, optimizing data processing algorithms, shortening detection distances, multi-beam scanning, and ambient light compensation. These methods can be used individually or in combination, depending on the application scenario and requirements. In this embodiment, multiple distances are specified for low-reflectivity materials because low-reflectivity materials such as glass affect the physical properties of the echoes; without visual input, other echoes must be discarded; with visual input, other echoes penetrating the glass can be recalled. The material is only used to determine whether it is a special low-reflectivity material like glass, and does not change the distance. This allows for the retention of more echo signals, and abundant echo signals help improve subsequent detection accuracy.

[0028] In some optional embodiments, the visual information is an image acquired by the heterogeneous sensor. Visual features can be extracted using a deep learning network. The visual information, combined with a first echo sequence signal and its echo information, is used to determine a second echo sequence signal for point cloud generation. This includes: selecting one or more echoes from the first echo sequence signal, wherein the first echo sequence signal includes multiple echoes; then, based on visual features within the same time period as the selected current echo, inputting the echo information of the current echo and the visual features into the deep learning network to obtain the echo selection output by the deep learning network; finally, using all echo selections for the first echo sequence signal as the second echo sequence signal for point cloud generation, thereby generating a richer point cloud. The deep learning network's function is to simultaneously input raw LiDAR information and visual information, and to provide echo selection output. Visual features are a collective term for the intermediate results after the image is fed into the deep network for convolution and other operations.

[0029] In some optional embodiments, the visual information, combined with the first echo sequence signal and its echo information, is used to determine the second echo sequence signal for point cloud generation. This includes: extracting low-level features of the LiDAR sensor using a deep learning network, wherein visual information is used as an aid during the training of the deep learning network to extract low-level features; selecting one or more echoes from the first echo sequence signal, wherein the first echo sequence signal includes multiple echoes; then, inputting the low-level features within the same time period as the selected current echo into the deep learning network to obtain the echo selection output by the deep learning network; finally, using all echo selections for the first echo sequence signal as the second echo sequence signal for point cloud generation. Low-level features refer to the collective intermediate results after the raw output of the LiDAR sensor is fed into the deep network for convolution and other operations.

[0030] In a specific example, the inputs to the point cloud processing unit include: 1. Echo sequence signals provided by the optical receiving device, including but not limited to the peak-valley values, peak-valley times, half-peak values, and half-peak times of the echoes. 2. One or more pieces of information provided by the visual sensor within the same or adjacent time window (typically within 0ms-100ms), including but not limited to semantics, texture, and depth.

[0031] The point cloud processing unit fully utilizes the information provided by the visual sensor to determine the nature of the echo and generate the corresponding point cloud. Specific point cloud generation methods include, but are not limited to, the following rule-based methods. These rules can be executed in parallel to filter out erroneous results and obtain richer information; this application does not impose any restrictions on them. 1. Based on the echo sequence, select one or more echoes to represent potential distance measurements along the optical path; based on the semantic information provided by the visual sensor within the same time window, determine whether there is rain, fog, dust, or exhaust fumes from the vehicle ahead. If there is optical path interference, combine the echo information to remove interfering echoes from the echo sequence and select the echo of the actual object being measured in the environment; if there is a binocular sensor, based on the depth information provided by the visual sensor within the same time window, add the echo information to remove interfering echoes from the echo sequence and select the echo of the actual object being measured in the environment for more accurate point cloud generation.

[0032] 2. Based on the echo sequence, select one or more echoes to represent potential distance measurements along the optical path. Using semantic and texture information provided by the visual sensor within the same time window, determine whether only a portion of the potential object along the optical path is within the light emission angle. If so, combine the echo information to provide multiple actual distance measurements in the echo sequence for richer point cloud generation. Specifically, if the semantic information within the light spot area indicates an actual object, including people, vehicles, signs, etc., a single measurement is provided; if the semantic information contains multiple objects, such as a person in the foreground and a building in the background, multiple echoes can be provided.

[0033] 3. Based on the echo sequence, select one or more echoes to represent potential distance measurements along the optical path. Using semantic and texture information provided by the visual sensor within the same time window, determine if only a portion of the potential object along the optical path is within the light emission angle. If so, combine the echo information to provide a single actual distance measurement within the echo sequence, and based on the information provided by the visual sensor, adjust the emission angle of the optical path accordingly for more accurate point cloud generation. The single measured distance is typically the distance corresponding to the nearest echo.

[0034] 4. Based on the echo sequence, select one or more echoes to represent potential distance measurements along the optical path; based on the semantic and texture information provided by the visual sensor within the same time window, determine whether the potential object along the optical path is made of low-reflectivity materials such as glass. If so, combine the echo information to provide multiple distances for multiple echoes corresponding to low-reflectivity and normal reflection in the echo sequence, and generate richer point clouds.

[0035] Furthermore, specific point cloud generation methods also include, but are not limited to, the following deep learning-based methods: 1. Based on the echo sequence, select one or more echoes to represent the potential distance measurement on the optical path; based on the visual features extracted by the visual sensor through the deep learning network within the same time window, feed the potential echo distance and low-level features into the deep learning network, and output echo selection information to generate a more accurate and richer point cloud.

[0036] 2. Based on the echo sequence, select one or more echoes to represent the potential distance measurement on the optical path; based on the low-level features extracted by the lidar sensor through a deep learning network within the same time window, output echo selection information to generate a more accurate and richer point cloud.

[0037] Because existing lidar technologies only select the nearest echo by default, the solution in this application, by incorporating semantic information, may provide the nearest echo or multiple echoes including the nearest echo, resulting in a richer and more accurate point cloud.

[0038] In other embodiments, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that can execute the point cloud solving method in any of the above method embodiments; In one embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured as follows: Acquire the first echo sequence signal captured by the lidar; Acquire visual information from heterogeneous sensors; By combining the visual information with the first echo sequence signal and the echo information of each echo in the first echo sequence signal, a second echo sequence signal for point cloud generation is determined.

[0039] Non-volatile computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the point cloud computing device, etc. Furthermore, the non-volatile computer-readable storage medium may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the non-volatile computer-readable storage medium may optionally include memory remotely configured relative to the processor, and these remote memories may be connected to the point cloud computing device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0040] This invention also provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions, which, when executed by a computer, cause the computer to perform any of the point cloud solving methods described above.

[0041] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present invention, such as... Figure 2As shown, the device includes one or more processors 210 and a memory 220. Figure 2 Taking a processor 210 as an example, the device for point cloud calculation may also include an input device 230 and an output device 240. The processor 210, memory 220, input device 230, and output device 240 can be connected via a bus or other means. Figure 2 Taking a bus connection as an example, the memory 220 is the aforementioned non-volatile computer-readable storage medium. The processor 210 executes various server functions and data processing by running non-volatile software programs, instructions, and modules stored in the memory 220, thereby implementing the point cloud calculation method described in the above embodiment. The input device 230 can receive input digital or character information and generate key signal inputs related to user settings and function control of the point cloud calculation device. The output device 240 may include a display screen or other display device.

[0042] This application also provides a mobile platform, which includes a vehicle body, a power system, and electronic devices as described in the above embodiments. The power system is installed on the vehicle body to provide power; the principle and implementation of the electronic devices are consistent with those described in the above embodiments and will not be repeated here. The electronic devices may be sensors, controllers, or other computing devices installed on the mobile platform. Optionally, the mobile platform may include at least one of the following: a vehicle, a mobile robot, or an unmanned vehicle.

[0043] The above-described product can execute the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.

[0044] In one embodiment, the above-described electronic device is applied in a point cloud computing apparatus, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: Acquire the first echo sequence signal captured by the lidar; Acquire visual information from heterogeneous sensors; By combining the visual information with the first echo sequence signal and the echo information of each echo in the first echo sequence signal, a second echo sequence signal for point cloud generation is determined.

[0045] The electronic devices described in this application exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include: smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones, etc.

[0046] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0047] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes: audio and video players (e.g., iPods), handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0048] (4) Server: A device that provides computing services. The components of a server include a processor, hard disk, memory, system bus, etc. Servers are similar to general computer architectures, but because they need to provide highly reliable services, they have higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0049] (5) Other electronic devices with data interaction functions, such as cameras, lidar and other sensors.

[0050] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0051] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A point cloud solution method, comprising: Acquire the first echo sequence signal captured by the lidar; Acquire visual information from heterogeneous sensors; By combining the visual information with the first echo sequence signal and the echo information of each echo in the first echo sequence signal, a second echo sequence signal for point cloud generation is determined.

2. The method according to claim 1, characterized in that, The visual information includes semantic information and / or depth information. Using the visual information in combination with the first echo sequence signal and its echo information, a second echo sequence signal for point cloud generation is determined, including: Select one or more echoes from the first echo sequence signal, wherein the first echo sequence signal includes multiple echoes; Using semantic and / or depth information from heterogeneous sensors within the same time period as the selected current echo, determine whether the current echo is an optical path interference echo; If the current echo is an optical path interference echo, the optical path interference echo is removed by combining the semantic information and / or the depth information with the echo information of the current echo. The signal obtained by removing all optical path interference echoes from the first echo sequence signal is determined as the second echo sequence signal used for point cloud generation.

3. The method according to claim 1, characterized in that, The visual information includes semantic information and texture information. Using this visual information in conjunction with the first echo sequence signal and its echo information, a second echo sequence signal for point cloud generation is determined, including: Select one or more echoes from the first echo sequence signal, wherein the first echo sequence signal includes multiple echoes; Using semantic and texture information from heterogeneous sensors within the same time period as the selected current echo, it is determined whether only a portion of the potential objects in the optical path of the current echo are at the light emission angle. If only part of the potential object in the optical path of the current echo is at the light emission angle, multiple actual measurement distances are provided in the current echo by combining the semantic information and the texture information with the echo information of the current echo. The first echo sequence signal and multiple actual measured distances of at least one echo are used to determine the second echo sequence signal for point cloud generation.

4. The method according to claim 1, characterized in that, The visual information includes semantic information and texture information. Using this visual information in conjunction with the first echo sequence signal and its echo information, a second echo sequence signal for point cloud generation is determined, including: Select one or more echoes from the first echo sequence signal, wherein the first echo sequence signal includes multiple echoes; Using semantic and texture information from heterogeneous sensors within the same time period as the selected current echo, it is determined whether only a portion of the potential objects in the optical path of the current echo are at the light emission angle. If only part of the potential object in the optical path of the current echo is at the light emission angle, the semantic information and the texture information are combined with the echo information of the current echo to provide a single actual measured distance in the current echo, and the emission angle of the optical path of the current echo is offset accordingly based on the semantic information and the texture information. The first echo sequence signal, the individual actual measured distance of at least one echo, and the offset corresponding to the departure angle of the at least one echo are determined as the second echo sequence signal for point cloud generation.

5. The method according to claim 1, characterized in that, The visual information includes semantic information and texture information. Using this visual information in conjunction with the first echo sequence signal and its echo information, a second echo sequence signal for point cloud generation is determined, including: Select one or more echoes from the first echo sequence signal, wherein the first echo sequence signal includes multiple echoes; Using semantic and texture information from heterogeneous sensors within the same time period as the selected current echo, it is determined whether potential objects in the optical path of the current echo contain low-reflectivity materials. If the potential object in the optical path of the current echo contains a low-reflectivity material, the semantic information and the texture information are combined with the echo information of the current echo to provide multiple distances corresponding to the low-reflectivity material and the normal reflectivity material, respectively. The first echo sequence signal and at least one echo at multiple distances corresponding to low-reflectivity and normal-reflectivity materials are determined as the second echo sequence signal for point cloud generation.

6. The method according to claim 1, characterized in that, The visual information is the visual features extracted from the images acquired by the heterogeneous sensor through a deep learning network. Using this visual information, combined with the first echo sequence signal and its echo information, a second echo sequence signal is determined for point cloud generation, including: Select one or more echoes from the first echo sequence signal, wherein the first echo sequence signal includes multiple echoes; Based on the visual features within the same time period as the selected current echo, the echo information and low-level features of the current echo are input into the deep learning network to obtain the echo selection output by the deep learning network. The low-level features are obtained by sending the original output of the lidar into the deep learning network. All echoes from the first echo sequence signal are selected as the second echo sequence signal for point cloud generation.

7. The method according to claim 1, characterized in that, Using the visual information combined with the first echo sequence signal and the echo information of the first echo sequence signal, a second echo sequence signal for point cloud generation is determined, including: The low-level features of the LiDAR are extracted using a deep learning network, wherein visual information is used as an aid during the training of the deep learning network by extracting the low-level features. Select one or more echoes from the first echo sequence signal, wherein the first echo sequence signal includes multiple echoes; The low-level features within the same time period as the selected current echo are input into the deep learning network to obtain the echo selection output by the deep learning network; All echoes from the first echo sequence signal are selected as the second echo sequence signal for point cloud generation.

8. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 7.

9. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 7.

11. A portable platform comprising the electronic device as claimed in claim 8.

Images