Point cloud denoising and segmentation method based on visual feature fusion and related equipment

Through the integration and mapping of visual features and lidar features, the problem of low accuracy of point cloud denoising method in complex scenarios is solved, and more efficient environmental perception is achieved.

CN118941907APending Publication Date: 2024-11-12SHENZHEN DEEPROUTE AI CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410992256.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In the prior art, point cloud denoising methods rely on point cloud data themselves and are difficult to deal with complex scenarios. The environmental perception information obtained through a single data source is relatively accurate.

Method used

Using a method based on visual feature fusion, the three primary color images and lidar point cloud data are obtained, and the features of the three primary color images and lidar point cloud data are converted to bird's eye view space for fusion, mapped back to the point cloud space, and semantic segmentation is performed to identify noise.

Benefits of technology

Through the fusion of multi-sensor and multi-modal methods, the noise in the point cloud can be effectively identified and the accuracy of environmental perception is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941907A_ABST
    Figure CN118941907A_ABST
Patent Text Reader

Abstract

The invention discloses a visual feature fusion-based point cloud denoising and segmentation method and related equipment. The method comprises the following steps: acquiring image features extracted from three-primary color images of a plurality of visual angles, and acquiring laser radar features extracted from laser radar point cloud data of the same scene; converting the image features and the laser radar features to a bird's-eye view space for fusion to obtain fused bird's-eye view features; mapping the fused bird's-eye view features back to a point cloud space to obtain bird's-eye view laser radar features; fusing the bird's-eye view laser radar features and the laser radar features to obtain fused laser radar features; and performing semantic segmentation according to the fused laser radar features to obtain point cloud semantic information. According to the invention, the image data provided by the visual sensor and the laser radar point cloud data are subjected to feature level fusion processing, the bird's-eye view features and the laser radar features are subjected to feature interaction, a multi-sensor and multi-mode fusion mode is adopted, noise in the point cloud is effectively identified, and the accuracy of environmental perception is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of point cloud deep learning technology, and in particular to a point cloud denoising and segmentation method based on visual feature fusion and related equipment. Background Art

[0002] In the fields of modern autonomous driving, robot navigation, and 3D reconstruction, LiDAR technology has become an important tool for obtaining environmental information due to its high precision and high resolution. LiDAR generates point cloud data by emitting a laser beam and measuring its reflection time, thereby constructing a 3D spatial model of the current scene. However, LiDAR point cloud data is affected by various noises during the acquisition process, such as multipath reflection, ambient light interference, and sensor errors. In the natural environment, LiDAR point cloud data is also easily affected by meteorological conditions such as rain, snow, fog, and sandstorms, which causes the laser beam to be scattered and absorbed during propagation, thereby introducing additional noise.

[0003] Traditional point cloud denoising methods mainly include statistical filtering, radius filtering and conditional filtering, etc. Although they can remove noise to a certain extent, they have limitations when dealing with complex scenes. Moreover, these methods are highly dependent on the information of the point cloud data itself, and the accuracy of environmental perception information obtained through a single data source is low.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0005] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a point cloud denoising and segmentation method and related equipment based on visual feature fusion are provided, aiming to solve the problem that the point cloud denoising method in the prior art relies on the point cloud data itself, is difficult to process complex scenes, and the accuracy of the environmental perception information obtained through a single data source is low.

[0006] The technical solution adopted by the present invention to solve the problem is as follows:

[0007] In a first aspect, an embodiment of the present invention provides a point cloud denoising and segmentation method based on visual feature fusion, the method comprising:

[0008] Obtain image features extracted from three-primary color images of several viewing angles, and obtain lidar features extracted from lidar point cloud data of the same scene;

[0009] Converting the image features and the laser radar features into a bird's-eye view space for fusion to obtain a fused bird's-eye view feature;

[0010] Mapping the fused bird's-eye view features back to the point cloud space to obtain bird's-eye view lidar features;

[0011] Fusing the bird's-eye view laser radar feature with the laser radar feature to obtain a fused laser radar feature;

[0012] Semantic segmentation is performed according to the fused laser radar features to obtain point cloud semantic information.

[0013] In one embodiment, the step of obtaining image features extracted from three-primary color images of a plurality of viewing angles and obtaining lidar features extracted from lidar point cloud data of the same scene includes:

[0014] Acquire the three-primary-color images at several viewing angles through a visual sensor;

[0015] Acquiring the laser radar point cloud data by using a laser radar sensor arranged in the same scene as the visual sensor;

[0016] The three-primary-color image and the laser radar point cloud data are respectively input into their corresponding encoders to obtain the image features and the laser radar features.

[0017] In one embodiment, converting the image feature and the laser radar feature into a bird's-eye view space for fusion to obtain a fused bird's-eye view feature includes:

[0018] Convert the image features and the laser radar features to the bird's-eye view space respectively to obtain the image features in the bird's-eye view format and the laser radar features in the bird's-eye view format;

[0019] splicing the image features in the bird's-eye view format and the laser radar features in the bird's-eye view format to obtain spliced ​​data;

[0020] The spliced ​​data is input into a bird's-eye view encoder to obtain the fused bird's-eye view feature.

[0021] In one implementation, mapping the fused bird's-eye view features back to the point cloud space to obtain the bird's-eye view laser radar features includes:

[0022] Generate a coordinate index of each point cloud in a bird's-eye view space according to the laser radar feature;

[0023] Constructing the mapping data according to the coordinate index of each point cloud;

[0024] The fused bird's-eye view feature is mapped back to the point cloud space according to the mapping data to obtain the bird's-eye view lidar feature.

[0025] In one embodiment, generating a coordinate index of each point cloud in a bird's-eye view space according to the laser radar feature includes:

[0026] Z-axis compression and rasterization are performed according to the laser radar features to obtain the coordinate index of each point cloud in the bird's-eye view space.

[0027] In one implementation, mapping the fused bird's-eye view feature back to the point cloud space according to the mapping data to obtain the bird's-eye view laser radar feature includes:

[0028] According to the inverse mapping data corresponding to the mapping data, the fused bird's-eye view feature is projected onto the corresponding point cloud to obtain the bird's-eye view lidar feature.

[0029] In one implementation, performing semantic segmentation according to the fused laser radar features to obtain point cloud semantic information includes:

[0030] Get a pre-trained multilayer perceptron;

[0031] The fused laser radar features are input into the multi-layer perceptron for semantic segmentation to obtain the point cloud semantic information, wherein point clouds with different semantics in the point cloud semantic information are labeled in different ways.

[0032] In a second aspect, an embodiment of the present invention further provides a point cloud denoising and segmentation device based on visual feature fusion, the device comprising:

[0033] A feature acquisition module, used to acquire image features extracted from three-primary color images of several viewing angles, and to acquire lidar features extracted from lidar point cloud data of the same scene;

[0034] A first fusion module is used to convert the image features and the laser radar features into a bird's-eye view space for fusion to obtain a fused bird's-eye view feature;

[0035] A feature mapping module, used for mapping the fused bird's-eye view feature back to the point cloud space to obtain a bird's-eye view lidar feature;

[0036] A second fusion module is used to fuse the bird's-eye view laser radar feature with the laser radar feature to obtain a fused laser radar feature;

[0037] The semantic segmentation module is used to perform semantic segmentation according to the fused laser radar features to obtain point cloud semantic information.

[0038] In a third aspect, an embodiment of the present invention further provides a terminal, comprising a memory and one or more processors; the memory stores one or more programs; the program comprises instructions for executing any of the above-described point cloud denoising and segmentation methods based on visual feature fusion; and the processor is used to execute the program.

[0039] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a plurality of instructions are stored, wherein the instructions are suitable for being loaded and executed by a processor to implement the steps of the point cloud denoising and segmentation method based on visual feature fusion as described in any one of the above 7.

[0040] Beneficial effects of the present invention: The embodiment of the present invention obtains image features extracted from three-primary color images of several perspectives, and obtains lidar features extracted from lidar point cloud data of the same scene; converts the image features and the lidar features to a bird's-eye view space for fusion to obtain fused bird's-eye view features; maps the fused bird's-eye view features back to the point cloud space to obtain bird's-eye view lidar features; fuses the bird's-eye view lidar features with the lidar features to obtain fused lidar features; performs semantic segmentation based on the fused lidar features to obtain point cloud semantic information. The present invention performs feature-level fusion processing on the image data provided by the visual sensor and the lidar point cloud data, and performs feature interaction between the bird's-eye view features and the lidar features, and adopts a multi-sensor, multi-modal fusion method to effectively identify noise in the point cloud, thereby improving the accuracy of environmental perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 It is a flowchart of a point cloud denoising and segmentation method based on visual feature fusion provided by an embodiment of the present invention.

[0043] Figure 2 It is a schematic diagram of a framework for fusing laser radar features with bird's-eye view features provided in an embodiment of the present invention.

[0044] Figure 3 It is a schematic diagram of the framework of semantic segmentation provided by an embodiment of the present invention.

[0045] Figure 4 It is a module schematic diagram of a point cloud denoising and segmentation device based on visual feature fusion provided in an embodiment of the present invention.

[0046] Figure 5 It is a principle block diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention discloses a point cloud denoising and segmentation method based on visual feature fusion and related equipment. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0048] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0049] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.

[0050] In view of the above-mentioned defects of the prior art, the present invention provides a point cloud denoising and segmentation method based on visual feature fusion, the method obtains image features extracted from three-primary color images of several perspectives, and obtains laser radar features extracted from laser radar point cloud data of the same scene; converts the image features and the laser radar features to the bird's-eye view space for fusion to obtain fused bird's-eye view features; maps the fused bird's-eye view features back to the point cloud space to obtain bird's-eye view laser radar features; fuses the bird's-eye view laser radar features and the laser radar features to obtain fused laser radar features; performs semantic segmentation based on the fused laser radar features to obtain point cloud semantic information. The present invention performs feature-level fusion processing on the image data provided by the visual sensor and the laser radar point cloud data, and performs feature interaction between the bird's-eye view features and the laser radar features, adopts a multi-sensor, multi-modal fusion method to effectively identify the noise in the point cloud, thereby improving the accuracy of environmental perception.

[0051] like Figure 1 As shown, the method includes:

[0052] Step S100: Obtain image features extracted from three-primary color images of several viewing angles, and obtain lidar features extracted from lidar point cloud data of the same scene.

[0053] Specifically, this embodiment first needs to obtain three-primary color images (RGB images) of one or more perspectives. The three-primary color images can reflect the two-dimensional visual information of the current scene. At the same time, it is necessary to obtain the laser radar (LiDAR) point cloud data of the current scene. Each point of the laser radar point cloud data contains three-dimensional coordinate information. Then, feature extraction is performed on the three-primary color image and the laser radar point cloud data respectively to obtain two feature data used to reflect the current scene information, namely image features and laser radar features (LiDAR Features), which provide multi-sensor analysis data for subsequent point cloud denoising and environmental perception.

[0054] In one implementation, the step of obtaining image features extracted from three-primary-color images of a plurality of viewing angles and obtaining lidar features extracted from lidar point cloud data of the same scene includes:

[0055] Acquire three-primary-color images of several viewing angles through a visual sensor;

[0056] Obtaining lidar point cloud data through a lidar sensor deployed in the same scene as the visual sensor;

[0057] The three primary color images and lidar point cloud data are respectively input into their corresponding encoders to obtain image features and lidar features.

[0058] Specifically, for the three-primary color image, the current scene is photographed from multiple perspectives by a visual sensor to obtain the three-primary color image. The three-primary color image is input into a preset encoder to extract image features, such as using an image feature encoder. For the lidar point cloud data, the current scene can be detected by a lidar sensor to obtain the lidar point cloud data. The lidar point cloud data is input into a preset encoder to extract the lidar features, such as using a deep learning model that specifically processes point cloud data as a lidar feature encoder.

[0059] For example, Figure 2 As shown in the figure, a camera can be used as a visual sensor. First, a multi-view RGB image is obtained through the camera, and the lidar point cloud data is obtained through the lidar. Then, the encoder is used to obtain the corresponding image features (i.e., camera features) and lidar features.

[0060] Step S200: convert the image features and the laser radar features into a bird's-eye view space for fusion to obtain a fused bird's-eye view feature.

[0061] Specifically, since the denoising method for the point cloud data itself needs to be highly dependent on the accuracy and reliability of the point cloud information, if the point cloud data is inaccurate, the subsequent denoising accuracy will also be reduced. In order to reduce the dependence on point cloud data, this embodiment processes the image data and the lidar data separately, and then fuses the feature data of the two in the bird's-eye view space. Since the coordinate systems corresponding to the image features and the lidar features are not the same, in order to achieve the fusion of multi-sensor data at the feature level, this embodiment needs to convert the two feature data to the same space and then fuse them. For image features, it is necessary to convert the image features from the camera coordinate system to the world coordinate system, and then from the world coordinate system to the two-dimensional bird's-eye view space. For the lidar features, the lidar point cloud data itself is already a representation in the world coordinate system, so the lidar features can be directly converted to the two-dimensional bird's-eye view space. Finally, the converted image features and lidar features are fused in the bird's-eye view space to obtain the fused bird's-eye view features.

[0062] In one implementation, converting the image feature and the laser radar feature into a bird's-eye view space for fusion to obtain a fused bird's-eye view feature includes:

[0063] Convert the image features and the laser radar features to the bird's-eye view space respectively to obtain the image features in the bird's-eye view format and the laser radar features in the bird's-eye view format;

[0064] splicing the image features in the bird's-eye view format and the laser radar features in the bird's-eye view format to obtain spliced ​​data;

[0065] The spliced ​​data is input into a bird's-eye view encoder to obtain the fused bird's-eye view feature.

[0066] In summary, if Figure 2 As shown, the image features and the lidar features are transferred to the bird's-eye view (BEV) space respectively, and then spliced ​​together. The fused bird's-eye view features are then extracted by the bird's-eye view encoder to obtain the final fused bird's-eye view features. Specifically, for image features, the camera parameters (intrinsic and extrinsic matrix) can be used to convert the image features from the pixel coordinate system to the world coordinate system, and then the Z axis is processed to project to the bird's-eye view space. For the lidar features, the Z axis can be processed to project to the bird's-eye view space. The two feature data are spliced ​​in the bird's-eye view space. For example, the splicing process can be achieved by simple feature superposition or element-by-element addition. The spliced ​​data is then input into the bird's-eye view encoder, and the bird's-eye view encoder outputs the fused bird's-eye view features.

[0067] Step S300: Map the fused bird's-eye view features back to the point cloud space to obtain bird's-eye view lidar features.

[0068] Specifically, the goal of this embodiment is point cloud denoising and environmental perception, so the fused bird's-eye view features obtained in the bird's-eye view space need to be remapped back to the point cloud space, and these remapped feature data are the bird's-eye view lidar features (BEV LiDAR Features).

[0069] In one implementation, mapping the fused bird's-eye view feature back to the point cloud space to obtain the bird's-eye view laser radar feature includes:

[0070] Generate a coordinate index of each point cloud in a bird's-eye view space according to the laser radar feature;

[0071] Constructing the mapping data according to the coordinate index of each point cloud;

[0072] The fused bird's-eye view feature is mapped back to the point cloud space according to the mapping data to obtain the bird's-eye view lidar feature.

[0073] Specifically, the laser radar feature corresponds to the 3D space coordinate system, and the bird's-eye view space corresponds to the 2D space coordinate system. This embodiment uses the laser radar feature to construct the coordinate index of each point in the bird's-eye view 2D space, thereby constructing a surjective but non-injective mapping data, which can be defined as mapping F. The laser radar feature can be mapped from the point cloud space to the bird's-eye view space through the mapping F. Conversely, the fused bird's-eye view feature can be mapped from the bird's-eye view space back to the point cloud space through the inverse mapping of the mapping F, thereby obtaining the bird's-eye view laser radar feature.

[0074] In one implementation, generating a coordinate index of each point cloud in a bird's-eye view space according to the laser radar feature includes:

[0075] Z-axis compression and rasterization are performed according to the laser radar features to obtain the coordinate index of each point cloud in the bird's-eye view space.

[0076] Specifically, Figure 2 As shown, this embodiment compresses the laser radar features of the 3D space coordinate system on the Z axis and performs rasterization processing to obtain a bird's-eye view grid feature. The two-dimensional coordinates of the bird's-eye view grid feature in the bird's-eye view 2D space can be used to generate the coordinate index of each point in the bird's-eye view 2D space.

[0077] In one implementation, mapping the fused bird's-eye view feature back to the point cloud space according to the mapping data to obtain the bird's-eye view laser radar feature includes:

[0078] According to the inverse mapping data corresponding to the mapping data, the fused bird's-eye view feature is projected onto the corresponding point cloud to obtain the bird's-eye view lidar feature.

[0079] Specifically, the mapping data is used to achieve forward mapping, that is, data mapping from point cloud space to bird's-eye view 2D space. In order to achieve data mapping from bird's-eye view 2D space to point cloud space, the corresponding inverse mapping data is required. Through inverse mapping, the fused bird's-eye view features are projected to the corresponding points in the point cloud space. After the projection is completed, the bird's-eye view lidar features are obtained.

[0080] For example, the mapping F constructed above is used to map the fused bird's-eye view features from the bird's-eye view 2D space back to the point cloud space through the dense2point operation. The dense2point operation is the inverse process of the mapping F. The bird's-eye view grid features are projected to the corresponding points through the dense2point operation, and the bird's-eye view lidar features are obtained after the projection is completed.

[0081] Step S400: Fusing the bird's-eye view lidar feature and the lidar feature to obtain a fused lidar feature.

[0082] Specifically, in order to improve point cloud denoising and improve the accuracy of environmental perception, this embodiment fuses the bird's-eye view lidar features and lidar features that are fused with different sensor data at the feature level, thereby avoiding the impact of a single data source on the reliability of data results.

[0083] For example, Figure 2 As shown, the bird's-eye view LiDAR features and LiDAR features can be finally spliced ​​(cat fusion), and the final output can be obtained through simple linear layer processing, that is, the fused LiDAR features (Fused LiDAR Featrues). The mapping interaction between the bird's-eye view features and the LiDAR features in this embodiment realizes a lightweight and efficient fusion process, which can meet the real-time requirements and has strong scalability and can be applied to any BEV architecture.

[0084] Step S500: performing semantic segmentation according to the fused laser radar features to obtain point cloud semantic information.

[0085] Specifically, the fused lidar features are the result of multi-sensor fusion and multi-modal feature mapping interaction. Therefore, the fused lidar features can reflect the current scene information more comprehensively and accurately. By fusing the lidar features for semantic segmentation, each point cloud can be accurately classified, and the correct semantics of each point cloud can be obtained.

[0086] In one implementation, performing semantic segmentation according to the fused laser radar features to obtain point cloud semantic information includes:

[0087] Get a pre-trained multilayer perceptron;

[0088] The fused laser radar features are input into the multi-layer perceptron for semantic segmentation to obtain the point cloud semantic information, wherein point clouds with different semantics in the point cloud semantic information are labeled in different ways.

[0089] Specifically, in order to achieve noise removal, this embodiment sets a multi-layer perceptron to perform semantic segmentation tasks. The multi-layer perceptron is pre-trained with massive data and learns the complex mapping relationship between input and output. In practical applications, the currently obtained fused laser radar features are input into the trained multi-layer perceptron, and the multi-layer perceptron can accurately perform semantic segmentation on the input fused laser radar data and output the semantic information of each point cloud. In order to better distinguish different categories of point clouds, different annotation methods can be used to annotate point clouds with different semantics.

[0090] For example, Figure 3 As shown in FIG. 1 , in order to achieve noise removal, a segmentation module (i.e., segmentation head) specifically used for performing image segmentation tasks can be connected after the generation module that integrates the lidar features. The segmentation module can be composed of a multi-layer perceptron (MLP). The semantic label of the final lidar point cloud data can be obtained through the segmentation module, for example Figure 3 The semantic segmentation results in include rain noise points generated on rainy days. The multi-layer perceptron can be pre-trained with massive data to improve the accuracy of semantic segmentation, and the true value of the trained semantic label can be obtained by manual point-by-point annotation.

[0091] The advantages of the present invention are:

[0092] 1) In the present invention, multi-sensor fusion is realized through feature-level interaction, and a deep learning model is combined to achieve more robust noise recognition.

[0093] 2) The present invention can output more fine-grained point-level results through the interactive fusion of bird's-eye view (BEV) fusion features and point-level features, which is more refined and accurate than traditional BEV outputs. The point cloud denoising framework based on the fusion of LiDAR features and BEV features in the present invention can effectively fuse LiDAR features with BEV features, solving the problem of insufficient LiDAR features in many scenarios that make noise difficult to identify. The feature fusion module is simple and lightweight, and has scalability and real-time performance.

[0094] 3) The present invention can be applied in actual autonomous driving scenarios, greatly reducing the misidentification problem caused by point cloud noise, and providing strong technical support for the safety of autonomous driving and robot navigation.

[0095] Based on the above embodiments, the present invention also provides a point cloud denoising and segmentation device based on visual feature fusion, such as Figure 4 As shown, the device comprises:

[0096] Feature acquisition module 01, used to acquire image features extracted from three-primary color images of several viewing angles, and to acquire lidar features extracted from lidar point cloud data of the same scene;

[0097] The first fusion module 02 is used to convert the image features and the laser radar features into a bird's-eye view space for fusion to obtain a fused bird's-eye view feature;

[0098] A feature mapping module 03 is used to map the fused bird's-eye view feature back to the point cloud space to obtain a bird's-eye view laser radar feature;

[0099] The second fusion module 04 is used to fuse the bird's-eye view laser radar feature with the laser radar feature to obtain a fused laser radar feature;

[0100] The semantic segmentation module 05 is used to perform semantic segmentation according to the fused laser radar features to obtain point cloud semantic information.

[0101] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be shown as follows: Figure 5 As shown. The terminal includes a processor, a memory, a network interface, and a display screen connected through a system bus. Among them, the processor of the terminal is used to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a point cloud denoising and segmentation method based on visual feature fusion is implemented. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0102] Those skilled in the art will understand that Figure 5 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0103] In one implementation, the memory of the terminal stores one or more programs, and is configured to be executed by one or more processors. The one or more programs include instructions for performing a point cloud denoising and segmentation method based on visual feature fusion.

[0104] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0105] In summary, the present invention discloses a point cloud denoising and segmentation method and related equipment based on visual feature fusion. The method obtains image features extracted from three-primary color images of several perspectives, and obtains laser radar features extracted from laser radar point cloud data of the same scene; converts the image features and the laser radar features to a bird's-eye view space for fusion to obtain fused bird's-eye view features; maps the fused bird's-eye view features back to the point cloud space to obtain a bird's-eye view laser radar feature; fuses the bird's-eye view laser radar feature and the laser radar feature to obtain a fused laser radar feature; performs semantic segmentation based on the fused laser radar feature to obtain point cloud semantic information. The present invention performs feature-level fusion processing on the image data provided by the visual sensor and the laser radar point cloud data, and performs feature interaction between the bird's-eye view features and the laser radar features, and adopts a multi-sensor, multi-modal fusion method to effectively identify the noise in the point cloud, thereby improving the accuracy of environmental perception.

[0106] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A point cloud denoising and segmentation method based on visual feature fusion, characterized in that: The method comprises: Obtain image features extracted from three-primary color images of several viewing angles, and obtain lidar features extracted from lidar point cloud data of the same scene; Converting the image features and the laser radar features into a bird's-eye view space for fusion to obtain a fused bird's-eye view feature; Mapping the fused bird's-eye view features back to the point cloud space to obtain bird's-eye view lidar features; Fusing the bird's-eye view laser radar feature with the laser radar feature to obtain a fused laser radar feature; Semantic segmentation is performed according to the fused laser radar features to obtain point cloud semantic information.

2. The point cloud denoising and segmentation method based on visual feature fusion according to claim 1, characterized in that: The step of obtaining image features extracted from three-primary color images of a plurality of viewing angles and obtaining lidar features extracted from lidar point cloud data of the same scene includes: Acquire the three-primary-color images at several viewing angles through a visual sensor; Acquiring the laser radar point cloud data by using a laser radar sensor arranged in the same scene as the visual sensor; The three-primary-color image and the laser radar point cloud data are respectively input into their corresponding encoders to obtain the image features and the laser radar features.

3. The point cloud denoising and segmentation method based on visual feature fusion according to claim 1, characterized in that: The step of converting the image features and the laser radar features into a bird's-eye view space for fusion to obtain a fused bird's-eye view feature includes: Convert the image features and the laser radar features to the bird's-eye view space respectively to obtain the image features in the bird's-eye view format and the laser radar features in the bird's-eye view format; splicing the image features in the bird's-eye view format and the laser radar features in the bird's-eye view format to obtain spliced ​​data; The spliced ​​data is input into a bird's-eye view encoder to obtain the fused bird's-eye view feature.

4. The point cloud denoising and segmentation method based on visual feature fusion according to claim 1, characterized in that: Mapping the fused bird's-eye view features back to the point cloud space to obtain the bird's-eye view laser radar features includes: Generate a coordinate index of each point cloud in a bird's-eye view space according to the laser radar feature; Constructing the mapping data according to the coordinate index of each point cloud; The fused bird's-eye view feature is mapped back to the point cloud space according to the mapping data to obtain the bird's-eye view lidar feature.

5. The point cloud denoising and segmentation method based on visual feature fusion according to claim 4, characterized in that: Generating a coordinate index of each point cloud in a bird's-eye view space according to the laser radar feature includes: Z-axis compression and rasterization are performed according to the laser radar features to obtain the coordinate index of each point cloud in the bird's-eye view space.

6. The point cloud denoising and segmentation method based on visual feature fusion according to claim 4, characterized in that: Mapping the fused bird's-eye view feature back to the point cloud space according to the mapping data to obtain the bird's-eye view laser radar feature includes: According to the inverse mapping data corresponding to the mapping data, the fused bird's-eye view feature is projected onto the corresponding point cloud to obtain the bird's-eye view lidar feature.

7. The point cloud denoising and segmentation method based on visual feature fusion according to claim 1, characterized in that: The performing semantic segmentation according to the fused laser radar features to obtain point cloud semantic information includes: Get a pre-trained multilayer perceptron; The fused laser radar features are input into the multi-layer perceptron for semantic segmentation to obtain the point cloud semantic information, wherein point clouds with different semantics in the point cloud semantic information are labeled in different ways.

8. A point cloud denoising and segmentation device based on visual feature fusion, characterized in that: The device comprises: A feature acquisition module, used to acquire image features extracted from three-primary color images of several viewing angles, and to acquire lidar features extracted from lidar point cloud data of the same scene; A first fusion module is used to convert the image features and the laser radar features into a bird's-eye view space for fusion to obtain a fused bird's-eye view feature; A feature mapping module, used for mapping the fused bird's-eye view feature back to the point cloud space to obtain a bird's-eye view lidar feature; A second fusion module is used to fuse the bird's-eye view laser radar feature with the laser radar feature to obtain a fused laser radar feature; The semantic segmentation module is used to perform semantic segmentation according to the fused laser radar features to obtain point cloud semantic information.

9. A terminal, characterized in that: The terminal includes a memory and one or more processors; the memory stores one or more programs; the program contains instructions for executing the point cloud denoising and segmentation method based on visual feature fusion as described in any one of claims 1-7; and the processor is used to execute the program.

10. A computer-readable storage medium having a plurality of instructions stored thereon, characterized in that: The instructions are suitable for being loaded and executed by a processor to implement the steps of the point cloud denoising and segmentation method based on visual feature fusion as described in any one of claims 1 to 7 above.

Citation Information

Cited By

  • Millimeter wave radar point cloud generation method based on diffusion model

    CN119445030A

  • Stamping part steel warehousing positioning and position identification method

    CN120431171A